I don’t use Jaiku or Dodgeball, although there is sure to be a wailing and gnashing of teeth over the closing of Dodgeball. I had been using Google Noteboook, semi-extensively and although it is not being shutdown, development on Notebook is ceasing, which for my purposes anyway, amounts to the same thing.

This just reinforces one of my current productivity goals: to get all of my stuff in one place. Where is that one place? Really nicely solving this problem is 1) hard and 2) lucrative.

Hm…

Instead, I am going to invert this. I think that I already know how synchronization should work so I am going to describe that model and then see how my devices compare to my model and what could be changed about them to move them closer to the model.

What kinds of synchronization?

Right now I am addressing the user’s perspective on synchronizing a small peripheral device, a Palm, a cell phone, or a watch, to a more fixed central storage location on a desktop, laptop or server. There are three cases, 1) the peripheral is a data collection device with limited storage, so the goal is to transfer most data from the peripheral to the central location (although there may be some data transfer in the opposite direction), 2) the peripheral is a data storage device (data from the central location is sent out to the peripheral) and 3) true synchronization, the data in the domain of interest can originate at either location and need to move to the other location.

The Ideal Case
The ideal case is simple. The user would do nothing at all and the data would be synchronized as appropriate. There could be an initialization step where the devices would be introduced, the types of data and the role of the device would be specified, rules for dealing with conflicting data agreed upon, but beyond that nothing would be required of the user. The devices would periodically synchronized when the data changed on one.

The nothing-at-all requirement is the hardest to meet, in practice, but since it is my ideal, here is an example. A user downloads a train schedule to their Bluetooth enabled PDA. At the train stations there are Bluetooth transmitters at each entrance. When the user walks past the transmitter, their schedule is updated.

I am currently using a Linksys Wireless-G Notebook Adapter (WPC54G) on my Windows 2000 notebook. Windows attempts to establish a connection to the shared drive before the wireless network connection is established. This causes Windows to throw up an error message, “Connection to server cannot be established, blah, blah, blah,” with a an option to not reconnect next time and an OK to click. Nothing too objectionable there, except that this halts the login so when I come back ten minutes later, expecting to get to work, I have to click the OK and wait.

Part of this is a Linksys specific problem. My previous adapter (a D-Link) installed earlier. I would only get this message if the network weren’t available, or I was out of range. Even in that case though, this is not a helpful message. I mapped this drive and I have synchronized files from it. I want to be available if physically possible and if not, use the synchronized files.

I created four Subversion repositories: for my home directory and basic personal files, for code (personal and archive), for training documents, and for work related documents. All four of these are checked out to my server. The work documents are not checked out to my desktop, the code doesn’t go to my laptop and the training documents don’t go to work. I am not sure this is the ultimate mechanism. It might be easier to create only one repository and then control what is checked out to various locations by ignore directives (see below.)

There have been a couple of problems. I am learning Subversion, but the mechanism for ignoring files seems clumsy in comparison to CVS. In CVS, there are .cvsignore files that list patterns to ignore. The is a repository level ignore directives and then directory level .cvsignore files. The mechanism is pretty straight-forward. In Subversion, you add the pattern to the properties of the repository and the directory. I assume these are also stored in the file system, but they may be stored in the repository. In any case I can’t located them to edit easily.

The result of being at the low point on the learning curve is that I am getting files into the repository that aren’t really useful to share between multiple machines (.emacs.desktop, .bash_history) (Maybe I want different branches for the different machines?)

This problem is compounded by problems I am having with TortoiseSVN, the Windows client. Tortoise installs on the Windows Explorer context menu so that it is available when you right click, but I’ll be damned if I can understand the context rules. There is an “Add to Ignore List” command that appears when right clicking on some files or folders. Whether it is folders that are or are not currently under version control I can’t tell. The it seems like there is a low probability of it appearing.

Worse, it doesn’t work. Every time I have been luck enough to encounter the menu, I get the message, “Could not add file to the ignore list!” (The exclamation point is inside the quotes on purpose.)

I have also had a couple of problems with not being able to do anything because files are lock, or cannot be renamed for mysterious reasons which cause Tortoise to tell me to “Please execute the ‘Cleanup’ command.” Then the Cleanup command fails.

To be fair, I haven’t read the Tortoise documentation so perhaps I am doing something fundamentally wrong.

If you are following this shaggy dog story carefully, you will realize that I am having to do this synchronization by hand. This is a large drawback, and also a drawback because I have four repositories, so I have go navigate into each relevant directory (the root of the corresponding tree) to perform the synchronization. This can eventually be handled with scripting.

Otherwise, this system deals well with my seven objections to Windows synchronization.

1) The synchronization is fast and it happens on my schedule.

2) The response times are fast and uniform. All operations are working on files on my local disk.

3) This mechanism should work for all applications.

4) The issues of different config files on different boxes should be resolvable.

5) There are no problems with Windows synchronization.

6) I haven’t tested it yet, but I should be able to move a file or a folder on one box and have it migrate on the others.

7) There is a central storage point, so I can back it up regularly.

I haven’t tried this with my Music files yet; the next thing to do is to move all my web server files into the repositories.

The other drawback is that committing a check in requires a comment. This will save my butt many times in the future I am sure, but there are times it is more of a bother.

[Update 2005-02-17] I have added a link to TortoiseSVN. Having read the documentation, I see that I can add an ignore on a per directory basis through the directories properties. This simplifies things for me, but I don’t understand the context menu command not working. Perhaps I am still missing something. This seems easier to make happen using the command line.

The other property that I came across is the svn:external which allows you to have a single directory that comes for a separate svn repository. This will come in handy for my multiple repositories.

Example using bash:

stork$ svn propset svn:externals 'Training http://fq.d.n/Training
> Work http://fq.d.n/svn/Work
> code http://fq.d.n/svn/code' .

The newlines separate the properties, so they are required.
I cribbed this example from here. The use of propset in Subversion is poorly documented in general; most of the examples show the results of setting the properties via svn propget, but that is not necessarily helpful.

With this in place, I can do a single svn up or svn status and see the status of the whole tree. For commits and adds, I have to move into the subdirectory, which seems like a good thing, although perhaps this should be configurable.

Also, according to this page the support for this mechanism is not necessarily permanent.

Interestingly, the subversion developers consider externals a horrible hack which is likely to go away post-1.0. Fortunately, they also claim it will be replaced with a yet-unspecified “something better”.

I like this particular horrible hack for my purposes. It is useful for synchronization to have multiple data sources and to have them interoperate.

1) Synchronization is slow and doesn’t always happen at a convenient time.
2) The response time in unpredictable. When using a local copy operations can be very quick; when using the copy from the network – especially writing – can be very slow. This isn’t surprising, but I don’t like it.
3) Applications don’t always find the local copy when the network is unavailable. (Does the application call the wrong level of the API?)
4) Some programs cannot use the same preferences when working on a different box. For instance, Emacs saves a list of the files that I am currently using when it exits and reloads these files next time. Obviously the versions of this file that are saved on Windows have paths that are different than those saved on Linux, but even saving on two different Windows boxes causes problems.
5) Windows file synchronization doesn’t work correctly with the vaguries of wireless networking, particularly with Microsoft applications. Example: when editing an Excel doc that is available on and offline, if the network is lost, and then found, it is impossible to save the document.
6) It is limiting to have to change three boxes to move one folder.
7) The are multiple copies of each file saved in various location. On the one hand this is a good backup mechanism, but I would be more comfortable having known backups in known locations.

I have installed Subversion on all three boxes. I am using the Server as the repository and so far using the TortoiseSVN extensions on Windows, although I expect to use the command line interface to script some of this synchronization.

I have tried a couple of setups so far. Currently I am using 5 repositories, more or less parallel to the categories of this blog: Computers, Code, Training, Household and Home (as in ~ or home directory or My Documents). I envision that only the sever will have all of these categories, the others will be selected to different machines depending on use.

I may also use a different repository for iTunes – sharing between multiple servers with iTunes is very awkward, and iTunes doesn’t work well with a share drive. There is no need to keep versioned music files, but this would have the advantage of consistency.

The only problem that I have had so far is security. I am currently using http to synchronize between the various sources because it is available from anywhere, but I wanted to protect some of the files with a password. Somehow the version of Apache that I have has a bug in htpasswd and I can’t create a password file, so I am using Apache directives to block access except from my home. I will probably move this to ssh, but I don’t understand the client ssh configuration.

[Updated 2005-02-16: Some text in point 5) was lost, or maybe I never completed the thought.

I got it working on my desktop following the instructions in this article.

The important bit:

If you want to get started with Bluetooth, then you might want to get the Bluetooth Hotync working. This will at least verify that the Bluetooth on your desktop and Palm are working properly and that they can be paired with each other.

Be sure to update your Bluetooth drivers as per the above section, then follow the directions in your device’s user guide. The Tungsten T-series and Sony UX50 and TH55E (Euro version) all have decent Bluetooth Hotsync information in them. If you are using an SD card, you need the Palm Bluetooth Card Handbook (PDF).

These instructions cover using a “Serial Profile” which is the easiest way to get Hotsync working. The Palm Connection should be to a “PC”, not to a LAN. You must have “Local” check in the Hotsync Manager on your PC and you must have the COM port for it assigned to whatever COM port your BT Serial service is attached to. In my particular setup this is COM5, but yours may vary.

If you also want to have internet connectivity from your Palm, then continue reading the rest of this article.

The downsides are that I need to remember to turn the Bluetooth on and off on the Palm, else it runs down the battery really quickly and I need to remember to manually synchronize. My dream is that I could just set the Palm down in the vicinity of the desktop and they would find each other and synchronize automatically. Why do I need to be involved?

Does anyone know if the throughput would be faster if I went to the trouble of configuring the Bluetooth LAN support?