Jan-Piet Mens

I had intended doing the migration via SyncML but wasn't able to because the new SIM hasn't arrived yet. Instead I whipped up a small Perl script to get the data directly from the database and massaged that
into a text file that I imported into the Señora's Lotus Notes address book with a COL definition file, a bit like this one.

All that under six weeks. Twenty one minutes actually.

Another silent update from Nexthaus: SyncJE 2.23 for BlackBerry. I've been using SyncJE for some time now, and I'm very pleased with it.

Even though they have an OTA installer, you have to install the Windows executable to find the change log. I've done that for you:

Changes to Version 2.11
* Added option to set client timeout in settings
Version 2.20
* Added option to choose between WAP, MDS or BIS-B Internet connection method.
Version 2.21
* Fixed problem with local time and reminders
* Fixed wrong number of recurring events for week and month
* Added X-RIM-DCID field to contacts for PTT or DC support for iDen phones
Version 2.22
* Fixed registration problem
Version 2.23
* Fixed all day events problem with certain servers

If you look at their News page, the last update was in February of 2005! Why? What is the point? These guys make good software. Why don't they publicize it better? A few months ago, I even went to the trouble of writing them a message to this effect, but I was rewarded with silence.

Nexthaus have silently updated their SyncJE for BlackBerry to version 2.10. The only news concerns the Windows based clients, so I have asked their support to detail the changes to the BlackBerry version. I'll update this if and when I get an answer.

Changes to Version 2.10
* Made adjustment to device info for problem with GooSync and time zone
* Will now automatically add ?/dr(-n,n) to Calendar folder name when date range filter is used.
* New Registration process
* Get Folders will run in a separate screen

After solving the issues with the duplicates in my SyncML setup, I added another synchronization node to the picture ∗.

I've actually had that in there all the time, but I didn't want to complicate the issue by mentioning it.

The original setup hasn't really changed, but instead of altering contacts on Lotus Notes, I do so in AddressBook on the Mac.

It works, and I am now really happy. I've been hammering deletes, updates and inserts to all sides of this party, and I've yet to find a problem.

∗ Until the next fubar happens with my setup.

The advantage of relying on the "person" documents contained in the iNotes (Domino Web Access, DWA) mailfile instead of those in the personal addressbook on a Notes workstation, is that they travel with the mailfile, i.e. They replicate with mail and calendar documents.

I have stopped using the address book synchronization, as I recently explained. Instead, a small agent removes existing addresses in the PNAB and copies those from the mailfile.

Before continuing, you ought to create a backup of the personal address book (file, database, new copy)

Create a new Lotusscript agent as follows:

And copy this script into it

Depending on whether you prefer names sorted by first or last names, adjust the line which sets the preference. I want names sorted by surname, which is how it is set. If you prefer sorting by first or given name, set NameDisplayPref to the value 1.

Notice that I have removed the Synchronize Address Book agent from my mail file in so as to not use it by mistake.

Choose Recreate PNAB People from the Actions menu to launch the agent. If you already have people in your personal address book on the workstation (names.nsf), the agent will request permission to delete those.

If you answer with No, the agent terminates and doesn't otherwise alter anything. Answering yes to the question lets the agent remove all person (not connection or other documents) from your names.nsf, and it then copies all person and group documents from your mail file into your personal address book, indicating how many where thus created

This function can be used on each of your workstations without endangering the addresses in the person documents of the mail file. Do note that any changes to the names.nsf will of course be clobberred at the next run.

If you want read/write access to the address book in your mailfile, use iNotes to access it.

In the course of the last two weeks I've had trouble with my synchronization setup, but I think I've found the cause of the problem!

I quickly purchased the Nexthaus SyncJE client program for the BlackBerry because I considered that to be the best way to get my BlackBerry device synchronized with my syncml server, which it was is.

Thusly, I had a three way sync going: the Nokia N70 with the SyncML server, the BlackBerry with it too, and by virtue of the BlackBerry being connected to a BlackBery Enterprise Server with a Lotus Domino backend, the device's address book was being synchronized with my Lotus Domino mail file. The addresses being in the Lotus mail file, as soon as they were replicated via NRPC I could simply synchronize those with my Lotus Notes address book.

What else could I want? That was the perfect setup.

Hah!

After a bit of there and back, dozens of duplicate contacts suddenly appeared on the two devices, and I couldn't for the life of me determine what it was.

I then started investigating the SyncML server's log files, and at first glance noticed, that whenever the Nexthaus client performed a sync, the log was full of unknown property: UID messages. Being the clever chappie I am, I determined that must be the reason why Nexthaus' SyncJE couldn't associate its vCards with the server's database. That seemed quite logical to me.

Things got even worse, when I set up SSL on the server. I messed about with the server's URL on the mobile phone, and got not only duplicates, but bunches of them; eighty or so each time I synced.

I stopped using SyncJE for BlackBerry and set about cleaning up all my duplicate contacts, resetting the phone, etc. etc. etc.; you get the picture.

I've since logged an email call to both the support at Synthesis and at Nexthaus, and both parties answered astonishingly quickly (both within a couple of hours), for which they have my appreciation. It turns out, that actually neither was doing anything wrong, at least according to the two companies.

Yeah, right. So? What is the problem?

I spent half a day on the weekend synchronizing the shit out of the two devices, keeping logs and making notes of all changes. Have a look at a the initial list of the directories in which I'm keeping the stuff.

01-n70-okay-slowsync-to-bb/
02-resync-from-bb/
03-bb-added-1-updated-pam-telephone/
04-sync-n70/
05-bb-change-pam/
06-both/
07-apple-sync-n70/
08-n70-change-to-HTTPS/
08-n70-change-to-HTTPS.1/
09-fromBB/
09-fromBB.1/
10-change-url-on-bb/
11-remove-dup-hanna-on-n70/
...

Believe me, there is smoke coming out of my ears, after keeping notes on all of this.

It turns out, both Synthesis and Nexthaus are right: there isn't a problem. Everything works as it is supposed to, with the exception of a single duplicate calendar entry, that was created along the way. It happened during the slow sync in 08-n70-change-to-HTTPS; as far as I can tell due to an alarm setting on the phone (ALARM_MSG = z:systemSystemSoundsalarm.wav). I deleted the duplicate and thats it.

So?

The problem is here. For some reason which I don't in the slightest feel like debugging, the Lotus Notes synchronize address book action is updating documents in the (iNotes-)mail file even if I haven't changed anything in the local address book (PAB). When they are sent back to the mail file and finally find their way back to the BlackBerry device, there is something in them that causes the Nexthaus SyncJE client to create duplicates of the contacts. A cursory glance at the code indicates that the problem might be in the iN_CopyDocToDB function in the Notes design template. It would appear that documents are not simply being modified (i.e.: the name has changed, we change the name) but actually deleted and re-created, which might of course influence SyncJE into believing it is a new document. This is simply a suspicion, as I don't know exactly how SyncJE records "seen" entries.

As I say, I currently cannot be bothered to find the exact cause of the problem because I can live with the situation. I won't use this functionality any longer. I will limit myself to using the Person documents which are in the mail-file proper, only copying them to the personal address book (copy, paste) when necessary.

Now my iCal feed from SyncML database is reliable.

After installing Synthesis SyncML Server and using it for a bit, I decided to look into its innards to see if I could use some of the data. You may recall I'm using this server in a three-way combination, and this adds another piece to the mosaic. I don't use Microsoft Outlook but Lotus Notes. My calendar entries and tasks are replicated onto my BlackBerry, and from there, they are synchronized via SyncML to my server. It is from here, that I wanted to attempt to utilize the calendar and task entries in order to feed other programs, notably Apple's iCal.

I am using the "PRO" version of Synthesis AG's product which stores all data received via SyncML into a backend database via ODBC which, in my case, lands in a MySQL datbase on the server. One of the great advantages of this setup is that I have access to my data.

It occurred to me to have a closer look at the tasks and events tables, which are respectively named SYNC_TASKS and SYNC_EVENTS (it is a bit unfortunate that in the default configuration, the table names are capitalized; I usually reserve caps for reserved words in SQL statements, but I'm forced to capitalize the table names here, or the examples will not work). Although I describe tasks in a bit more details, the schema for the calendar (events) and for contacts is also easy to determine.

Let me first have a look at a task I created on my BlackBerry, and which I then synchronized to the server with the Nexthaus SyncJE for BlackBerry SyncML client. I have filled out all of the fields available on the BlackBerry.

When pushing this task down to the SyncML server, it receives the task in iCalendar format (RFC 2445).

BEGIN:VCALENDAR
VERSION:1.0
PRODID:Blackberry
BEGIN:VTODO
DESCRIPTION:Test iCal integration
DUE;VALUE=DATE-TIME:20070121T160000Z
PRIORITY:5
UID:1806671339
SUMMARY:Todos in Synthesis SyncML
STATUS:NEEDS ACTION
END:VTODO
END:VCALENDAR

I note that not all of the fields I entered on the BlackBerry are contained in the vCard; the categories are missing, and possibly more important, so is the time zone of the entry. Nexthaus have documented this, and it is probably due to my version 4.0 of BlackBerry OS. Their readme clearly says:

There are some limitations in syncing certain fields depending on BlackBerry OS Version. Limitations prior to BlackBerry OS version 4.1: Contacts, will only sync one address either home or work; Calendar, All-day events flag will not be set; Tasks, Time zone setting in task is ignored, the timezone set in device settings will be used instead.

and their documentation also lists further limitations:

There are some fields in the Blackberry PIM that can either not be accessed or is outside the scope of the standard formats used. Here is a list of some fields that are not supported in the vCard, vCal standard or not accessible for sync.

Contacts (vCard): Birthday, PIN, Categories, Web Page and User fields 1-4 in Contacts are not supported.

Calendar (vCal): Recurring events: Monthly by Position and Yearly by Date not supported. Sensitivity (Private/Public) not supported.

Tasks (vCal): Categories not supported.

Upon receiving the vCard, the SyncML server converts that card into SQL statements and inserts or updates the task in the database. For the event we are looking at, the resulting row in the database is

mysql> SELECT * FROM SYNC_TASKS WHERE summary LIKE '%Synthe%';
*************************** 1. row ***************************
   TASKS_KEY: 60
   FOLDERKEY: 6
     SYNCVIS: 10
  CATEGORIES:
       CLASS:
    MODIFIED: 2007-01-16 23:36:49
     CREATED: NULL
     SUMMARY: Todos in Synthesis SyncML
 DESCRIPTION: Test iCal integration
    LOCATION:
    TIMEZONE: NULL
         DUE: 2007-01-21 17:00:00
    PRIORITY:
      STATUS: NEEDS ACTION
    ATTENDEE:
  ALARM_TIME: NULL
ALARM_SNOOZE:
ALARM_REPEAT:
   ALARM_MSG:
1 row in set (0.00 sec)

What can I do with this information? Well I can further process it. In fact I'm going to create iCalendar files on the fly for import into Apple's iCal or Mozilla Sunbird. Here is a couple of screen shots of the result.

I was originally inspired by the article Using PHP to Make Basic vCalendar/iCalendar Events; it seemed so easy, that I took up the task of creating something similar. The program itself is quite simple and supports both the calendar with recurring events as well as tasks or todos. I haven't tested all possible combinations of recurring events, but it works for me. You are bound to have to change the timezone data.

How do I use this program? I have it running under an Apache web server in its own directory. Since iCal complains when the subscription file is not called .ics, the program itself is named thus, but I have to tell Apache how to handle such files.

$ cat .htaccess
AddType application/x-httpd-php .ics

Order Deny,Allow
Deny from all
Allow from 127.0.0.1
Allow from 192.168

This forces Apache to use PHP when processing files ending in .ics. You might like to, as I have, protect the resource from prying eyes with Apache's Access Control directives.

The program itself uses Synthesis' SYNC_USER table to perform authentication, although that only works if I use the clear-text password in the database table (the default configuration file for Synthesis SyncML server defines true).

In iCal, use the Calendar menu to Subscribe to a new calendar and enter the URL to your installation in the Subscribe to field. After clicking on Subscribe, iCal will attempt to retrieve the calendar from our PHP program, but it will be requested to authenticate. Use your SyncML username and password (the same values to use in your mobile phone) to access your calendar. At this point you can chose a new name for the subscription and request that iCal refreshes your calendar periodically.

Note that calendars subscribed to thusly are read-only.

This might or might no work with Microsoft Outlook; I haven't tested it. If you do succeed, drop me a note.

From the SyncML Discussion Forum:

Hai All
Anyone Know How To Install Syncml Client
If No Then
Tell Me How To Develop

Not bad; if he can't install it, he'll develop a new one.