How to Synchronise Android with Thunderbird.. without Relying on a Trusted Party!

It seems that there is some interest in synchronising a mobile phone with a calendar and an addressbook that are maintained on a PC. The most widely cited solution is to use Gmail, GCalendar - essentially handing all data to Google seems to be the preferred solution. In some cases (e.g., with business data involved) such a solution is not acceptable.

Searching further we find reports of users being happy with how their iPhone syncs well with their MacBook and also their iPad. This seems nice as all data stays in the hands of the user. Drawback of this solution is that this creates a lock-in situation leaving the use no choice over what hardware to buy and what OS to use. Again in a case with company-provided hardware we have no choice over the latter and probably cannot not implement this solution either.

We were convinced that using (mostly) open source software and some (hopefully) cheap hardware it is possible to convince an email client (thunderbird) to synchronise with a mobile phone (any android 2.1).

This is a report (and a manual) for people who consider setting up their own system (or want to see what it would take). We spent a lot of time searching for the right information and consequently we go for a complete description on this page. This has the drawback that we repeat certain information already given on other pages. We were most careful in clearly indicating where the content comes from. We welcome feedback and questions (email: patrik bla bichsel@gmx bla net, where you replace the bla with a . ) if you get stuck using our instructions - clearly, we cannot provide any guaratee that the instructions will work, neither we can guarantee that we will respond to any request for information.

Harware Requirements

We don't need much but we do need:

We might use a new machine such as a Zotac ZBOX or a Mac mini as our server. If this is the case, we might have to update the drivers but we still need to go for the Ubuntu LTS release as Zentyal releases around LTS only.

Setup of the Server

We decided to install Ubuntu Server (32-bit edition/64-bit depending on the age of the hardware) and afterwards install Zentyal server to have a nice and tidy way of setting the main configuration.

Let's get started..

  1. Download Ubuntu server 10.04 LTS (required for Zentyal!)
  2. Install it.. (We choose to install OpenSSH because we want to manage the server using SSH later on. Maybe we choose other components (like Tomcat) that are not managed by Zentyal.)
  3. Following the instructions of the Zentyal Installation guide we first add the repository containing Zentyal by adding the following lines to /etc/apt/sources.list. To do so we open the file with our favourite editor (e.g., nano): 
    sudo nano /etc/apt/sources.list
    Then add the following lines at the bottom of the file. 
    # Zentyal repository
deb http://ppa.launchpad.net/zentyal/2.0/ubuntu lucid main
# Zarafa (Groupware) repository
deb http://archive.canonical.com/ubuntu lucid partner
  4. We need to add the key for the repositories by executing: 
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 10E239FF
  5. Finally we update, upgrade and reboot. 
    sudo apt-get update
sudo apt-get upgrade
sudo reboot

Configuration of Zentyal

We suggest the use of Zentyal as it makes several basic server configurations very easy to setup. For example, using Zentyal we can easily setup a CA and start issuing certificates (which we need for using SSL connections).

Let us start by installing the basic Zentyal packages together with the networking packages required for having configuring the server as our access point (AP). When installing the DHCP package there will be a number of other packages that Zentyal will install. One of them is the dyndns package, which will ask for several configuration settings. If your server is behind a router you can ignore these settings.

sudo apt-get install zentyal
sudo reboot

During the installation we have to set the HTTPS port of the administration interface. We do not choose the standard HTTPS port (443) as we want to access the groupware on the standard HTTPS port. We stick for now with the default port (as changing it now did not work for us) and we'll change it later using the Web frontend.

If you only want to set up an access point, installing Zentyal seems to be too much of an overhead. However if you want to synchronise your android phone with thunderbird, then you will need the Zarafa groupware later on. So we'll install Zarafa already. Assuming that your server has the IP 192.168.0.2 we can connect to:

https://192.168.0.2/

Using the Web frontend we install the Zarafa Gateway modules. In addition, we install the Zarafa groupware module, which can be found in Software Management → Zentyal Components. We select and install the groupware (Zarafa) module as well as the DHCP module.

There is not much that we change from the default configuration. One thing that we change is the port where the management console is listening. We go to System → General and set the port to 445 such that the Zarafa instance will be able to listen on the standard HTTPS port. From now on we have to connect to the Web frontend using:

https://192.168.0.2:445/

Let us summarise some important configurations of Zentyal.

Setup of an Access Point (AP)

If we want to use the server as an access point, we have to configure the network correctly. The reason for doing so might be that you don't have a WiFi router and still want to connect the mobile phone to the Internet using a (free) WiFi connection.

One word about the network topology might be appropriate here. We use two different networks, one for the wired infrastructure where the router is the gateway (and the DHCP server), and another network for the wireless infrastructure. Let us assume that the wired infrastructure uses the network 192.168.0.0/24 with the gateway having the address 192.168.0.1. The wireless AP uses IP 192.168.1.1, with the network 192.168.1.0/24. The AP serves as DHCP server and uses WPA-PSK to secure the wireless channel. Thus, the configuration of the AP will look as follows:

We now need to do the following:

  1. Forward IP packets between the wireless and wired networks (through the AP).
  2. Masquerade the traffic (from the AP to the wired infrastructure).
  3. Configure the AP (e.g., IP, DHCP, encryption).

Let's get started..

  1. We first enable the packet forwarding between the wireless and wired network. 
    sudo nano /etc/sysctl.conf
    Now we have to uncomment the following line 
    net.ipv4.ip_forward = 1
    optional
    Optionally we can also forward IPv6 packets by uncommenting: 
    net.ipv6.conf.all.forwarding = 1
  2. Coming to enabling masquerading. We have the goal that neither restarting the server nor restarting the firewall module will disables masquerading. To attain this goal we follow the instructions from the Zentyal developper page and enable a simple script that will run after the firewall module is restarted. Let us edit the file /etc/ebox/hooks/firewall.postservice and add acustom rule. 
    sudo nano /etc/ebox/hooks/firewall.postservice
    We make it a script by adding the following lines. 
    #add custom rules here
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
    alternative setup

    Following the description from Ubuntu Documentation we can implement the IP forwarding and masquerading using the ufw rules. For completeness reasons we repeat the instructions here.

    The rules files located in /etc/ufw/*.rules are a great place to add legacy iptables rules used without ufw, and rules that are more network gateway or bridge related.

    The rules are split into two different files, rules that should be executed before ufw command line rules, and rules that are executed after ufw command line rules.

    1. Firstly, packet forwarding needs to be enabled in ufw. We adjust two configuration files, in /etc/default/ufw we change the DEFAULT_FORWARD_POLICY to "ACCEPT": 
      sudo nano /etc/default/ufw
      DEFAULT_FORWARD_POLICY="ACCEPT"
      Then we edit /etc/ufw/sysctl.conf and uncomment the forwarding of IPv4 packets: 
      sudo nano /etc/ufw/sysctl.conf

      net/ipv4/ip_forward=1
      optional
      Similarly, for IPv6 forwarding we uncomment: 
      net/ipv6/conf/default/forwarding=1
    2. Secondly, we will add rules to the /etc/ufw/before.rules file. The default rules only configure the filter table, and to enable masquerading we will configure a nat table. We add the following to the top of the file just after the header comments or at the bottom of the file (after the COMMIT for the filter): 
      sudo nano /etc/ufw/before.rules

      # nat Table rules
*nat
:POSTROUTING ACCEPT [0:0]

# Forward traffic from eth1 through eth0.
-A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT
  3. Finally, we come to the IP configuration of the AP. You can establish the configuration using the ifconfig, iwconfig, wpa_supplicant, etc. commands manually. As mentioned before, we will use the Zentyal infrastructure to set the basic configuration. For the advanced wireless settings we'll use hostapd (hostapd documentation).
    sudo apt-get install hostapd
sudo /etc/init.d/hostapd stop
    optional
    Setting the configuration file of hostapd might not be extremely staightforward. Let us provide a short list of commands that help in detecting the hardware and the drivers associated with the hardware. You can skip this part if you just want to go along with the hope that our settings work on your machine too. 
    lspci | grep net
lspci -nnk | grep -i net -A2
iw list
iwlist chan 
sudo iwlist scan
lsmod 
dmesg
    We'll assume that you are running some hardware compatible with the nl80211 driver. Let's start with the hostapd configuration to figure out if everything works. We'll edit the hostapd configuration file /etc/hostapd/hostapd.conf: 
    sudo nano /etc/hostapd/hostapd.conf

    interface=wlan0 
driver=nl80211 
ssid=yourSSID 
hw_mode=b 
channel=3
max_num_sta=3
macaddr_acl=0
accept_mac_file=/etc/hostapd/hostapd.accept
ap_max_inactivity=3000
wpa=1 
wpa_passphrase=some secret that you will have to type when connecting to the network
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP CCMP
wpa_group_rekey=600 
wpa_strict_rekey=1
wpa_gmk_rekey=86400 
wpa_ptk_rekey=600
    alternative setup
    If you use newer hardware with a 802.11n WiFi card you should rather use the following values: 
    hw_mode=g
channel=11
max_num_sta=3
macaddr_acl=1
accept_mac_file=/etc/hostapd/hostapd.accept
ap_max_inactivity=3000
ieee80211n=1
ht_capab=? (use the commands from above to detect the right settings)
    We have to create the /etc/hostapd.accept and add all the MAC addresses of the devices we allow. Then we can check if the configuration is correct an fire up hostapd. 
    sudo nano /etc/hostapd.accept
sudo hostapd /etc/hostapd/hostapd.conf
    optional
    If you get any errors such as the description from our Zotac tutorial might help.
    Given that the deamon starts, we can log into Zentyal and configure the network. To do this you connect to the host (from the wired infrastructure as the wireless is not yet running). Assuming that your server has the IP 192.168.0.2 you can connect with (remember that we have chosen a port different from the usual HTTPS port) 
    https://192.168.0.2:445/
    Using the GUI you can start the DHCP module. In addition, we have to adjust the configuration of the wireless interface. We can do this be navigating to Network → Interfaces → wlan0. We set the method of wlan0 to static and configure its IP to be 192.168.1.1. Note that you can adapt the choice of the networks used for both wired and wireless configuration. You can ignore the mon.wlan0 interface (it is created by hostapd for monitoring purposes). Now your wireless should be accessible..
    Before celebrating we have to make the hostapd deamon start on startup of the system. We can do this by changing uncommenting the appropriate lines in /etc/default/hostapd. 
    sudo nano /etc/default/hostapd
    RUN_DEAMON="yes"
DEAMON_CONF="/etc/hostapd/hostapd.conf"
DEAMON_OPTS="-d"
    Let's restart and enjoy the wireless access.. 
    sudo reboot

Setup of a CA

To allow for secure communication, we need to create a CA and then issue certificates. In addition we will have to import the CA-cetificate on the client side to make all certificates that we issue trustworthy.

Zentyal offers support when it comes to this process such that creating a CA is done with a few clicks. We have to do two things:

After we created the certificate we go to Certificate Authority → Certificate Services and set the certificate to be used to the one we just created. We use the certificate for all the services. Possibly we have to use the trick described under hacks to change the certificate for some module.

Finally, we have to download the certificates to the clients.

  1. Firstly, we import the certificates onto the Android phone. We can import elements using an application from the phone company or an SD card (if available).
  2. Secondly, import them into Thunderbird to make it accept them when needed. Note that when the certificate is added to the Thunderbird list of certificates, we might have to add exceptions for the ports 443, 993 and 8443 (assuming that we used the standard ports).

Setup of Zarafa

Note that you have to setup a CA and use the right certificates for the different modules. If we have done so, we allow secure access to the Zarafa Web frontend and configure Zentyal as follows:

Synchronise the Addressbook

This is probably the most important task as we want to use the Thunderbird addressbook to initialise the addresses on the Android phone. During operation the synchronisation is a little less important as there are far fewer changes to the addressbook than to the calendar.

Unfortunately, we did not find an open source solution that allows to synchronise Zarafa with Thunderbird. The same situation is actually the case for synchronising Zarafa and Android. However, there is reasonably priced software that does the job. We will use:

Let us have a look how we can make those guys synchronise our addresses. We proceed in three steps: (1) Setup the phone to sync with Zarafa, (2) setup the server side of the Z-Sync plugin, and (3) setup thunderbird to use Z-Sync.

  1. Let us start with the Android synchronisation where we use the RoadSync application (we know, open source would be better.. but hey, this works at least!). As we want to have our server accessible from the road, we establish a dynamic DNS domain that we forward to the server. You might use www.dyndns.com or another service. Let us assume that we have reserved the domain thunderbird.android.dyndns.com.

    Before being able to connect to the server (actually the Zarafa groupware running on the server), we have to create a user. We navigate to Users and Groups → Users and create a user. In the RoadSnc configuration we only need to enter the hostname (thunderbird.android.dyndns.com), the username and the corresponding password. Now we check if the synchronisation works.

    optional
    As described on the Zarafa Wiki you might have to change the provisioning to make the synchronisation work.
    We may need to disable provisioning, unfortunately this will also disable the ability to wipe your phone remotely. The location of the config.php depends on where you installed Z-Push. We start looking in /etc/z-push/config.php. 
    sudo nano /etc/z-push/config.php
    In the file we change PROVISIONING to false and LOOSE_PROVISIONING to true. 
    define('PROVISIONING', false);
define('LOOSE_PROVISIONING', true);

    To enable secure exchange of information we enable SSL. To be able to do so, we have to copy the certificate to the phone as described in the CA setup.

  2. After registering with Z-Sync we get access to the code. Now we need to copy the server components (basically the folder z-sync) to a directory accessible to the web server on our server. Let us suggest that we use /var/www/. If we have the folder on another host, we'll use scp to copy it to the server.

    scp -r z-sync_1.3.6/z-sync/ admin@thunderbird.android.dyndns.info:/home/admin/z-sync/

    Now we log in to the server and copy it to the target directory.

    ssh admin@thunderbird.android.dyndns.info
sudo mv z-sync/ /var/www/
    optional

    As we have not yet configured file sharing between the server and the network, we copy the contents onto an USB stick. To mount the stick we use

    sudo mkdir /media/usb
sudo mount /dev/sdb1 /media/usb

    Then we copy the folder to the root of the web server.

    sudo cp -R /media/usb/z-sync /var/www/

    As indicated on the installation page, we make the subfolders accessible to the web server.

    sudo chown www-data:\ /var/www/z-sync/logs
sudo chown www-data:\ /var/www/z-sync/syncdata

    Out of curiosity we have a look at the configuration file config.php and change things if necessary.

    sudo nano /var/www/z-sync/config.php

    The synchronisation of extensive addressbooks with more than 1000 entries may take longer then 30 seconds. Thus, we may adjust the "max_execution_time" in the /etc/php5/apache2/php.ini to 120.

    sudo nano /etc/php5/apache2/php.ini
    max_execution_time = 120
  3. Now we need to get the client (thunderbird) to use the Z-Sync plugin. We install the plugin by going to the Thunderbird menu Tools → Add-ons. There we click the install button and navigate to the z-sync.xpi to install the extension. Then we enter the URI of our server https://thunderbird.android.dyndns.com/ (if you have copied it to a subfolder of the /var/www/ folder, don't forget to append this here!). Note that the Add-on will append z-sync to your URI! Then we hit the button Tools → Synchronise Addressbook and hope the extension does not complain.

Synchronise the Calendar

This task is twofold as we have a server that is authoritative about the data, we need to synchronise the local client (Thunderbird) with the server as well as the Android phone.

  1. We decribed the Android part already in the addressbook synchronisation part.
  2. The server configuration is explained on the Zarafa Wiki. Note that we need to change the Zentyal stub, which is located at /usr/share/ebox/stubs/zarafa/ical.cfg.mas. We set the following values: 
    sudo nano /usr/share/ebox/stubs/zarafa/ical.cfg.mas
    icals_enable = yes
ssl_private_key_file = /var/lib/ebox/CA/private/thunderbird.android.dyndns.com.pem
ssl_certificate_file = /var/lib/ebox/CA/certs/3948A19F61447E.pem
    Note that the certificate file uses the finderprint of the certificate as there might be several (revoked) certificates for the same URI.
    optional
    If we want to test the configuration the we just created we can overwrite the configuration file and start the ical service. 
    sudo cp /usr/share/ebox/stubs/zarafa/ical.cfg.mas /etc/zarafa/ical.cfg
sudo /etc/init.d/zarafa-ical start
    Possibly we have to check the log... 
    sudo tail /var/log/zarafa/ical.log
  3. Now you can access the calendar using Thunderbird and the Lightning add-on using the CalDAV protocol the URI https://thunderbird.android.dyndns.com:8443/caldav/user1.
    Alternatively, we can use the iCal version (which is not recommended but might offer better caching possiblities) using the URI https://thunderbird.android.dyndns.com:8443/ical/user1

To make Thunderbird accept the secure connection to our host, we have to add the certificate to its lists of trusted certificates. To do so we go in the Preferences of Thunderbird to the Advanced tab. Here we have a look at the certificates and add the certificate from the server.

Tipps, Tricks and Workarounds

Importing Data

Just to make you not too enthusiastic. Importing the calendar did not work for us when using an *.ics file exported from a Thunderbird client. We got the following reply:



As we did not have so many events in the future, we decided to
add them manually to the new calendar instead of trying to fix this
issue.



If we have an *.ics file from a Zarafa
server (e.g., a backup of our calendar or when we move to new hardware),
we can follow the description from the Zarafa
Wiki. We'll need curl for that:


sudo apt-get curl


In addition we need to make sure that the Zarafa-ical gateway is
running (follow this to get it running),
that we have a Zarafa user (we'll call her user) as well as a
Zarafa administrator (we'll call her zarafaAdmin). We create
those users with the Zentyal Web frontend.



Let us assume that we have a calendar.ics
file (on a machine different from the server). We start by sending the
file to the server. Note that we authenticate using the admin
user and we copy the file to her home directory. Thus, from the other
machine we execute:


scp calendar.ics admin@thunderbird.android.dyndns.info:/home/admin/calendar.ics



Now we connect to the server using ssh
and then simply need to curl it to the running Zarafa-ical
gateway. So we make sure that the server is running! Note that we will
use the zarafaAdmin (which we cofigured using the Zentyal
Web frontend) to upload the calendar to the user's calendar. The
-k switch is to avoid certificate checking (which would
fail due to using a self-signed certificate).


ssh admin@thunderbird.android.dyndns.info


curl -k -u zarafaAdmin:adminPW -T calendar.ics https://localhost:8443/ical/user



Dont't forget to configure the Zentyal firewall to allow
traffic from the internal network to Zentyal! Then you can check
access by connecting to:


https://thunderbird.android.dyndns.info:8443/ical/user




Zarafa Timezone Issues



Zarafa
forum