Replication using lsyncd

Sometimes it happens that you want to sync to servers, this could be e.g. load balancing of a site, surely you can always make a master/slave MySQL setup for your database, but you also need to sync all the data, so this post will be about replication using lsyncd which is a small daemon you install on your server.

So what is lsyncd actually?

lsyncd stands for live syncing daemon, and it works the way that you configure it to watch a directory using event monitoring interfaces such as inotify or fsevents, it queues up these events in a queue and every x second (Can be defined) it will execute these events and copy/create the files in another directory either locally or remotely.

The tool itself makes use of rsync as well as ssh for remote transfers, this allows you to make sure that no data is lost due to the checksums that rsync looks at, as well as when using ssh you ensure that all data is encrypted.

The installation of lsyncd is quite easy:

yum install -y lsyncd

or

apt-get install -y lsyncd

This will install lsyncd as well as putting some examples in the /usr/share/doc/lsyncd-*/examples/ folder.

The most useful examples you’ll use is lrsync.lua and lrsyncssh.lua

When using the lrsyncssh.lua¬†you should use pub keys for authentication to the remote host, either by using a passphrase protected key, or passphrase less, it’s also highly recommended to use a user for the remote transfer that only has access to specific folders if you care about your security.

There’s some good things to have in mind when configuring lsyncd here’s a copy of the config that I use including explanations what each thing does.

settings {
 logfile = "/var/log/lsyncd.log", -- Sets the log file
 statusFile = "/var/log/lsyncd-status.log" -- Sets the status log file
 }
sync{
 default.rsyncssh, -- uses the rsyncssh defaults Take a look here: https://github.com/axkibe/lsyncd/blob/master/default-rsyncssh.lua
 delete = false, -- Doesn't delete files on the remote host eventho they're deleted at the source. This might be beneficial for some not for others
 source="/var/www/domain.com", -- Your source directory to watch
 host="remote.host", -- The remote host (use hostname or IP)
 targetdir="/var/www/domain.com", -- the target dir on remote host, keep in mind this is absolute path
 rsync = {
 archive = true, -- use the archive flag in rsync
 perms = true, -- Keep the permissions
 owner = true, -- Keep the owner
 _extra = {"-a"}, -- Sometimes permissions and owners isn't copied correctly so the _extra can be used for any flag in rsync
 },
 delay = 5, -- We want to delay the syncing for 5 seconds so we queue up the events
 maxProcesses = 4, -- We only want to use a maximum of 4 rsync processes at same time
 ssh = {
 port = 8213 -- This section allows configuration for ssh specific stuff such as a different port
 }
 }

The above config should work just fine for most people. It’s pretty fast syncing the data, it’s all depending on your network and disk speed but it works just like rsync so you can expect same speeds.

To actually use lsync you can run the following command:

lsyncd /path/to/lrsyncssh.lua

Let me know if you have any questions or suggestions, but basically this would give you the possibility to sync directories between multiple hosts, can be good for either load balanced environments or near-realtime backups.

  • HamRadio

    Hi,
    on my Debian Wheezy your config file works only when changing line 1 from

    settings {
    to
    settings = {
    However the problem is that ownership and permissions are not replicated, they rather take those of the user who launched the lsyncd process.

    • lucasrolff

      Hi HamRadio,

      It might be that Debian requires the eq sign, but it’s deprecated to use the equal sign, and if you use it on CentOS 6 e.g., you’ll get the error:
      Warn: settings = { … } is deprecated.
      please use settings{ … } (without the equal sign)
      Error: Nothing to watch!

      If the rsync options as in the config described above, it should keep owner and permissions for the synced files (Also verified to work on at least CentOS 6), and probably earlier.

      • HamRadio

        Hi lucasrolff,

        I made the same test on a Debian Jessie box and everything works fine here. Ownerships and permissions are preserved, as long as the process is launched by root, as I do.

        And you’re right, now the equal sign is deprecated.
        In fact Debian Jessie uses version 2.1.5 while Debian Wheezy uses 2.0.7, that’s the culprit.

        Thank you!

  • Ajay Negi

    Hi can we transfer the files using user wise not a root wise,,,if possible then help us

  • Dnk

    Question – When running `lsyncd /path/to/lrsyncssh.lua` – does this run as a daemon? Or is this just something you run at boot and so on?