Replication using lsyncd

Sometimes it happens that you want to sync to servers, this could be e.g. load balancing of a site, surely you can always make a master/slave MySQL setup for your database, but you also need to sync all the data, so this post will be about replication using lsyncd which is a small daemon you install on your server.

So what is lsyncd actually?

lsyncd stands for live syncing daemon, and it works the way that you configure it to watch a directory using event monitoring interfaces such as inotify or fsevents, it queues up these events in a queue and every x second (Can be defined) it will execute these events and copy/create the files in another directory either locally or remotely.

The tool itself makes use of rsync as well as ssh for remote transfers, this allows you to make sure that no data is lost due to the checksums that rsync looks at, as well as when using ssh you ensure that all data is encrypted.

The installation of lsyncd is quite easy:

yum install -y lsyncd

or

apt-get install -y lsyncd

This will install lsyncd as well as putting some examples in the /usr/share/doc/lsyncd-*/examples/ folder.

The most useful examples you’ll use is lrsync.lua and lrsyncssh.lua

When using the lrsyncssh.lua¬†you should use pub keys for authentication to the remote host, either by using a passphrase protected key, or passphrase less, it’s also highly recommended to use a user for the remote transfer that only has access to specific folders if you care about your security.

There’s some good things to have in mind when configuring lsyncd here’s a copy of the config that I use including explanations what each thing does.

settings {
 logfile = "/var/log/lsyncd.log", -- Sets the log file
 statusFile = "/var/log/lsyncd-status.log" -- Sets the status log file
 }
sync{
 default.rsyncssh, -- uses the rsyncssh defaults Take a look here: https://github.com/axkibe/lsyncd/blob/master/default-rsyncssh.lua
 delete = false, -- Doesn't delete files on the remote host eventho they're deleted at the source. This might be beneficial for some not for others
 source="/var/www/domain.com", -- Your source directory to watch
 host="remote.host", -- The remote host (use hostname or IP)
 targetdir="/var/www/domain.com", -- the target dir on remote host, keep in mind this is absolute path
 rsync = {
 archive = true, -- use the archive flag in rsync
 perms = true, -- Keep the permissions
 owner = true, -- Keep the owner
 _extra = {"-a"}, -- Sometimes permissions and owners isn't copied correctly so the _extra can be used for any flag in rsync
 },
 delay = 5, -- We want to delay the syncing for 5 seconds so we queue up the events
 maxProcesses = 4, -- We only want to use a maximum of 4 rsync processes at same time
 ssh = {
 port = 8213 -- This section allows configuration for ssh specific stuff such as a different port
 }
 }

The above config should work just fine for most people. It’s pretty fast syncing the data, it’s all depending on your network and disk speed but it works just like rsync so you can expect same speeds.

To actually use lsync you can run the following command:

lsyncd /path/to/lrsyncssh.lua

Let me know if you have any questions or suggestions, but basically this would give you the possibility to sync directories between multiple hosts, can be good for either load balanced environments or near-realtime backups.