[NTLUG:Discuss] File Replication & 2-way Synchronization

[Scott Womer]

Hi Stan,

I've been down this road, and really tried to make it work... learn from my
pain (read extreme pain).  There's many, many issues with this setup, but
the two biggest things in your list which will soon drive you and your users
insane are:

1) You have a max throughput of 128k to-n-from any given site, which is good
for up to two users who don't use shared network resources beyond imap
mailboxes.

2) ADSL is unstable and unreliable, resulting in intermittent drops &
disconnects which will cause windows & NFS to puke, in turn making your
phone ring incessantly.

Most of your initial rsync issues can be addressed (to a degree), but the
danger is; once your users start getting accustomed to sharing data, that
usage will spike and they will soon become intollerent of the sync delay and
go balistic over the data loss and corruption as the likelihood of
simultaneous data access increases along with the spiking usage.

Dedicated WAN links are your best options, but if that's out of your
budget's reach, then you will need to at least upgrade your sites to
business class SDSL with performance and availability gurantees, then
connect them to each other with VPNs.  To further optimize your budget
dollar You will need to select one site to be the host of all "shared" data.
That site should get at least 1Mb/1Mb SDSL and the others could make do with
512K/512K SDSL.  Any data that is site specific should be housed locally,
but any shared data needs to be shared from the main 1Mb site.

There's obviously a lot more to it than this, but I'm sure you get the idea.

Good luck,
Scott Womer


----- Original Message -----
From: "Stan Tigrett" <stigrett at killer-webs.com>
To: <discuss at ntlug.org>
Sent: Wednesday, February 13, 2002 11:49 AM
Subject: [NTLUG:Discuss] File Replication & 2-way Synchronization


>
> PROBLEM: We have 3 offices in different locations who need access to each
of the other office's files.  The offices each run a linux/samba fileserver,
with clients using win9x to connect.  Each office is connected to the
internet via a ADSL line (1Mb/128kb) and run a linux firewall (a different
machine than the fileserver).  One office has about 10 gigs of data, and the
other two have about 2 gigs each.  The users can't handle using FTP - they
will only accept a 'new drive mapping' in windows.  It is not very likely
that the same file will be accessed by different users at different
locations at the same time.
>
> POSSIBLE SOLUTIONS:
> 1 - Mount the data remotely using NFS across the internet, using strict
firewall rules to only allow exporting to the 3 office IP addresses.  Use
samba to share the nfs mount to the users.
> Pros - Data will always be current.  Changes will occur in real time.
> Cons - The DSL links not very stable.  Would probably need a cron job to
test the nfs connection & re-mount it if neccessary.
> Data access will be extremely slow.
>
> 2 - Use rsync to perform 2-way synchronization of the files, using the
same strict firewall rules above...  Use samba to share the replicated
directories.
> Pros - Data access will be as fast as local data.
> Don't have to worry about the DSL links
> Cons - Data may be stale.
> Initial replication will take a long time...
> Deleting files from one server to the other...
>
> It seems that #2 is the best solution so far, using a cron job to
synchronize the files every hour or so, but I have one major caveat.  One of
the accounting programs these offices use creates several temporary *.dbf
files while they are working on it.  Using rsync without the --delete option
would eventually leave hundreds of these files laying around.  However, if I
do use the --delete option, those temp files may be deleted if a
synchronization occurs while a user is still working with them.
>
> If there were a way to only delete files that were created BEFORE the last
synchronization occured, then we would be in good shape, I think.  I haven't
found any kind of option like that, though.
>
> Any ideas? Advice? Other approaches?
>
> Thanks -
> Stan
>
> _______________________________________________
> http://www.ntlug.org/mailman/listinfo/discuss
>