Windows Server Almost Real-Time Directory Replication

The environment

• Windows 2003 R2 Standard File server
• Windows 2003 Web Edition server

The challenge

Need a way to mirror a directory (or share) from the file server to the web server. It needs to be "kinda real-time" with changes synced in a matter of seconds, not minutes. The two servers are on the same gigabit LAN.

We're looking at one directory with about 10K files in just shy of 500 sub-directories (not quite 2GB total size). A one-way deal is fine, all changes should be happening on the file server.

What would you suggest or what have you used in similar situations? Relatively inexpensive (or free!) would be preferred if possible.

Note: DFS-R isn't an option since Microsoft Web Edition never got the R2 option. The older DFS/FRS has caused me too much grief in the past, so no go there either.

Another Note: We used Microsoft's FolderShare to something very similar to this 4 or 5 years ago. Worked great! Alas, it got folded into Windows Live Sync and that appears to have a 20 folder limit.

From serverfault Chris_K

• And DFS isn't realtime. You probably need block-level duplication at your storage layer, which is usually a feature of high-end SAN and NAS units. Can you please describe your actual business requirement? That will help us give you more complete answers.

  Chris_K : Here's a link to my original version of this question: http://is.gd/ftejt . Partially looking for some mitigation around 2003 web edition constraints (but more details at the link)

  mfinni : OK, I understand better. So, since you mismatched the needs and requirements (the limit on Windows share session on Web), can you swap the servers? Put the Web Edition into production (where I assume you won't have people tinkering all the time) and put the Standard Edition into your dev environment? Or install something like an FTP server into dev and have your folks move files that way? Or WebDAV, and then mount drives to the WebDAV? /edit - reporting as possible answer.
  From mfinni

• Well, hbe happy with wishfull thinking. As in: does not exist, will never exist.

  You need that - get a real time replicated SAN, possibly a LOT more expensive than your IT budget for this decade.

  Your best option is DFS-R and that basically means not using web server for the data storage on the server side.

  Chris_K : and yet I used to do it with FolderShare for free. Ain't that a kick?

  TomTom : No, a lie. It is not real time. There simply is no real time scenario that does not involve block level commit on both ends. You may lie all what you want to yourself, but your original post makes it clear you need a real time solution, and for me anything with more thn - hm - 50ms delay on both ends does not fullfill your own requirement of this being real time or damn close to real time.

  Chris_K : Sorry, I really didn't mean to upset you. I'll loosen up the real-time lingo in my original post.

  Warner : With a Web app, the data is often stored in a database instead of using a storage solution. A storage solution is likely only to be justified when you're dealing with a large volume of multimedia.

  Warner : Oh, and DRBD is block level replication that's OSS and free. You don't necessarily need a huge budget for a storage solution.
  From TomTom

• As Clint mentioned in his comment, Robocopy might do the job for you if you need near real time file synching. Here's a snippet of some of the options available:

  /mon: Monitors the source, and runs again when more than N changes are detected.

  /mot: Monitors source, and runs again in M minutes if changes are detected.

  For the full list of options and parameters see here:

  http://technet.microsoft.com/en-us/library/cc733145(WS.10).aspx

  Chris_K : I love learning new things. /mon and /mot may be the ticket here.

  Chris_K : Look upon me and dispair! :-) robocopy "\\srv1\share" "\\srv2\share" /mir /mon:1 /mot:1 /log:c:\logs.txt /NJH /NJS /NDL /NP /Z Works wonderfully.

  joeqwerty : Glad to help...

  Jess : What if files are in use? Any issues then?
  From joeqwerty

• The currently equivalent of FolderShare would still work for you if you were running Vista/2008 or newer. It's currently known as Live Sync in the Live Essentials Beta, but it will be renamed to Live Mesh once the new Live Essentials is released. This shouldn't be confused with the older Live Mesh that never made it out of beta. Though technically, this new Live Sync/Mesh is using most of the old Live Mesh code.

  You might still be able to use the older Live Mesh Beta that is compatible with XP/2003. But no one knows how much longer they'll keep the service active.

  The file/folder limits you read about only apply when you're syncing to the cloud. Peer to Peer connections don't have those limits.

  Chris_K : The download page for Windows Live Sync (I believe that's the latest product name?) indicates that Server 2003 SP2 is an option :-) https://sync.live.com/clientdownload.aspx?ibm=10 Good to know about the file/folder limits, thanks! I have no need to sync to the cloud.

  Ryan Bolger : I think that's actually the *old* Live Sync (annoyingly confusing, I know). I haven't used it, so I'm not sure what its limitations are.

  Chris_K : You have a link to the current Live Sync?

  Chris_K : Ah found it: http://explore.live.com/windows-live-essentials-beta and you're right, no XP or 2k3 support. Bah and piffle.

  Ryan Bolger : Heh, yeah. The link was in my original answer. You have to run the installer for the whole Live Essentials Beta and just choose to only install the Sync component.

  Chris_K : Doh. Sorry about my link mixup.
  From Ryan Bolger

• It's perfectly achievable but not using anything that comes standard with Windows. I don't know of a ready made program to do this but Windows has an API function to monitor files and folders - the same one that allows Windows Explorer to show changes occurring in a file or folder as you watch. If you find nothing suitable I have something I wrote that should be pretty close to what you're after but I'll need to write some docs for it.

  From John Gardeniers

• OK, I understand better. So, since you mismatched the needs and requirements (the limit on Windows share session on Web), can you swap the servers? Put the Web Edition into production (where I assume you won't have people tinkering all the time) and put the Standard Edition into your dev environment? Or install something like an FTP server into dev and have your folks move files that way? Or WebDAV, and then mount drives to the WebDAV? – mfinni 0 secs ago edit

  Chris_K : Sadly, no more licenses -- and no budget to get more. We wanted the dev web server to be just like the prod web server (the old dev web was win2k server!). We just didn't anticipate the UNC session limitation! I think the robocopy answer will do the job.
  From mfinni

• Have you thought of using Unison

  Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

  To tell you the truth I have not used Unison in a Windows environment, only in Linux. It wouldn't hurt to give it a try.

  From Luis Ventura

• Hey Chris, Saw this post on your linkedin and figured id chime in. I have the requirement of moving large digital assets around multiple geographical locations. Sound like the machines are sitting right next to each other which makes things easier. Here are a couple ideas off the top of my head with a limited budget (isn't it always).

  beyond compare - This is a cheap utility that could synch over ftp getting around your UNC issues.

  Virtual SAN on VMware - This would address your block level real time, but seems like an overkill for this situation.

  cygwin rsynch - this is more for linux / windows synching, but should work.

  Webservice on web edition - installing a simple webservice on the web edition and either push the files as byte arrays over http soap packets (inefficient, but latency would be minimal over a lan) or have the webservice pull from the file server. Not the most elegant but it gets around your problem with relative simplicity.

  Chris_K : Hey, thanks for contributing Mr. Cutting :-) Out of curiosity, how are you handling this over long distances?
  From B Cutting