Active Directory and Exchange Architecture Questions and Issues

Here's the background on our situation...

Right now, we are setup as three distinct companies with three complete Active Directory and Exchange systems. The three offices (One in the US, two in Europe) are connected via a three way VPN setup (so each office has secure communication to the other two). There is a two-way trust relationship setup in Active Directory for each setup. All systems are running Server 2003 and Exchange 2003.

There are about 160 mailboxes between the companies and 80 users (the additional mailboxes are either for IT subsystems, forwarding accounts or other uses).

The companies are officially merging together (instead of just having a trust relationship). So we're looking into a combined solution (based on a new name) where each office will be on the same systems (Exchange and Active Directory) as well as consolidating our IT infrastructure (there's a lot of duplication).

They hired an external company to come in and audit our IT infrastructure. They have made an official recommendation to outsource the IT infrastructure (and guess what, they want to provide the service).

I've been tasked with figuring out what to do. I've thought about it quite a bit, and I've come up with two options. The basic difference is where Exchange is hosted (internally our outsourced). Since outsourced is easy to fathom, I'll just detail the internal setup.

Since high availability is required, we want some geographic redundancy built in. So, what I've come up with is as follows (I'll call the offices Site1, Site2 and Site3):

Site1:

• FSMO Active Directory Role
• Exchange Mailbox Role - Primary
• Exchange Client Access, Hub Transport Server Roles
• DFS File Share Role (for shared drives)

Site2:

• Active Directory Role - Replicated from Site1
• Exchange Mailbox Role - Secondary, replicated using CCR Replication
• Exchange Client Access, Hub Transport Server Roles
• DFS File Share Role

Site3:

• Active Directory Role - Replicated from Site1
• Exchange Client Access, Hub Transport Server Roles
• File Share Witness (for failover)
• DFS File Share Role

So basically the cluster should be able to survive a single site failure without bringing down any of the other sites (or any of the systems). In the event of a double site failure, Exchange would stop completely.

So, my concerns are as follows:

1. Is this a reasonable setup? Or am I over complicating things?
2. The number of servers required (3 at each site since CCR Mailbox roles must be the only role installed).
3. Will it even work as summized (where it will automatically fail-over to the available node should a site or server go down)?
4. Since each office would specify a local Client Access server for its users, that server becomes a single point of failure for all local requests (But this is solvable by a manual DNS change)
5. Do all of these servers need to be on the same IP subnet for this to work? Or can I get away with using a hiearchial DNS for it (clientaccess.site1.foo.com, etc)?
6. This will let me set each office as an MX record (since there is a hub transport server in each office to connect to the internet) so if one office goes down we still should be able to receive email in the others, correct?
7. Maintainability. I have a fear that this setup will be too complicated to maintain in the long run (adding offices, removing offices, upgrading servers (both OS and hardware), etc). Is that a justified fear?

Now, there's also the question on whether to go with server 2003 or 2008... If we go the internal Exchange route, I think I can convince the powers to upgrade to 2008 (in fact we would need to upgrade to use Exchange 2010)... But is it really necessary or is that just one of my "wants" sneaking in to the plans (rather than a justifiable upgrade)...

Now, part of me just wants to go with outsourced Exchange since it'll alleviate some of these issues (or most of them). However after looking at the costs, the break-even point is about 1 year, so after that outsourcing will be considerably more expensive. Couple that with the fact that some features we depend upon are not possible outsourced -- at least with the companies we looked at-- (such as Shared Mailboxes, Active Directory coupling including SSO, centralized management, data security, etc). So I'm really torn as to where to go with this...

This is the first project of this scale that I'm attempting, so any help would be greatly appreciated...

Thanks in advance (and sorry for the book)...

From serverfault ircmaxell

• We are in a similar situation, except for the fact that in our case we are already one company. But we have offices in Cambridge, London, Stockholm, Shanghai and Atlanta. All connected via VPN. Three of these have Exchange servers (2 on Exchange 2010, the third one will be upgraded very soon). Most of our domain controllers run Windows 2003, but we are on the way to upgrade them all to Windows 2008. We have around 150 staff, spread all over the place. So very similar to your situation.

  So here are some answers from my point of view:

  1. If you have a decent IT team, then I would never ever consider outsourcing. In fact, even if your team isn't decent, I would rather spend some effort on making it decent. Your response times will be much better, your security setup is simpler, but most important of all: your IT team will have its primary focus on keeping the IT infrastructure operating at its best. The outsourcing provider's main focus is on making the most money out of you, not providing the best service.
  2. Your planned setup is very much feasible. Your main challenge will be to migrate everything onto a common domain, but that can be done step by step.
  3. Servers for most of what you need will not cost an arm and a leg. If you need to buy additional servers, the capital outlay for that will be small.
  4. Whether it works or not as summarized depends on how well you have your public DNS and your internal routing configured. It definitely can be made to work.
  5. I would highly recommend having separate subnets for each office. Makes life as a sysadmin a LOT easier. Use one decent sized subnet for each office, and then either use static routing for traffic between the sites or OSPF (most decent VPN routers will offer OSPF off the shelf). We actually have 2 separate subnets in most offices, keeping normal corporate traffic separate from engineering traffic (since our engineers tend to do a lot of funky stuff with DNS, DHCP, video streaming and what not else). And it works beautifully. In fact, we even have it to the point where engineers in any office can use a video stream from a streamer anywhere else, without having to know where it comes from.
  6. DO NOT attempt to have all computers on one big subnet. You will rip your hair out. Promise.
  7. We have three public mail gateways (located in the offices with the highest Internet connection bandwidth), which are all configured exactly the same and all forward to the nearest Exchange server, from where mail is distributed to the final mailboxes. No problem at all.
  8. Once you have a basic grip on routing and the like, you will find that this is not difficult to maintain. I have a total of around 150 servers spread over all these sites, about half dozen VPN routers, several dozen managed switches. We are a mixed setup (30% Windows, 70% Linux, on servers and workstations) and I have 4 people reporting to me. Not a problem at all.

  Trust your ability to learn, and you will succeed. The plan is good. I would go with Windows Server 2008 and migrate the Exchange Servers one by one to Exchange 2010. For the migration of Exchange you may need some external help (we needed it, and my guys are generally quite good with Exchange), but if you are afraid of the initial capital layout, you can also migrate everything one by one. There is no need to do this all in one big swell foop.

  ircmaxell : Wow! What an answer! Well, let me respond to some of the points you make. First, I wouldn't put everything on one subnet unless absolutely necessary. What I'm thinking is to put everything on one class C subnet, with specific subnets for each office (so for example 172.25.50.0 for site1 computers, 172.25.55.0 for site1 servers, 172.25.60.0 for site2 computers, etc)... Then everything can be managed by simply specifying the mask... One note, do you suggest multiple mailbox servers (one per site)? Or a single monolithic mailbox server replicated for redundancy?

  ircmaxell : Or is that exactly what you were warning against with respect to subnets? Would I be better off giving each office a completely separate subnet (not even part of the same \C)?

  wolfgangsz : Subnetting: what you describe is what very similar to what we do and what I had in mind, I didn't want to be prescriptive, as circumstances dictate the detail layout.

  wolfgangsz : Email: I would suggest having local mailboxes for all the users on site, with DAGs for the other sites mailboxes (that's an Exchange 2010 concept and very useful).

  ircmaxell : Fair enough (I noticed that in 2010, and it appears to be exactly what we want). I'm a developer by trade. But when our sys-admin left about a year ago I took over responsibilities (for my site). I like it and have learned a lot, but still have much to learn... So it's really good and useful to get a sanity check on these things... Thanks a bunch!
  From wolfgangsz