A sharp co-worker of mine pointed me to this post which adds a few more bits of info on how to identify the dependencies. Good read!!
Prefacing my Preface…
I want to first say that the only mention I’ve found with anything remotely related to what I am about to describe is jamestechman’s post: Update Rollup 3 for Exchange Server 2010 SP2 Installation Procedures. Although everyone alludes to the “trouble”, I have yet to find a cohesive document that describes WHY this crap breaks if you don’t massage Forefront. The dependancies that get created confuse the hell out of me because I have yet to see a product so perfectly integrated to the point of failure due to the “protection” services.
I know there are mentions out there about installing Exchange patches with the fscutility /disable and /enable commands, but here’s a strange scenario for you:
It was a day, not unlike today…
Picture that you were recently having major issues with Forefront in regards to downloads, mass error logs, etc. in your highly sensitive and very secure network. You disable the Forefront services on your Exchange servers while you troubleshoot the problem, parse through the logs, etc… Still with me?
While actively monitoring the system during this troubleshooting phase, you are not made aware of any health issues with the system. Patches occur as per your SLA and schedule dictates, your servers are rebooted from time to time during updates etc., and things are just churning along fine…
Two patch cycles later (hey, you know how this line of work is!) inadvertently pushes through a Service Pack update that was also cataloged as critical by MS, and as per policy, critical updates were pushed out. Half of the Exchange servers received the updates. This turns out to be an issue of course since all your servers need to be on the same version, so you immediately schedule an emergency patch for the remaining too.
IT DOESN’T MAKE SENSE!
Now, remember that half the systems have been already patched, rebooted, and actually had people working for a while due to the lucky draw of attaching to the correct CAS Server/MBX Server combo in regards to Service Pack level. There is nothing to expect out of the ordinary as you execute your emergency patching procedure.
Monitored via SCOM 2012, and using basic troubleshooting tests (Event Log Monitoring, BPA result testing, etc.), You are not receiving any reported alerts, and all Mailbox Databases (and their respective copies) are all green and healthy.
But nothing works.
Outlook just spins. OWA just spins. Everything wants to work, but nothing will. You are to the point of chucking your stress ball out the window.
The Hail Mary
The clocks winding down. You have less than 10 minutes to troubleshoot before you must admit defeat and need to open an MS Support Case…*sigh*…
Not this day my friend. Not this day…
Out of the darkness comes one ray of hope, one glimmer of a chance and maybe….just maybe…it’ll work.
You re-enable the disabled Forefront Services (on BOTH mailbox servers in the DAG, even though one patched fine and was running fine without it), and immediately the floodgates are opened. Services start sucking up RAM, the steam whistle blows. “Back to work I go Boss!!! Thanks for the kickstart!!!”
Queues start processing. Your 15 browser windows that were sitting there spinning immediately forward you to your communications portal. Your mailbox, *sigh*…Operations reports all systems are go, you can tell your manager that all is well now with the world, and crisis has been averted.
The End….Or is it???
Now while I have done some pretty cool #$&9 with Exchange, I am not going to sit here and say I know everything about it and all of it’s sub-components. I think I missed the class on why the hell ForeFront “randomly” (NOT) was keeping my Exchange system from firing up, when it was functioning (well, if you were lucky enough to be connected with the right CAS/DAG combo) just FINE prior to an afternoon meltdown after lunch.
If anyone know what the nuances are here that I’m missing, please fill in the puzzle for me in the comments, or at least direct me to a few blogs that fully describe how interdependant Exchange becomes on ForeFront after it is installed.
Alright, that’s it for story time. ‘Till next time…