My Technical Life

About 3-4 weeks ago we suddenly experienced a loss of the ability to sync our Entourage clients to the Solerant internal Exchange 2007 server.  This event happened out of the blue with no recent changes to the server that we could see.  The most recent change was the application of Exchange 2007 SP1 Hotfix Roll-up 3 about a week prior to the issue.  We opened a case with Microsoft and immediately began getting the run-around.  Exchange support didn’t want to touch it since the best we could tell it only affected Entourage users, and Mac Office support couldn’t help us on the server.  We stuck through it and began working with the Entourage team to find the key symptoms.

• Entourage was unable to sync folders that had more than 140-150 items in them.  This included Contacts, Calendars, etc.
• Using tcpflow to analyze the stream to the server you could see it litereally die mid-stream while receiving the list of items in the folder
• Synchronization was extremely slow, even in the folders that were working properly.
• OWA and ActiveSync were operating properly as far as we could tell.  (We later realized they were slow as well.)

Unfortunately due to the politics between the Mac Office team and the Exchange team this issue was open for over 3 weeks.  I can’t tell you for sure why it was not escalated to the proper people sooner, but after about 3 weeks of constant battling with the Entourage team I was scheduled for another late night call with an Exchange team member who did some server-side captures from diagnostic tools on the Exchange server.  After multiple traces were completed and uploaded to Microsoft the call was terminated with the expectation that I would receive an update as soon as they had something.  Apparently IIS processes were hanging, crashing, and restarting and we were not the only people having this issue.  The data was being forwarded to the Exchange product team for further analysis.

The following day I received a call from a US-Based (Thank God!) Exchange support engineer who had pulled my case.  Apparently he had isolated an issue which was causing serious performance issues with OWA and Activesync and noticed the similarities to a growing list of Entourage tickets.  At his instruction we looked in the registry and found exactly what he was expecting.

The engineer had identified a driver on all of these systems that was creating the performance problems.  (This part is out of my area of expertise, but I’ll try.)  This driver is a low-level driver that hooks into the file system driver stack interrupting all of the reads/writes.  Its purpose is to be an open file backup agent.  We stopped this driver from the command line with “net stop famv4″ and the server started operating completely as before and all Entourage clients synchronized immediately.

The most interesting part of this discovery was that there was no backup software of any kind installed on the Exchange server at the time of this discovery.  Microsoft had no idea where this driver was coming from which left it up to me to figure it out.  The only candidate in my mind was an online backup service we had tried out for a while and had issues with.  I contacted the vendor and confirmed that the driver was in fact their Open File backup driver.  Additionally, they were aware that it had been causing problems with OWA and Activesync but had heard nothing about Entourage issues.  They also confirmed that they had issues in the past with it not properly uninstalling the driver but were certain it had been fixed.  (Clearly it had not been!)  This software apparently updated itself while it was still installed causing our issue.  We removed the application during troubleshooting but it did not remove the offending driver.

I provided both the vendor and Microsoft with the appropriate information so they could contact each other and resolve the matter properly.  Unfortunately this vendor cost me over 35 hours of my personal life (family time, lost sleep, etc.) and probably another 15 hours on the clock and its unlikely they’ll do anything to make up for this.

Lessons Learned

• Microsoft Exchange and Entourage support teams do not work together in a collaborative way.  This may cause significant delay in getting an issue resolved
• Business Critical issues taken to the Mac Office team are not treated the same way that they are when they go to Exchange.  We’ve stayed on the phone for 24+ hours with Exchange support in order to resolve an issue.  The Mac Office team is not willing or able to do this.
• Be careful with backup software, especially those that handle open files.  I have a long history of issues with Open File Agents and this is a perfect example why.

I debated over the weekend wether or not to post/link the company here.  It would not be proper to do so and therefore I will not.  if you are concerned and need to know, please email me directly.