I just received the official case summary from Microsoft.  I thought I would publish it here for the benefit of anyone using old man Google to solve this problem.

—————-
Symptoms
=-=-=-=-=-=-=
When a large number of items is put into the inbox, calendar, contacts, etc., Exchange Active Sync (EAS) or Entourage Client Sync will fail.

Errors
=-=-=-=-=-=-=
The errors can vary from the standard “TIMEOUT” errors to 0×85010014.  In a DAV trace, to summarize, you will see I/O consistently “PENDING” and 0 bytes read from file.  This ‘file’ is the streaming file.

More Information
=-=-=-=-=-=-=-=-=-=-=
EAS and Entourage request mail from IIS/DAV.  DAV requests the mail from the STORE.  If the store needs to package a large amount of items, it will package them using a STREAM file.  IIS/DAV reads from this stream file.  IIS/DAV returns the information to EAS/Entourage.

Problem
=-=-=-=-=-=
IIS/DAV uses Kernel32::ReadFile() to read from the stream.  A 3rd party kernel driver (FAMv4.sys) intercepts these calls to ReadFile() and returns bad data.  This causes our ‘read’ thread to go into a perpetual ‘PENDING’ state.  For every “PENDING” returned, a POLLING thread is spawned, causing performance problems for the W3WP.exe ExchangeAppPool as well.

Resolution
=-=-=-=-=-=-=
Open REGEDIT.exe
Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
Look for FAMv4 under the Services Key.
Set the “Startup” value to 4 so that it disables the FAMv4 service.
Open a cmd prompt.
Type NET STOP FAMV4.  This stops the FAMv4 service.
Sync your EAS or Entourage clients without issue.

More Information about FAMv4
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
FAMv4 stands for File Access Manager and it is made by Vision Works Solutions Incorporated.  Their website is http://www.vwsolutions.com/.  It is essentially an open file utility that allows backup programs to backup open files.  You can find out more about how it works from http://www.vwsolutions.com/FAM/howitworks.aspx. It works with several backup programs and since it is ‘hidden’ in the registry and not listed in services.msc, it is probably licensed by other backup companies as their open file backup solution.

We ask that our customers contact Vision Works Solutions at 1.888.310.6706 or the customer’s individual backup solution to obtain a fix for FAMv4.sys.

About 3-4 weeks ago we suddenly experienced a loss of the ability to sync our Entourage clients to the Solerant internal Exchange 2007 server.  This event happened out of the blue with no recent changes to the server that we could see.  The most recent change was the application of Exchange 2007 SP1 Hotfix Roll-up 3 about a week prior to the issue.  We opened a case with Microsoft and immediately began getting the run-around.  Exchange support didn’t want to touch it since the best we could tell it only affected Entourage users, and Mac Office support couldn’t help us on the server.  We stuck through it and began working with the Entourage team to find the key symptoms.

  • Entourage was unable to sync folders that had more than 140-150 items in them.  This included Contacts, Calendars, etc.
  • Using tcpflow to analyze the stream to the server you could see it litereally die mid-stream while receiving the list of items in the folder
  • Synchronization was extremely slow, even in the folders that were working properly.
  • OWA and ActiveSync were operating properly as far as we could tell.  (We later realized they were slow as well.)

Unfortunately due to the politics between the Mac Office team and the Exchange team this issue was open for over 3 weeks.  I can’t tell you for sure why it was not escalated to the proper people sooner, but after about 3 weeks of constant battling with the Entourage team I was scheduled for another late night call with an Exchange team member who did some server-side captures from diagnostic tools on the Exchange server.  After multiple traces were completed and uploaded to Microsoft the call was terminated with the expectation that I would receive an update as soon as they had something.  Apparently IIS processes were hanging, crashing, and restarting and we were not the only people having this issue.  The data was being forwarded to the Exchange product team for further analysis.

The following day I received a call from a US-Based (Thank God!) Exchange support engineer who had pulled my case.  Apparently he had isolated an issue which was causing serious performance issues with OWA and Activesync and noticed the similarities to a growing list of Entourage tickets.  At his instruction we looked in the registry and found exactly what he was expecting.

The engineer had identified a driver on all of these systems that was creating the performance problems.  (This part is out of my area of expertise, but I’ll try.)  This driver is a low-level driver that hooks into the file system driver stack interrupting all of the reads/writes.  Its purpose is to be an open file backup agent.  We stopped this driver from the command line with “net stop famv4″ and the server started operating completely as before and all Entourage clients synchronized immediately.

The most interesting part of this discovery was that there was no backup software of any kind installed on the Exchange server at the time of this discovery.  Microsoft had no idea where this driver was coming from which left it up to me to figure it out.  The only candidate in my mind was an online backup service we had tried out for a while and had issues with.  I contacted the vendor and confirmed that the driver was in fact their Open File backup driver.  Additionally, they were aware that it had been causing problems with OWA and Activesync but had heard nothing about Entourage issues.  They also confirmed that they had issues in the past with it not properly uninstalling the driver but were certain it had been fixed.  (Clearly it had not been!)  This software apparently updated itself while it was still installed causing our issue.  We removed the application during troubleshooting but it did not remove the offending driver.

I provided both the vendor and Microsoft with the appropriate information so they could contact each other and resolve the matter properly.  Unfortunately this vendor cost me over 35 hours of my personal life (family time, lost sleep, etc.) and probably another 15 hours on the clock and its unlikely they’ll do anything to make up for this.

Lessons Learned
  • Microsoft Exchange and Entourage support teams do not work together in a collaborative way.  This may cause significant delay in getting an issue resolved
  • Business Critical issues taken to the Mac Office team are not treated the same way that they are when they go to Exchange.  We’ve stayed on the phone for 24+ hours with Exchange support in order to resolve an issue.  The Mac Office team is not willing or able to do this.
  • Be careful with backup software, especially those that handle open files.  I have a long history of issues with Open File Agents and this is a perfect example why.

I debated over the weekend wether or not to post/link the company here.  It would not be proper to do so and therefore I will not.  if you are concerned and need to know, please email me directly.

In the corporate world most organizations have a pretty clear definition of what devices are allowed to communicate directly to their Exchange servers.  Usually this is limited entirely to devices that the company provides.  From a support perspective this makes life easy, but is clearly unpopular.  In Church IT, as in Small Business, this is very unlikely to be the case.  Most mobile devices are owned by individuals, even if they are required to do one’s job.

Solerant has recommended for years that it’s customers limit mobile devices to those that can access over Activesync or use Blackberry devices with a Blackberry Enterprise Server so that users can have full sync’ing capability.  While some organizations chose to use Blackberry or other devices without a BES, those devices were capable of obtaining their mail through OWA.  We intentionally avoided enabling/opening POP3 due to negative experiences with supporting POP3 as well as the security uncertainty it brings.  While we did enable and open IMAP in organizations with Mac users, we did not often use this for mobile devices.

This policy of support was received well by the majority of our customers because, in general, most people had no desire to carry a device around with their email on it.  When the original iPhone launched we had a sudden increase in users that wanted to be connected.  As organizations requested it we set up IMAP w/SSL and alternative SMTP submit ports for them to use with their new gadgets.  While this created some severe support headaches for a while from users that expected full wireless sync comparable to the Blackberry or Treo they just tossed aside, it waned pretty much immediately as users really only needed to send/receive email to be happy.

With the upcoming release of the iPhone Firmware 2.0 as well as the iPhone 3G full Activesync is being brought to the table.  Many organizations may be faced with implementing Activesync for the first time to support their users, while many others may have to prepare for a sudden onslaught of requests to switch to iPhones instead of their aging Windows Mobile, Palm, or Blackberry devices.  This may be significantly compounded by the more affordable pricing of the 3G iPhone.

How will this affect your individual churches?  I have some examples below and I would love to hear some feedback from others on this topic.

  • User/executive demand to implement Microsoft Exchange or an alternative that supports Activesync in organizations that have otherwise not had it to date.
  • Pressure to standardize on the iPhone 3G as a mobile device of choice.
  • Increased mobile device support demands on IT Staff due to the influx of personal iPhones into the organization from people who would have never otherwise needed/wanted mobile email.

Thoughts?

Right out of the gates in the lab this week we had a significant issue trying to access data on our file server.  I bound the Mac to AD using the Directory Utility and logged in as a user.  When accessing shares on the AD Controller I was fine, but when hitting the file server I ran into some snags.  While the AD controller could be mounted by the system name alone I found that to mount the file server share I had to use the Fully Qualified Domain Name.  After mounting the share through the FQDN I was unable to write any data to the share and received an Access Denied error when I tried.  This morning after updating to 10.5.3 on the test machine I was able to open the share and have been reading/writing data without any further issues.  I still have to use the FQDN to hit the shares, but I have a feeling that may be a configuration issue with the DNS search prefixes not working.  I’ll deal with that problem seperately.

What Apple Says About It

In Apples release notes for 10.5.3 they have very little to say about the fix/issue.  In fact, they sum it up with a single sentance; “Improves Active Directory binding and login.”  If anyone has any further details on what has been fixed in 10.5.3 in relation to this please forward it over!

UPDATE - I was able to fix the FQDN issue by setting the machine to static IP.  Apparently the Mac ignores the domain suffix search list if you are DHCP, even if you manually enter one.

May was a heavy travel month for me which kept me away from the office and took away my will to blog for a bit so I wanted to give a bit of an update.

At the beginning of May we made the decision to move forward on building our Mac testing lab in order to build out our standards and best practices for Mac integration into the Church IT environment, particularly in working with Active Directory.  To accomplish this we purchased a small stable of Mac Minis and several copies of OSX Server 10-user edition.  Additionally, I made the decision to convert to running on a Mac full-time and purchased a MacBook Pro.  (That will be the subject of a future blog, I’ll focus on the lab for now.)

What will we be testing?

Since our church client-base can vary drastically in size we want to address configurations that will be applicable to all of them.  The following scenarios will be built in the lab to document the strengths and weaknesses of each.

Lab Configuration

The current lab configuration consists of 4 Mac Minis and a PowerEdge SC1420.  Two of the Minis have been upgraded to 2GB RAM and are running OSX Server 10-User edition to act as our test server environment.  The remaining Minis will be used to test client operations both in and out of the office.  The SC1420 currently runs VMWare Server on top of CentOS 5.1 x64 and hosts two virtual machines, the Lab DC and the Lab File Server.  Additional virtual servers will be brought online as needed during the third-party tools portion of the lab.

Goals

The end result of this experience is to define solid best-practices and a comprehensive solution for the integration of Macs into existing Windows IT infrastructure, specifically in the church environment.

I recently read an excellent article on InfoWorld titled “Why ‘no Macs’ is no longer a defensible IT strategy” which made some very good points on the topic. Having supported the IT environment in many churches for over six years, there has been no shortage of Mac users. Up until the last few years, however, this was limited to key individuals in media-driven ministries. Even within those ministries it was not 100%, but those times are changing. Not only are these ministries rapidly approaching 100% Mac adoption, it is spreading to pretty much any individual within the organization who works with these departments.

Why do my users want/need a Mac?

My rule has always been “Use the right tool for the job.” Looking back 4-5 years things were very different. Within the Audio/Video production end of the ministries, as well as the Graphics ministries, any perceived need was actually just platform preference. Most of the tools used by these professionals not only existed on both platforms, but in most cases were released with a preference for the PC platform. Mac users and PC Users could easily exchange files with each other, but even then a mixed environment created support issues. For this reason it was easier as a support department to force the issue and keep them on a PC platform.

Apple has continued to work heavily to recover their strength in media-production. While some would argue this with me, most people we work with have run screaming from the cross-platform Avid for the Mac-only Final Cut Pro. We’ve seen this particularly as they’ve transitioned to High Definition for broadcast. Additionally, other tools that are cross-platform (Adobe CS3, ProTools, etc.) have reached 100% release parity with the PC platform, or shifted the other direction to a Mac preference. It has become clear that in a media-driven church the production ministry cannot do their job effectively without standardizing on the Apple platform.

Several legitimate factors are beginning to drive demand outside of the A/V and Graphics ministries. Some, or all, of these likely apply to somebody within your organization.

  • More users outside of the traditional media groups are putting together content for inclusion in services or broadcast. The availability of the less-expensive Final Cut Express has allowed a subset of the tools used by media to be put directly in their hands.
  • ProPresenter has established a strong presence and is quickly displacing EasyWorship and MediaShout even in the small churches. If you’re using the Mac platform to present, many prefer to develop their content on the platform as well.

There are other factors driving demand within the church and it is very important to be aware of these.

  • The “Peer Factor” - Simply put, Mac users love to show off the cool toys they’ve found. This can create a lot of buzz around the cool factor. Unfortunately it often creates demand within users who would rarely, if ever, use the functionality they’ve been shown.
  • A “Free” Experience - In most organizations the Mac users have traditionally been completely self-supporting and are not integrated at all into the network. This means that the user experience often appears unencumbered by all the inconveniences that the average PC user has to deal with.
  • Excellent marketing - Face it… Apple has created a phenomenal marketing engine which is driving unheard of growth in the market. People see it, and believe it, even if what they’ve heard isn’t the whole truth. Don’t believe me? Tell a barely technical user you are giving them a new computer with MS Vista on it. Nine out of ten times you will get an immediate negative reaction from a non-technical user with no direct knowledge of Vista.
Where does this leave you for support?

If you don’t provide help, they will do it without you!! Often, the media group will go this route without any support from IT at all. This can immediately create a huge rift between IT and media. Other groups within your organization won’t typically “break away” like this, but it is not in your interest to immediately say “NO” without working with the department to help them figure out what they truly need. I’m going to steal directly from Clif Guy here on this one… Users rarely come to you with the actual problem, but rather their perceived solution to the problem. Your default answer to a problem should be yes, assuming that it is in-line with the needs and strategy of the ministry. We don’t want to be in the habit of providing toys to anyone who asks for them, but you must provide tools when truly needed.

Do Macs cost more… to buy?

I have seen every possible argument on why a Mac is cheaper than a PC, but the reality of the matter is that if I take a typical PC user in any church and replace the PC with a Mac I have spent significantly more. This typical PC user has a glorified terminal which is used to access their email, run MS Office, and run their ChMS of choice. Assuming you are even buying a brand new system from Dell/HP/Etc this is an $800 system at most including the monitor and a 3-year, full-service, on-site warranty with priority support. The reality, though, is that the machine they are using today is likely more than adequate for their needs, even well beyond the typical 3-5 year lifetime of a system in business.

Assuming you go with the cheapest option from Apple you are looking at $599 for a base model Mac Mini plus the cost of the monitor, keyboard, mouse, and AppleCare. While more expensive already than a comparable PC it is close enough at this point you could justify it… assuming this user can work 100% within OSX. If, like most churches we work with, you are bound by one (or many) Windows applications, the cost does not end here. The biggest hurdles in this segment are the ChMS (Shelby, ACS, etc.) and any applications (hosted or otherwise) that require Internet Explorer. Since dual-booting is not a reasonable option to inflict on your users, often you are looking at the cost of virtualization software (Parallels or VMWare Fusion) plus a copy of Windows XP/Vista for this user on top of the cost already invested in the machine. Then, if a user demands Outlook (Not an Entourage fan, eh?), you now need to invest in another license for the Windows version of MS Office for your Mac user as well.

This rift grows even greater when you enter into the iMac and MacBook/MacBook Pro. While these systems have incredible style and grace, a feature-for-feature comparison to other hardware vendors rarely lands in their favor.

Do Macs cost more… to support?

As we’ve said above, in the past, the Mac users were a self-supporting island. As the Mac user base expands, even within isolated departments, the need for proper support and integration grows. Would you drop an XP or Vista computer on somebody’s desk in workgroup mode with no password on it? In a typical church environment that is exactly what every single Apple system on your network is. Beyond the support issues of these users having to access domain resources or change their domain passwords, the security risk in this scenario is just unimaginable. This brings about the issue of integrating your Mac users into your existing network properly. There are multiple options to do this, but that is a topic for another article.

Another support issue is warranty. While this is irrelevant for comparison to the 5 year old donated PC your user may be on today, we have to compare to new to be a fair comparison. A new Dell/HP/Gateway/etc. is going to likely be purchased with a three-year warranty that includes next business day on-site service. This goes for any system… desktop, laptop, server, etc. With AppleCare the standard support is mail-in or in-store. Per Apple’s current Terms and Conditions state “This comprehensive plan includes expert telephone technical support, global repair coverage, on site repairs for desktop computers, web-based support resources, and powerful diagnostic tools.” This is dependent on you being within 50 miles of an Apple service center and is clearly limited to their desktop products only. If your MacBook or MacBook Pro is broken you *will* be without it for a while as it is being repaired. You need to plan for this in your support plan (spare machines, etc.) if you plan to shift to a largely Mac-enabled workforce.

As a side note to the warranty issue, please be aware that Apple does not currently offer an equivalent to the “CompleteCare” option that Dell and others often include to cover accidental damage. Even with AppleCare you will be paying for the repair if you drop your MacBook or spill your Caramel Macchiato on the keyboard of your MacBook Pro.

How do I decide if a user is a good candidate for a Mac?

My number one rule is to ask if the user can do 100% of their job without loading Windows on the machine (Bootcamp or virtualization). There are multiple ways around this issue (Terminal Server, etc.), but if they cannot perform their duties without Windows there is a good chance they are not a good candidate.

Additionally, you need to know what they truly want/need the Mac for. What is their role within the organization? Will they need applications that are Mac only? Will they be working with your media ministries and exchanging content? Will they be sending work output to third parties that prefer/require the files be in Mac versions of their respective file formats?

I thought you said we should say “Yes”?

Ok.. I know. It sounds like I’ve given a TON of reasons NOT to use a Mac, but as I said before you must allow the right tool for the job. The goal of the above was to lay it all on the table up front. The fact of the matter is that the reasons to use a Mac in your environment often outweigh the disadvantages. If you are truly allowing the right tool for the job, it is no longer a question of “Should I allow Macs in our environment?” but rather “How do I best support my Mac users in a heterogeneous environment?”

I know that this is only the tip of the iceberg, but I hope that it has provided some value. This is only the first of many posts on the topic. I hope to continue digging in at both the when/where/why of the topic at a CIO level as well as some extremely detailed technical articles going forward.


© 2007 My Technical Life | Powered by Wordpress