For a while we would just reset the cache, accept the loss of a file, and move on.  Eventually one of our technicians discovered a correlation between this and the EMBASSY Trust Suite pre-installed by Dell on D620/630/820/830 systems for file encryption using the embedded TPM.  There was not really much available on Google that would help us equate it to a problem with the application at the time, but over and over we found that removing the application would resolve the problem.

Today I finally reached my max level of frustration on this issue and mandated that our technicians check all systems and remove this app from any that have it installed.  Searches via Google turn up tons of issues with this application and the way that it hooks into the kernel disrupting things in ugly ways, particularly when related to networking and the Offline Files feature.  Hopefully this rant will add one more to the pile that shows up when searching on Google!

For the last 10 minutes I’ve been staring at the Beta license agreement, and that combined with the fact that I can’t find anyone else posting reviews online leads me to believe I can’t really go into detail here.  However, I will say that overall I am very impressed with what has been brought to the table.  Not only is it syncing everything it should have from the beginning but it is syncing much more quickly.  I look forward to being able to bring this to all of the Mac users at clients with Microsoft Exchange 2007.

Now, Therein lies the problem.  Many of our clients have held off on upgrading to Exchange 2007 due to the investment in new hardware required to do so.  Many of our smaller clients have run Exchange 2003 for years on a shared server with one or more other roles.  With Exchange 2007 they are faced with putting in a dedicated 64-bit server just for Exchange.

For a church, particularly a smaller church, they really have to step back and question if it makes sense to do it at all.  Some will choose to continue using Exchange 2003 until well beyond the end of support from Microsoft.  Unfortunately most will eventually have to make a move that improves performance and functionality.  Email storage has become a commodity and large mailboxes are completely impractical on Exchange 2003.

Those faced with the decision to move on will likely look into Exchange alternatives.  Ironically, prior to the EWS Beta we had found that many of these worked better with Entourage than Exchange 2003/2007 did.  There are many Exchange alternatives out there (Kerio Mail Server, Zimbra, etc.) which will work with your existing Outlook, Entourage and Mobile clients.

Another option that cannot be ignored is Google Apps.  Google extends the Educational Edition for free to all non-profit organizations.  With the synchronization apps released for Blackberry and Windows Mobile we are getting over the mobility shortcomings that have made me reluctant to recommend this solution.  The gCal plug-in will synchronize your calendars down to Outlook, and the Mac Calendar can sync via CalDAV, but syncing your contacts is still a manual process.  Additionally, Google apps does not have any equivalent of Tasks or Memos.  You must rely on third party solutions (Remember the Milk, Toodledo, Things, etc) for the Tasks functionality.  Of course Google has made it pretty obvious from the beginning that they would much rather you use the web interface than a local client.  This became a viable option this week when Google released their Offline Access functionality.

In addition to being a Microsoft Certified Partner, Solerant is a reseller for Kerio and Google Apps.  Clearly it is in our best interest to ensure that we are recommending the best solution to each customer individually and not forcing a one-size-fits-all approach on our customers.  Personally, I feel that while the majority of our customers in the past 8 years have gon with Exchange we will likely see a significant growth in Kerio and Google Apps in 2009.

—————-
Symptoms
=-=-=-=-=-=-=
When a large number of items is put into the inbox, calendar, contacts, etc., Exchange Active Sync (EAS) or Entourage Client Sync will fail.

Errors
=-=-=-=-=-=-=
The errors can vary from the standard “TIMEOUT” errors to 0×85010014.  In a DAV trace, to summarize, you will see I/O consistently “PENDING” and 0 bytes read from file.  This ‘file’ is the streaming file.

More Information
=-=-=-=-=-=-=-=-=-=-=
EAS and Entourage request mail from IIS/DAV.  DAV requests the mail from the STORE.  If the store needs to package a large amount of items, it will package them using a STREAM file.  IIS/DAV reads from this stream file.  IIS/DAV returns the information to EAS/Entourage.

Problem
=-=-=-=-=-=
IIS/DAV uses Kernel32::ReadFile() to read from the stream.  A 3rd party kernel driver (FAMv4.sys) intercepts these calls to ReadFile() and returns bad data.  This causes our ‘read’ thread to go into a perpetual ‘PENDING’ state.  For every “PENDING” returned, a POLLING thread is spawned, causing performance problems for the W3WP.exe ExchangeAppPool as well.

Resolution
=-=-=-=-=-=-=
Open REGEDIT.exe
Navigate to HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServices
Look for FAMv4 under the Services Key.
Set the “Startup” value to 4 so that it disables the FAMv4 service.
Open a cmd prompt.
Type NET STOP FAMV4.  This stops the FAMv4 service.
Sync your EAS or Entourage clients without issue.

More Information about FAMv4
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
FAMv4 stands for File Access Manager and it is made by Vision Works Solutions Incorporated.  Their website is http://www.vwsolutions.com/.  It is essentially an open file utility that allows backup programs to backup open files.  You can find out more about how it works from http://www.vwsolutions.com/FAM/howitworks.aspx. It works with several backup programs and since it is ‘hidden’ in the registry and not listed in services.msc, it is probably licensed by other backup companies as their open file backup solution.

We ask that our customers contact Vision Works Solutions at 1.888.310.6706 or the customer’s individual backup solution to obtain a fix for FAMv4.sys.

• Entourage was unable to sync folders that had more than 140-150 items in them.  This included Contacts, Calendars, etc.
• Using tcpflow to analyze the stream to the server you could see it litereally die mid-stream while receiving the list of items in the folder
• Synchronization was extremely slow, even in the folders that were working properly.
• OWA and ActiveSync were operating properly as far as we could tell.  (We later realized they were slow as well.)

Unfortunately due to the politics between the Mac Office team and the Exchange team this issue was open for over 3 weeks.  I can’t tell you for sure why it was not escalated to the proper people sooner, but after about 3 weeks of constant battling with the Entourage team I was scheduled for another late night call with an Exchange team member who did some server-side captures from diagnostic tools on the Exchange server.  After multiple traces were completed and uploaded to Microsoft the call was terminated with the expectation that I would receive an update as soon as they had something.  Apparently IIS processes were hanging, crashing, and restarting and we were not the only people having this issue.  The data was being forwarded to the Exchange product team for further analysis.

The following day I received a call from a US-Based (Thank God!) Exchange support engineer who had pulled my case.  Apparently he had isolated an issue which was causing serious performance issues with OWA and Activesync and noticed the similarities to a growing list of Entourage tickets.  At his instruction we looked in the registry and found exactly what he was expecting.

The engineer had identified a driver on all of these systems that was creating the performance problems.  (This part is out of my area of expertise, but I’ll try.)  This driver is a low-level driver that hooks into the file system driver stack interrupting all of the reads/writes.  Its purpose is to be an open file backup agent.  We stopped this driver from the command line with “net stop famv4″ and the server started operating completely as before and all Entourage clients synchronized immediately.

The most interesting part of this discovery was that there was no backup software of any kind installed on the Exchange server at the time of this discovery.  Microsoft had no idea where this driver was coming from which left it up to me to figure it out.  The only candidate in my mind was an online backup service we had tried out for a while and had issues with.  I contacted the vendor and confirmed that the driver was in fact their Open File backup driver.  Additionally, they were aware that it had been causing problems with OWA and Activesync but had heard nothing about Entourage issues.  They also confirmed that they had issues in the past with it not properly uninstalling the driver but were certain it had been fixed.  (Clearly it had not been!)  This software apparently updated itself while it was still installed causing our issue.  We removed the application during troubleshooting but it did not remove the offending driver.

I provided both the vendor and Microsoft with the appropriate information so they could contact each other and resolve the matter properly.  Unfortunately this vendor cost me over 35 hours of my personal life (family time, lost sleep, etc.) and probably another 15 hours on the clock and its unlikely they’ll do anything to make up for this.

Lessons Learned

• Microsoft Exchange and Entourage support teams do not work together in a collaborative way.  This may cause significant delay in getting an issue resolved
• Business Critical issues taken to the Mac Office team are not treated the same way that they are when they go to Exchange.  We’ve stayed on the phone for 24+ hours with Exchange support in order to resolve an issue.  The Mac Office team is not willing or able to do this.
• Be careful with backup software, especially those that handle open files.  I have a long history of issues with Open File Agents and this is a perfect example why.

I debated over the weekend wether or not to post/link the company here.  It would not be proper to do so and therefore I will not.  if you are concerned and need to know, please email me directly.

Solerant has recommended for years that it’s customers limit mobile devices to those that can access over Activesync or use Blackberry devices with a Blackberry Enterprise Server so that users can have full sync’ing capability.  While some organizations chose to use Blackberry or other devices without a BES, those devices were capable of obtaining their mail through OWA.  We intentionally avoided enabling/opening POP3 due to negative experiences with supporting POP3 as well as the security uncertainty it brings.  While we did enable and open IMAP in organizations with Mac users, we did not often use this for mobile devices.

This policy of support was received well by the majority of our customers because, in general, most people had no desire to carry a device around with their email on it.  When the original iPhone launched we had a sudden increase in users that wanted to be connected.  As organizations requested it we set up IMAP w/SSL and alternative SMTP submit ports for them to use with their new gadgets.  While this created some severe support headaches for a while from users that expected full wireless sync comparable to the Blackberry or Treo they just tossed aside, it waned pretty much immediately as users really only needed to send/receive email to be happy.

With the upcoming release of the iPhone Firmware 2.0 as well as the iPhone 3G full Activesync is being brought to the table.  Many organizations may be faced with implementing Activesync for the first time to support their users, while many others may have to prepare for a sudden onslaught of requests to switch to iPhones instead of their aging Windows Mobile, Palm, or Blackberry devices.  This may be significantly compounded by the more affordable pricing of the 3G iPhone.

How will this affect your individual churches?  I have some examples below and I would love to hear some feedback from others on this topic.

• User/executive demand to implement Microsoft Exchange or an alternative that supports Activesync in organizations that have otherwise not had it to date.
• Pressure to standardize on the iPhone 3G as a mobile device of choice.
• Increased mobile device support demands on IT Staff due to the influx of personal iPhones into the organization from people who would have never otherwise needed/wanted mobile email.

Thoughts?

What Apple Says About It

In Apples release notes for 10.5.3 they have very little to say about the fix/issue.  In fact, they sum it up with a single sentance; “Improves Active Directory binding and login.”  If anyone has any further details on what has been fixed in 10.5.3 in relation to this please forward it over!

UPDATE – I was able to fix the FQDN issue by setting the machine to static IP.  Apparently the Mac ignores the domain suffix search list if you are DHCP, even if you manually enter one.

At the beginning of May we made the decision to move forward on building our Mac testing lab in order to build out our standards and best practices for Mac integration into the Church IT environment, particularly in working with Active Directory.  To accomplish this we purchased a small stable of Mac Minis and several copies of OSX Server 10-user edition.  Additionally, I made the decision to convert to running on a Mac full-time and purchased a MacBook Pro.  (That will be the subject of a future blog, I’ll focus on the lab for now.)

What will we be testing?

Since our church client-base can vary drastically in size we want to address configurations that will be applicable to all of them.  The following scenarios will be built in the lab to document the strengths and weaknesses of each.

• Stand-alone configuration
  • Most mac users operate this way already
• Native OSX Active Directory Binding
• Native OSX Binding with Apple Schema Extensions
• Active Directory/Open Directory Integration
  • A.K.A. the “Triangle”
• Third-Party integration solutions
  • Thursby ADMit Mac
  • Centrify DirectControl for MacOS
  • GroupLogic ExtremeZ-IP

Lab Configuration

The current lab configuration consists of 4 Mac Minis and a PowerEdge SC1420.  Two of the Minis have been upgraded to 2GB RAM and are running OSX Server 10-User edition to act as our test server environment.  The remaining Minis will be used to test client operations both in and out of the office.  The SC1420 currently runs VMWare Server on top of CentOS 5.1 x64 and hosts two virtual machines, the Lab DC and the Lab File Server.  Additional virtual servers will be brought online as needed during the third-party tools portion of the lab.

Goals

The end result of this experience is to define solid best-practices and a comprehensive solution for the integration of Macs into existing Windows IT infrastructure, specifically in the church environment.

Why do my users want/need a Mac?

My rule has always been “Use the right tool for the job.” Looking back 4-5 years things were very different. Within the Audio/Video production end of the ministries, as well as the Graphics ministries, any perceived need was actually just platform preference. Most of the tools used by these professionals not only existed on both platforms, but in most cases were released with a preference for the PC platform. Mac users and PC Users could easily exchange files with each other, but even then a mixed environment created support issues. For this reason it was easier as a support department to force the issue and keep them on a PC platform.

Apple has continued to work heavily to recover their strength in media-production. While some would argue this with me, most people we work with have run screaming from the cross-platform Avid for the Mac-only Final Cut Pro. We’ve seen this particularly as they’ve transitioned to High Definition for broadcast. Additionally, other tools that are cross-platform (Adobe CS3, ProTools, etc.) have reached 100% release parity with the PC platform, or shifted the other direction to a Mac preference. It has become clear that in a media-driven church the production ministry cannot do their job effectively without standardizing on the Apple platform.

Several legitimate factors are beginning to drive demand outside of the A/V and Graphics ministries. Some, or all, of these likely apply to somebody within your organization.

• More users outside of the traditional media groups are putting together content for inclusion in services or broadcast. The availability of the less-expensive Final Cut Express has allowed a subset of the tools used by media to be put directly in their hands.
• ProPresenter has established a strong presence and is quickly displacing EasyWorship and MediaShout even in the small churches. If you’re using the Mac platform to present, many prefer to develop their content on the platform as well.

There are other factors driving demand within the church and it is very important to be aware of these.

• The “Peer Factor” – Simply put, Mac users love to show off the cool toys they’ve found. This can create a lot of buzz around the cool factor. Unfortunately it often creates demand within users who would rarely, if ever, use the functionality they’ve been shown.
• A “Free” Experience – In most organizations the Mac users have traditionally been completely self-supporting and are not integrated at all into the network. This means that the user experience often appears unencumbered by all the inconveniences that the average PC user has to deal with.
• Excellent marketing – Face it… Apple has created a phenomenal marketing engine which is driving unheard of growth in the market. People see it, and believe it, even if what they’ve heard isn’t the whole truth. Don’t believe me? Tell a barely technical user you are giving them a new computer with MS Vista on it. Nine out of ten times you will get an immediate negative reaction from a non-technical user with no direct knowledge of Vista.

Where does this leave you for support?

If you don’t provide help, they will do it without you!! Often, the media group will go this route without any support from IT at all. This can immediately create a huge rift between IT and media. Other groups within your organization won’t typically “break away” like this, but it is not in your interest to immediately say “NO” without working with the department to help them figure out what they truly need. I’m going to steal directly from Clif Guy here on this one… Users rarely come to you with the actual problem, but rather their perceived solution to the problem. Your default answer to a problem should be yes, assuming that it is in-line with the needs and strategy of the ministry. We don’t want to be in the habit of providing toys to anyone who asks for them, but you must provide tools when truly needed.

Do Macs cost more… to buy?

I have seen every possible argument on why a Mac is cheaper than a PC, but the reality of the matter is that if I take a typical PC user in any church and replace the PC with a Mac I have spent significantly more. This typical PC user has a glorified terminal which is used to access their email, run MS Office, and run their ChMS of choice. Assuming you are even buying a brand new system from Dell/HP/Etc this is an $800 system at most including the monitor and a 3-year, full-service, on-site warranty with priority support. The reality, though, is that the machine they are using today is likely more than adequate for their needs, even well beyond the typical 3-5 year lifetime of a system in business.

Assuming you go with the cheapest option from Apple you are looking at $599 for a base model Mac Mini plus the cost of the monitor, keyboard, mouse, and AppleCare. While more expensive already than a comparable PC it is close enough at this point you could justify it… assuming this user can work 100% within OSX. If, like most churches we work with, you are bound by one (or many) Windows applications, the cost does not end here. The biggest hurdles in this segment are the ChMS (Shelby, ACS, etc.) and any applications (hosted or otherwise) that require Internet Explorer. Since dual-booting is not a reasonable option to inflict on your users, often you are looking at the cost of virtualization software (Parallels or VMWare Fusion) plus a copy of Windows XP/Vista for this user on top of the cost already invested in the machine. Then, if a user demands Outlook (Not an Entourage fan, eh?), you now need to invest in another license for the Windows version of MS Office for your Mac user as well.

This rift grows even greater when you enter into the iMac and MacBook/MacBook Pro. While these systems have incredible style and grace, a feature-for-feature comparison to other hardware vendors rarely lands in their favor.

Do Macs cost more… to support?

As we’ve said above, in the past, the Mac users were a self-supporting island. As the Mac user base expands, even within isolated departments, the need for proper support and integration grows. Would you drop an XP or Vista computer on somebody’s desk in workgroup mode with no password on it? In a typical church environment that is exactly what every single Apple system on your network is. Beyond the support issues of these users having to access domain resources or change their domain passwords, the security risk in this scenario is just unimaginable. This brings about the issue of integrating your Mac users into your existing network properly. There are multiple options to do this, but that is a topic for another article.

Another support issue is warranty. While this is irrelevant for comparison to the 5 year old donated PC your user may be on today, we have to compare to new to be a fair comparison. A new Dell/HP/Gateway/etc. is going to likely be purchased with a three-year warranty that includes next business day on-site service. This goes for any system… desktop, laptop, server, etc. With AppleCare the standard support is mail-in or in-store. Per Apple’s current Terms and Conditions state “This comprehensive plan includes expert telephone technical support, global repair coverage, on site repairs for desktop computers, web-based support resources, and powerful diagnostic tools.” This is dependent on you being within 50 miles of an Apple service center and is clearly limited to their desktop products only. If your MacBook or MacBook Pro is broken you *will* be without it for a while as it is being repaired. You need to plan for this in your support plan (spare machines, etc.) if you plan to shift to a largely Mac-enabled workforce.

As a side note to the warranty issue, please be aware that Apple does not currently offer an equivalent to the “CompleteCare” option that Dell and others often include to cover accidental damage. Even with AppleCare you will be paying for the repair if you drop your MacBook or spill your Caramel Macchiato on the keyboard of your MacBook Pro.

How do I decide if a user is a good candidate for a Mac?

My number one rule is to ask if the user can do 100% of their job without loading Windows on the machine (Bootcamp or virtualization). There are multiple ways around this issue (Terminal Server, etc.), but if they cannot perform their duties without Windows there is a good chance they are not a good candidate.

Additionally, you need to know what they truly want/need the Mac for. What is their role within the organization? Will they need applications that are Mac only? Will they be working with your media ministries and exchanging content? Will they be sending work output to third parties that prefer/require the files be in Mac versions of their respective file formats?

I thought you said we should say “Yes”?

Ok.. I know. It sounds like I’ve given a TON of reasons NOT to use a Mac, but as I said before you must allow the right tool for the job. The goal of the above was to lay it all on the table up front. The fact of the matter is that the reasons to use a Mac in your environment often outweigh the disadvantages. If you are truly allowing the right tool for the job, it is no longer a question of “Should I allow Macs in our environment?” but rather “How do I best support my Mac users in a heterogeneous environment?”

I know that this is only the tip of the iceberg, but I hope that it has provided some value. This is only the first of many posts on the topic. I hope to continue digging in at both the when/where/why of the topic at a CIO level as well as some extremely detailed technical articles going forward.

Day One

The event was hosted at Henderson Hills Baptist Church in Edmond, OK. The campus was beautiful and was definitely well-suited to hosting an event. I arrived early and wandered through the vendor expo to get a good look at who was supporting the event. I’m sure I’ll miss someone important here, but the vendors that stand out in my memory follow. (I have left off a few OKC-area local-only vendors)

• Fellowship Technologies
• ACS Technologies
• Shelby / Arena
• Church Community Builder
• Quicktron (Manufacturers of RapidRun)
• Ford Audio-Video
• AVL System Design
• PureStream/StreamingForJesus
• Willow Creek Association

The first session was Tony Morgan presenting on “Why Techies Scare Me.” This was a top ten list presented with a touch of humor with some great points made about why Church IT can intimidate church staff, and particularly the leadership. Tony did an excellent job of making his points and set the mood of the event completely for me. A lot of the points he made resonated through every presentation.

Prior to session two we had our first opportunity to network. I was able to meet up with a ton of people who I only knew through chatting online or through their blogs. Due to a change in scheduling I attended Jason Powell’s VMWare presentation. He had some serious laptop issues which made for a challenging demonstration, but it was nice to the success he has had using the free VMWare Server instead of using ESX/Virtual Infrastructure.

The remainder of the morning was lunch followed by more networking and the Peer to Peer sessions which provided even more time to get to know everyone. For the afternoon session I attended Jon Edmiston’s session entitled “Information: the I in IT.” This session focused heavily on how his church focused their IT strategy tightly around their church strategy and some creative ways they were using data analytics and integration to accomplish their needs. Most impressive was the use of geocoding and mapping within their data analysis to better understand their membership and the integration they have done with plug-ins between Asterisk and Arena ChMS. It was a good presentation and avoided too much pitch vs. the valuable content.

The final presentation of the day was “Determining IT Strategy in a Church” presented by Jeff Hook of Fellowship Technologies. Walking right along the ongoing theme of the event he focused heavily on aligning IT Strategy with the church’s needs. Jeff is an excellent speaker and his presentation completely avoided any sales pitch in my opinion, however much of the presentation centered (as with Jon above) on strategy which was core to ChMS needs.

Day Two

Day two started off with a last-minute change of plans from Terry Storch of LifeChurch.tv. His original presentation was entitled Tech[NO]babble but God moved him to speak on his move into working for the church with a presentation called “I’m an idiot.” This definitely hit home with pretty much everyone at the conference since they have all made a choice at some point to leave the mainstream business world for the Church.

Session two was “IT & AV Synergy” with Anthony Coppedge. The presentation focused a lot on themes that are common between IT and AV but was not quite what I was hoping to get out of it. Regardless, I ended up taking tons of notes in this session and I was highly intrigued by his approach to budgeting. Unfortunately I was not planning to attend his second session so I will have to wait on the CD to come in and listen to the second session at that time.

Session three and four flowed naturally into each other. These sessions were Tony Dye with “The Church IT Paradox” and Clifton Guy with “Users or Customers?” respectively. Tony and Clifton both sold themselves as looking at IT from completely opposite styles with Tony being all about policies & procedures with Clif more about always saying yes, but in the end they really end up saying a lot of the same thing… Let the ministry drive IT, not the other way around. Clif clearly has a much more open approach to this than many, but there was value and truth from both presentations.

Paul Braoudakis wrapped up the event with a presentation on “Becoming a Change Agent” and threw in several minutes on a topic we could all relate to, the “early adopter tax” we all pay by getting the latest greatest gadget right when it comes out.

After Paul’s presentation we went for a tour of the LifeChurch.tv OKC main campus. It was interesting to see the campus, but primarily what technology they use for their production and broadcast.