When I recently found out about the Entourage for EWS (Exchange Web Services) Beta I was very excited.  With all the trouble we’ve had supporting Entourage for years a change to EWS instead of WebDAV is very welcome.  Needless to say I jumped in on the Beta.

For the last 10 minutes I’ve been staring at the Beta license agreement, and that combined with the fact that I can’t find anyone else posting reviews online leads me to believe I can’t really go into detail here.  However, I will say that overall I am very impressed with what has been brought to the table.  Not only is it syncing everything it should have from the beginning but it is syncing much more quickly.  I look forward to being able to bring this to all of the Mac users at clients with Microsoft Exchange 2007.

Now, Therein lies the problem.  Many of our clients have held off on upgrading to Exchange 2007 due to the investment in new hardware required to do so.  Many of our smaller clients have run Exchange 2003 for years on a shared server with one or more other roles.  With Exchange 2007 they are faced with putting in a dedicated 64-bit server just for Exchange.

For a church, particularly a smaller church, they really have to step back and question if it makes sense to do it at all.  Some will choose to continue using Exchange 2003 until well beyond the end of support from Microsoft.  Unfortunately most will eventually have to make a move that improves performance and functionality.  Email storage has become a commodity and large mailboxes are completely impractical on Exchange 2003.

Those faced with the decision to move on will likely look into Exchange alternatives.  Ironically, prior to the EWS Beta we had found that many of these worked better with Entourage than Exchange 2003/2007 did.  There are many Exchange alternatives out there (Kerio Mail Server, Zimbra, etc.) which will work with your existing Outlook, Entourage and Mobile clients.

Another option that cannot be ignored is Google Apps.  Google extends the Educational Edition for free to all non-profit organizations.  With the synchronization apps released for Blackberry and Windows Mobile we are getting over the mobility shortcomings that have made me reluctant to recommend this solution.  The gCal plug-in will synchronize your calendars down to Outlook, and the Mac Calendar can sync via CalDAV, but syncing your contacts is still a manual process.  Additionally, Google apps does not have any equivalent of Tasks or Memos.  You must rely on third party solutions (Remember the Milk, Toodledo, Things, etc) for the Tasks functionality.  Of course Google has made it pretty obvious from the beginning that they would much rather you use the web interface than a local client.  This became a viable option this week when Google released their Offline Access functionality.

In addition to being a Microsoft Certified Partner, Solerant is a reseller for Kerio and Google Apps.  Clearly it is in our best interest to ensure that we are recommending the best solution to each customer individually and not forcing a one-size-fits-all approach on our customers.  Personally, I feel that while the majority of our customers in the past 8 years have gon with Exchange we will likely see a significant growth in Kerio and Google Apps in 2009.

There aren’t a whole lot of details floating around about Apple’s implementation of ActiveSync on the iPhone yet, but I stumbled across an article from last week that I missed on the Exchange Team Blog that had a few surprise details.

Obviously, we have been working together for a while on their implementation. What can we say today? They are doing Direct Push and most of the Exchange 2003 SP2 policies (including remote wipe). They are doing email, calendar and contacts sync, and global address lists. While previously you could get you Exchange mail on an iPhone via IMAP, getting contacts and calendar required a tethered sync through iTunes. Doing it wirelessly will be much better (IMHO).

Particularly noteworthy, Apple will implement a couple key Exchange 2007 EAS features.

  1. Autodiscover – This means those of you running Exchange 2007 can now make it super easy for your users to configure their iPhones to sync with Exchange. Here’s what you do as an admin. All the iPhone user does is enter his/her email address and password. Pretty cool.
  2. HTML Mail – See your mail in its full HTML glory. Obviously the iPhone shows mail in HTML format today, so it’s safe to assume your Exchange mail will retain its HTML formatting on the iPhone as well.

I don’t have a lot more to say about the iPhone. Perhaps after I’m worked with it a bit I’ll share some more thoughts.

More of the Article can be found here.

So after nearly 8 months of battling this ongoing issue Microsoft has decided to speak up and acknowledge it. Today they published KB 948496 titled “An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers” which addresses this very issue. I’m certain that the work of the Exchange team had a lot to do with the final recognition of this issue so much thanks to them!

To summarize, Microsoft has acknowledged that computers which have TOE capable network cards and have SP2 installed may face issues due to the Scalable Network Pack being enabled by default. This is due, primarily, to faulty drivers but as I stated in my previous blog entry even the most recent drivers direct from Broadcom do not address all of the issues. To address this problem they have released an update EXE which disables the Scalable Network Pack. Additionally, they provide methods to manually disable these features in the article.

While this is a major battle won in the war on this issue we cannot consider it complete won until Broadcom releases a driver that fully fixes the issue and Dell/HP/Etc. certify that driver for their customers.

In late August to early September of 2007 we had a sudden breakout of issues with brand new servers. We found that we were installing brand new Dell PowerEdge 2900/2950/1900/1950 units that were having significant Network Performance issues and Exchange performance problems.

Network Throughput

The first symptom that was noticed was network throughput. All of these servers were plugged into 1000BaseTX networks with clients running at 1000BaseTX or 100BaseT. Throughput varied from test to test, but overall speeds for CIFS/SMB file copy operations were at speeds that would be slow on a 10Mbit network. Various troubleshooting was attempted with Dell and various switch manufacturers and we found that putting the servers on a 100Mbit port resulting in full 100Mbit throughput. Various combinations of Flow Control on switch ports seemed to help but this was not a usable permanent solution.

Exchange MAPI Errors

A second symptom cropped up almost immediately after this one where certain users were unable to open their Exchange 2003/2007 mailbox through Outlook yet were able to open it through OWA or sync with their mobile devices. Profiles were re-created and attempts made on other desktops but this did not fix the issue. The problem appeared to be tied to the user and MAPI. A scan of the event logs on the server showed multiple errors related to MAPI. (9646 Errors)

Stumbling onto a Clue

In late October of 2007 Dell shipped out their October edition of the Server Updates disc in their OpenManage updates pack. Luckily a few of my clients subscribed to this and I noticed it come in. Out of desperation we popped the update disc into one of the servers that was giving us problems and found that nearly everything was in need of updating. (This was a bit of a surprise as this particular server was only 6 weeks old at this point.) Immediately after updating all of the drivers/firmware the first problem appeared to be completely resolved. We were now able to max out throughput over the Broadcom gigabit cards in the servers, even running a bonded pair. Something definitely changed.

Upon further testing we found that the fix came from the updated network drivers dated 10/20/2007 on Dell’s support site. The actual driver installed by this update was version 3.5.8.0 dated 7/27/2007, way earlier than the 10/20 date of release from Dell. (One can only assume Dell takes a long time to QA their drivers.)

Dell 10/20/2007 Driver

Exchange Issues Continue

Unfortunately, the updated driver did not resolve our Exchange issues. Being forced to dig further into the issue we stumbled across a post on the Exchange Team Blog detailing problems caused by network drivers (particularly Broadcom) and the Windows Scalable Networking Pack which is installed and enabled by default in Windows 2003 Service Pack 2.

The Fix

The blog post goes fairly extensively into the problem, however to summarize it appears that certain network vendors have been extremely slow to properly implement the features required by the Scalable Network Pack in their drivers. The key word in that sentence is “properly” as the pack tries to turn it on anyway resulting in serious issues. At best this causes performance issues/limitations which may not be noticed at all. For us this problem was causing severe limitation on the TCP stack that presented itself in many ways. The following are a few of the issues that were affecting us that we have later found to be a direct result of the problem.

  • MAPI errors resulting in limitations on the number of MAPI clients that could connect
    • Once the limit was reached no further MAPI sessions could be created on the server
    • Users unable to open their mailbox through Outlook
  • Randomly unable to make RDP connections to the server or connections dropped immediately after connecting
  • Intermittent RPC communications failures
  • Networking throughput is severely decreased (This was partially fixed by the October Dell driver update)

Unfortunately, the driver provided by Dell is far too old to address these problems. Per the Microsoft blog posting the Broadcom driver needs to be at least 3.7.19 to fully support the Scalable Network Pack. This leaves us in a tight spot to address the performance issues. We are left with two options.

  • Install unsupported drivers directly from the chipset manufacturer
    • Addresses the source of the problem
    • Potentially introduces support issues with your server vendor
  • Disable the Scalable Network Pack
    • Disables key improvements brought by SP2 as a workaround
    • Should make note of this and re-enable these features when drivers are later updated

Obviously it is in the best interest of our customers to stay supportable so we have chosen to address this by disabling the features of the Scalable Network Pack that are affecting performance on our servers. The easiest way to do this is by issuing the following command at a command prompt.

Netsh int ip set chimney DISABLED

Further details can be found on the Exchange Team Blog posting and I highly recommend you read it. This fix has addressed the key problems we experienced without our customers’ file servers and Exchange servers but we are keeping a close watch on this hoping for a permanent resolution. We have tested the latest drivers directly from Broadcom on test servers and have found that while it addresses these immediate issue we continue to have network throughput issues with Vista client systems under certain circumstances.

Conclusion

While this incident has severely questioned my trust in Broadcom I have to admit that they are not the only vendor having problems. While Broadcom seems to be the biggest offender here we have seen throughput issues (particularly with Vista clients) on servers using Intel NICs. Most of these are Dell and HP servers which haven’t received vendor driver updates in several years. A driver update from Intel instead of the Dell/HP/etc. site appears to resolve the problem immediately.

I see this problem as one so severe vendors should be calling their customers to address it yet if you call Dell/HP support they don’t even have a clue what you are talking about. Hopefully this post has provided you some insight into the problem. We are still working on this internally and hope to find a permanent solution. I will update with another post when we do.


© 2007 My Technical Life | Powered by Wordpress