March 23rd, 2008The Internet? Bah! (looking back to 1995)
I’ve been on Spring Break with the family for a week and I’m trying to catch back up… While poking around in the events of the week I came across this on Digg.com that really made my day. Check out this Newsweek article from 1995 on the Internet. I imagine you’ll be entertained as well.
March 12th, 2008Interesting details on iPhone Exchange ActiveSync Implementation
There aren’t a whole lot of details floating around about Apple’s implementation of ActiveSync on the iPhone yet, but I stumbled across an article from last week that I missed on the Exchange Team Blog that had a few surprise details.
Obviously, we have been working together for a while on their implementation. What can we say today? They are doing Direct Push and most of the Exchange 2003 SP2 policies (including remote wipe). They are doing email, calendar and contacts sync, and global address lists. While previously you could get you Exchange mail on an iPhone via IMAP, getting contacts and calendar required a tethered sync through iTunes. Doing it wirelessly will be much better (IMHO).
Particularly noteworthy, Apple will implement a couple key Exchange 2007 EAS features.
- Autodiscover – This means those of you running Exchange 2007 can now make it super easy for your users to configure their iPhones to sync with Exchange. Here’s what you do as an admin. All the iPhone user does is enter his/her email address and password. Pretty cool.
- HTML Mail – See your mail in its full HTML glory. Obviously the iPhone shows mail in HTML format today, so it’s safe to assume your Exchange mail will retain its HTML formatting on the iPhone as well.
I don’t have a lot more to say about the iPhone. Perhaps after I’m worked with it a bit I’ll share some more thoughts.
More of the Article can be found here.
March 12th, 2008KB948496 hits WSUS, Windows Update, and Automatic Updates
We’ve awoken this morning to most of our servers requesting to install the update which disables the Scalable Networking Pack. Surprisingly Microsoft has decided to push this update out globally. Since MS has decided it was a bad idea to turn on SNP by default, this is probably the end of the issue. Capable drivers or not, with it disabled everything should return to normal. It will, however, be very interesting to see what the fallout from this update is.
March 11th, 2008HP Drivers Updated for SNP Improvements
I did a little digging tonight and it looks like HP has updated the drivers for their on-board Broadcom NICs. The updates were posted on February 28th, 2008 and can be found here. The link is for the DL380 G4 but should apply to multiple servers based upon the release notes.
Based upon the driver version they are deploying this may or may not still leave you with Vista issues. This remains to be seen.
March 11th, 2008Update to the Scalable Network Pack Issues
So after nearly 8 months of battling this ongoing issue Microsoft has decided to speak up and acknowledge it. Today they published KB 948496 titled “An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers” which addresses this very issue. I’m certain that the work of the Exchange team had a lot to do with the final recognition of this issue so much thanks to them!
To summarize, Microsoft has acknowledged that computers which have TOE capable network cards and have SP2 installed may face issues due to the Scalable Network Pack being enabled by default. This is due, primarily, to faulty drivers but as I stated in my previous blog entry even the most recent drivers direct from Broadcom do not address all of the issues. To address this problem they have released an update EXE which disables the Scalable Network Pack. Additionally, they provide methods to manually disable these features in the article.
While this is a major battle won in the war on this issue we cannot consider it complete won until Broadcom releases a driver that fully fixes the issue and Dell/HP/Etc. certify that driver for their customers.
March 6th, 2008Windows 2003 Scalable Networking pack and NIC Drivers
In late August to early September of 2007 we had a sudden breakout of issues with brand new servers. We found that we were installing brand new Dell PowerEdge 2900/2950/1900/1950 units that were having significant Network Performance issues and Exchange performance problems.
The first symptom that was noticed was network throughput. All of these servers were plugged into 1000BaseTX networks with clients running at 1000BaseTX or 100BaseT. Throughput varied from test to test, but overall speeds for CIFS/SMB file copy operations were at speeds that would be slow on a 10Mbit network. Various troubleshooting was attempted with Dell and various switch manufacturers and we found that putting the servers on a 100Mbit port resulting in full 100Mbit throughput. Various combinations of Flow Control on switch ports seemed to help but this was not a usable permanent solution.
Exchange MAPI Errors
A second symptom cropped up almost immediately after this one where certain users were unable to open their Exchange 2003/2007 mailbox through Outlook yet were able to open it through OWA or sync with their mobile devices. Profiles were re-created and attempts made on other desktops but this did not fix the issue. The problem appeared to be tied to the user and MAPI. A scan of the event logs on the server showed multiple errors related to MAPI. (9646 Errors)
Stumbling onto a Clue
In late October of 2007 Dell shipped out their October edition of the Server Updates disc in their OpenManage updates pack. Luckily a few of my clients subscribed to this and I noticed it come in. Out of desperation we popped the update disc into one of the servers that was giving us problems and found that nearly everything was in need of updating. (This was a bit of a surprise as this particular server was only 6 weeks old at this point.) Immediately after updating all of the drivers/firmware the first problem appeared to be completely resolved. We were now able to max out throughput over the Broadcom gigabit cards in the servers, even running a bonded pair. Something definitely changed.
Upon further testing we found that the fix came from the updated network drivers dated 10/20/2007 on Dell’s support site. The actual driver installed by this update was version 18.104.22.168 dated 7/27/2007, way earlier than the 10/20 date of release from Dell. (One can only assume Dell takes a long time to QA their drivers.)
Exchange Issues Continue
Unfortunately, the updated driver did not resolve our Exchange issues. Being forced to dig further into the issue we stumbled across a post on the Exchange Team Blog detailing problems caused by network drivers (particularly Broadcom) and the Windows Scalable Networking Pack which is installed and enabled by default in Windows 2003 Service Pack 2.
The blog post goes fairly extensively into the problem, however to summarize it appears that certain network vendors have been extremely slow to properly implement the features required by the Scalable Network Pack in their drivers. The key word in that sentence is “properly” as the pack tries to turn it on anyway resulting in serious issues. At best this causes performance issues/limitations which may not be noticed at all. For us this problem was causing severe limitation on the TCP stack that presented itself in many ways. The following are a few of the issues that were affecting us that we have later found to be a direct result of the problem.
- MAPI errors resulting in limitations on the number of MAPI clients that could connect
- Once the limit was reached no further MAPI sessions could be created on the server
- Users unable to open their mailbox through Outlook
- Randomly unable to make RDP connections to the server or connections dropped immediately after connecting
- Intermittent RPC communications failures
- Networking throughput is severely decreased (This was partially fixed by the October Dell driver update)
Unfortunately, the driver provided by Dell is far too old to address these problems. Per the Microsoft blog posting the Broadcom driver needs to be at least 3.7.19 to fully support the Scalable Network Pack. This leaves us in a tight spot to address the performance issues. We are left with two options.
- Install unsupported drivers directly from the chipset manufacturer
- Addresses the source of the problem
- Potentially introduces support issues with your server vendor
- Disable the Scalable Network Pack
- Disables key improvements brought by SP2 as a workaround
- Should make note of this and re-enable these features when drivers are later updated
Obviously it is in the best interest of our customers to stay supportable so we have chosen to address this by disabling the features of the Scalable Network Pack that are affecting performance on our servers. The easiest way to do this is by issuing the following command at a command prompt.
Netsh int ip set chimney DISABLED
Further details can be found on the Exchange Team Blog posting and I highly recommend you read it. This fix has addressed the key problems we experienced without our customers’ file servers and Exchange servers but we are keeping a close watch on this hoping for a permanent resolution. We have tested the latest drivers directly from Broadcom on test servers and have found that while it addresses these immediate issue we continue to have network throughput issues with Vista client systems under certain circumstances.
While this incident has severely questioned my trust in Broadcom I have to admit that they are not the only vendor having problems. While Broadcom seems to be the biggest offender here we have seen throughput issues (particularly with Vista clients) on servers using Intel NICs. Most of these are Dell and HP servers which haven’t received vendor driver updates in several years. A driver update from Intel instead of the Dell/HP/etc. site appears to resolve the problem immediately.
I see this problem as one so severe vendors should be calling their customers to address it yet if you call Dell/HP support they don’t even have a clue what you are talking about. Hopefully this post has provided you some insight into the problem. We are still working on this internally and hope to find a permanent solution. I will update with another post when we do.