GregLedet.net Adventures in networking, security, and other things

QoS issues on Cisco 2960′s. (High CPU Utilization)

Recently I've run into two clients that were issues with the Cisco 2960G and 2960S switches.  Both clients are using PoE versions of the switch for VoIP applications.  They were noticing jitter, packet loss and poor call quality, even though QoS is configured on the switch.  After a lot of troubleshooting on the voice side of the house, they came to me to see if I could find anything going on.  In digging around in the first customer's network, I noticed that the CLI was pretty slow and did a quick "show processes cpu" and saw that the cpu utilization was around 80%.  By sorting the processes, I saw that the Hulc LED process was taking up about 15%.  A quick search of the Cisco Bug Toolkit brought up Bug ID CSCtg86211 (you need a CCO account to view), even though that's not 100% correct.  It's the only one that explained what's going on.

I had the client open a TAC case and TAC wanted to fight with the client, telling them that the high CPU shouldn't have any effect on the switch performance (really!).  I suggested that the client upgrade the switches to the latest version of IOS and once that was done, all the voice quality issues disappeared.  Total CPU utilization dropped to below 20%, calls cleared up, everything was beautiful.

Last week, I got an email from one of our project managers asking if I could look into an issue that another client was having.  If I hadn't known that this was a different client, I would have thought that she had cut and pasted the exact problems that the first client was having.  When I found out that they were using 2960's, I immediately thought of this and sent the client a copy of thev bug report and told him to open a TAC case.  This is the email I received from him:

I tested the CPU Utilization on all of our Cisco 2960Ss and they ranged between 68-99%.  I have a test switch on the bench with nothing connected and it was running at 75%.  I updated it with the new code and it dropped to the 20-35 % range.  I am going to update some additional switches before I call Cisco.  The first question they will probably ask is are you running the latest code.

He's right... Cisco will be wanting to know that.  I know that once the new IOS is on the switch, it'll solve his problems.  I just wanted to put this out there so you guys don't have to do all the searching that I did when/if you run across the same issues on your end.