QoS issues on Cisco 2960′s. (High CPU Utilization)
Recently I've run into two clients that were issues with the Cisco 2960G and 2960S switches. Both clients are using PoE versions of the switch for VoIP applications. They were noticing jitter, packet loss and poor call quality, even though QoS is configured on the switch. After a lot of troubleshooting on the voice side of the house, they came to me to see if I could find anything going on. In digging around in the first customer's network, I noticed that the CLI was pretty slow and did a quick "show processes cpu" and saw that the cpu utilization was around 80%. By sorting the processes, I saw that the Hulc LED process was taking up about 15%. A quick search of the Cisco Bug Toolkit brought up Bug ID CSCtg86211 (you need a CCO account to view), even though that's not 100% correct. It's the only one that explained what's going on.
I had the client open a TAC case and TAC wanted to fight with the client, telling them that the high CPU shouldn't have any effect on the switch performance (really!). I suggested that the client upgrade the switches to the latest version of IOS and once that was done, all the voice quality issues disappeared. Total CPU utilization dropped to below 20%, calls cleared up, everything was beautiful.
Last week, I got an email from one of our project managers asking if I could look into an issue that another client was having. If I hadn't known that this was a different client, I would have thought that she had cut and pasted the exact problems that the first client was having. When I found out that they were using 2960's, I immediately thought of this and sent the client a copy of thev bug report and told him to open a TAC case. This is the email I received from him:
I tested the CPU Utilization on all of our Cisco 2960Ss and they ranged between 68-99%. I have a test switch on the bench with nothing connected and it was running at 75%. I updated it with the new code and it dropped to the 20-35 % range. I am going to update some additional switches before I call Cisco. The first question they will probably ask is are you running the latest code.
He's right... Cisco will be wanting to know that. I know that once the new IOS is on the switch, it'll solve his problems. I just wanted to put this out there so you guys don't have to do all the searching that I did when/if you run across the same issues on your end.
Google Voice!
I've finally received my Google Voice invite! So far, this is cool as hell. I just signed up tonight, but I'm loving it. I even got a VERY cool number! Drop me a line!
DD-WRT? In my WRT54G2? It’s more likely than you think!
Happy Thanksgiving kiddies! I've decided to put together a little how-to for the home users that may be throwing around the idea of upgrading the firmware on their WRT54G2 to DD-WRT. "But the WRT54G2 isn't supported by DD-WRT yet Greg!". Well, that's not exactly true. Follow these simple instructions and you'll have it done in less than 10 minutes.
You'll need the following files:
Linksys TFTP utility
VxWorks Prep
VxWorks Killer
DD-WRT Firware
I've put them all here for your convenience.
Here is another How-to with a little more information that is based off this one.
Get and install Linksys tftp.exe, set your PC to static IP, 192.168.1.10.
1. Reset the router to defaults on the Linksys Admin page, and let it reboot or manually reboot it after its finished.
2. Set your computer to a static IP of 192.168.1.10/24 and plug Ethernet cable into one of the LAN ports on the router.
3. Close all your browser windows. Start the tftp utility, set server to 192.168.1.1. Password is "admin" and browse to the VxWorksPrep-G2V1.bin file. Click Upgrade. Wait a minute for it to reboot on its own, if it doesn't, then power cycle the router manually.
4. In the tftp utility, browse to the VxWorksKiller-G2V1.bin file and click Upgrade. Wait 2 minutes for it to reboot on its own, if it doesn't, then power cycle the router manually.
5. Tftp the DD-WRT firmware to the router, use "dd-wrt.v24-10709_NEWD_micro.bin"; after successful tftp, wait 3 min for the router to finish writing new nvram defaults, etc... It should reboot on it's own at least two times, so give it the 3 min and then open a browser to http://192.168.1.1 (If it not reboots on its own, wait another 1 min, and then power cycle it).
6. When if finishes booting up...do a hard reset on the unit...let it boot again, and configure.
That's it! Now you have a lot more control over your WRT54G2 v1!
Install MagicJack VoIP on Server 2003
I've been going without a land line for a few years now and it's starting to get old. Because of the plan that my wife and I are on with Sprint, it gets rather expensive when I start going over my minutes. And I sure as hell didn't want to fork out the kind of money that the phone and cable companies charge. I'm a Cisco Engineer for Christ's Sake! I should be able to set this up!
So I got my hands on the Cisco uBR924 you see in the rack. It uses H.323, but I couldn't find a reliable H.323 provider to give me a number. And I'm lazy. I'm sure I'll get around to using the H.323 in the modem sooner or later, but I wanted to try this product out anyway. I ended up ordering a MagicJack.
Well, MagicJack doesn't support Server 2003. But the only box that I have that stays on 24/7 is my server. And my server runs... you guessed it... Server 2003. My install went a little like this.
1. Plug the MagicJack into a USB port and let the drivers install.
2. Once the install runs, go to "My Computer" and run the Autorun on the MagicJack drive
3. Let MagicJack install (it downloads its software)
4. Get the message "No audio devices found no output/input devices are found".
5. Curse loudly at computer and say something along the lines of "Oh, you are GOING to work..."
Because I'm an idiot and didn't realize what I was doing, it was flat NOT going to work. The reason being is the only sound driver running was the Microsoft RDP Sound Driver. My server is headless, therefore I needed to be RDP'd into the box. I'm going to go ahead and make a long story short...
I plugged a keyboard into the box and logged in locally. I tried to "mstsc /v:server /console", but it still had the RDP driver. I haven't looked into it, but there may be someway to use the local drivers during that console session. Once I was logged in locally, I shadowed the local session from an RDP session. I turned on the Telephony Service, installed the sound drivers (they weren't installed), and started the Windows Audio service. Once that was done I restarted the MagicJack software and BAM! It worked.
***EDIT***
I just found out why the "/console" wasn't working when I was RDPing into the box. It seems that Microsoft changed /console to /admin in Vista for you to login to session 0. Here's the correct way to do it. In XP SP2, you want to run "
In Windows XP SP3, Windows Vista or Windows 7, run %systemroot%\system32\mstsc.exe /admin
If you are using Windows XP < SP3 the command is:%systemroot%\system32\mstsc.exe /console
Once you get the client up, you want to make sure that under the "Local Resources" tab you have the audio options set to "Leave at remote computer". This should fix that damn "No audio devices found no output/input devices are found" problem without having to plug a monitor and all into your server.
From what I've seen so far using WireShark, this is a simple SIP device that runs the G.711 Codec. Pretty straight forward. I'll mess with it some more and get back to you guys.
I shall call it “Mini-Lab”
I felt like putting something together here at the house to replace what I lost in the hurricanes. This may have been a Hurricane blog for a short time, but first and foremost it's a Network blog. I now present to you... Mini-Lab.
The 4500M actually has 2 NP-6E's in it (6 Ethernet Ports each). It's a work in progress, and I'll defidently be adding on to it, but this is just what I could get my hands on at the present time. Hell, I may even open it op to some of you guys that are working on their CCNA's and stuff to give you a little rack time. Just to break it down, here's what you're looking at:
Cisco uBR924 Cable Modem / Router. Actually supports H.323 voice too
2 x Cisco Catalyst 2924XL-EN switches. Need to move packets somehow!
Cisco 2620 Router. With WIC-1DSU-T1.
Cisco 3620 Router. With 2E-2W and WIC-1DSU-T1
Cisco 4500M Router. With 2 NP-6E's. We used these when I was working for an ISP. Real monsters back in the day
Like I said, it's a work in progress. I hope to really build this rack up and make it quite the monster. I'm looking into getting a 2821 to do VoIP with along with a 3560-12 PoE and a couple 7690's. It should be sweet!

