QoS issues on Cisco 2960′s. (High CPU Utilization)
Recently I've run into two clients that were issues with the Cisco 2960G and 2960S switches. Both clients are using PoE versions of the switch for VoIP applications. They were noticing jitter, packet loss and poor call quality, even though QoS is configured on the switch. After a lot of troubleshooting on the voice side of the house, they came to me to see if I could find anything going on. In digging around in the first customer's network, I noticed that the CLI was pretty slow and did a quick "show processes cpu" and saw that the cpu utilization was around 80%. By sorting the processes, I saw that the Hulc LED process was taking up about 15%. A quick search of the Cisco Bug Toolkit brought up Bug ID CSCtg86211 (you need a CCO account to view), even though that's not 100% correct. It's the only one that explained what's going on.
I had the client open a TAC case and TAC wanted to fight with the client, telling them that the high CPU shouldn't have any effect on the switch performance (really!). I suggested that the client upgrade the switches to the latest version of IOS and once that was done, all the voice quality issues disappeared. Total CPU utilization dropped to below 20%, calls cleared up, everything was beautiful.
Last week, I got an email from one of our project managers asking if I could look into an issue that another client was having. If I hadn't known that this was a different client, I would have thought that she had cut and pasted the exact problems that the first client was having. When I found out that they were using 2960's, I immediately thought of this and sent the client a copy of thev bug report and told him to open a TAC case. This is the email I received from him:
I tested the CPU Utilization on all of our Cisco 2960Ss and they ranged between 68-99%. I have a test switch on the bench with nothing connected and it was running at 75%. I updated it with the new code and it dropped to the 20-35 % range. I am going to update some additional switches before I call Cisco. The first question they will probably ask is are you running the latest code.
He's right... Cisco will be wanting to know that. I know that once the new IOS is on the switch, it'll solve his problems. I just wanted to put this out there so you guys don't have to do all the searching that I did when/if you run across the same issues on your end.
Xbox 360 Open NAT with Cisco ASA 8.3 or newer
I'm just getting on the Xbox 360 bandwagon here, so forgive this post being "late" for most people. But, if you're like me and you're just getting your console, you may have noticed that not everything is going to work properly. To fix this is real simple, and it just following the instructions I posted a while back for port forwarding on 8.3. You'll need to start this off by giving your Xbox 360 a static IP address. This can be done under settings. You'll also need the 3 ports that you're going to have to forward. That's tcp:3074, udp:3074 and udp:88.
Step 1: Create a new object group for your Xbox 360.
asa5505(config)# object network xbox
Step 2: Add the static IP of the Xbox to the network group.
asa5505(config-network-object)# host 10.11.12.13
Step 3: Forward the ports via the NAT command.
asa5505(config-network-object)# nat (inside,outside) static interface service tcp 3074 3074 asa5505(config-network-object)# nat (inside,outside) static interface service udp 3074 3074 asa5505(config-network-object)# nat (inside,outside) static interface service udp 88 88
Step 4: Exit back to the root and add the access lists
asa5505(config)# access-list outside_access_in extended permit tcp any object xbox eq 3074 asa5505(config)# access-list outside_access_in extended permit udp any object xbox eq 3074 asa5505(config)# access-list outside_access_in extended permit udp any object xbox eq 88
That's it! If need to know exactly what's going on here, please check my previous post on port forwarding. If you do your work through the ASDM, just use my post on port forwarding via the ASDM and make the necessary changes to ports, IP addresses and names. Now you just need to go to your Xbox 360 and retest the connection. You'll see that no longer does it say that you are using restricted NAT! Also, if you're having issues with voice over Xbox Live, this should solve those problems.
EDIT 7/12/2011
After further review, I realize that I'm an idiot. You can't add more than one NAT statement to a network object. Create 3 seperate objects (I called them xbox1, xbox2 and xbox3) and add one NAT statement per object. Once you do that, adjust the ACL accordingly.
Preparing for my JNCIA-EX
I think I'll have a little departure from my regular how-to's today. As you can see from the title of this post, I'm preparing to take my JNCIA-EX, or Juniper Networks Certified Internet Associate - Enterprise Switching. I'm sure you all know that I'm a born and bread Cisco kid. Hell, www.ciscokid.net actually points to this site! But I've been spending a lot of time working on Juniper gear lately and it's in my best interest to get some Juniper certs under my belt. Part of my preparation has been trying to stay out of my Cisco lab for a little while. I find that when I spend a lot of time playing with the Cisco gear, I instinctively try to use Cisco commands in a Juniper environment. So, if I don't get into the Cisco stuff for a little while, I'll train myself to think like a Juniper engineer!
Juniper has been a real pain in my ass lately. I absolutely love their hardware. It's some of the fastest gear I've ever worked on and I've never seen the kind of speed to the desktop on Cisco as I have seen on Juniper. It's just that I've spent so much of my career working on one thing that being thrown into something else is not fun. I thought it would be at first; learning something new would give me a challenge and be interesting. Well, it's hasn't been such a great honeymoon phase. The environment I'm working has some issues, and that's about all I'm going to say about it. I'll be going to Herndon, VA for a week to get the training I need to try to solve some of these issues, not to mention another week to attend a data center design class, and I hope that this training will bring me to the level that I can help these people.
I really wish I could speak my mind here and tell you guys about the problems I'm seeing and maybe get some ideas from my readers on how to fix some of this stuff, but I can't. I can't let out any info on the network and I won't talk good or bad about any particular hardware in public. Maybe one day I'll be able to put up a big post with a Visio to see if any of you guys can solve a problem. Hell, I just may create something and give away a prize to the first person to get the correct answer. If anyone has an idea, be sure to let me know!
ASA 8.3(2) is out, as is ASDM 6.3(3)
I got off my ass tonight and decided to update the ASA (yes, my licensing is correct!). I haven't had a chance to play with it a while lot (you can see it's only been up for 3 and a half hours), but it seems pretty cool. I sat down and read the release notes for 8.3(2) and there's a few things you should know. First off, you're going to need more memory. Well, if you look at mine, you can see I've maxed out what the thing will accept, so I'm perfectly safe; but chances are, you're not. Cisco has a nice table to let you know what your memory needs are. In fact, if you head over to this web page, you check out the release notes for both 8.3(1) and 8.3(2). You'll see the new features as well as caveats fixed with these versions. Now that I have informed all you guys about the upgrade, I'm going to start playing with it for a little bit before the sun comes up and kills me. Dammit! It's already 1am!
How to enable AHCI/RAID mode in Windows 7 without reinstalling
I recently got a wild hair up my ass to add a RAID to my desktop. My desktop is a Gateway FX6840-23 and it came with a 1TB drive. I bought an identical drive and thought that I'd put then in RAID 0 for the increased performance, seeing as my Experience Index was only 5.9 due to a slow HDD (all other indexes were in the mid-7's, and the drive is a 7200 RPM unit).
Digging around the BIOS I saw that the SATA controller was using AHCI mode. I cloned my current drive to another 1TB drive I had (yeah, I have 3 -1TB drives, a 500GB, and a 1.5 TB), rebooting into the BIOS and changed it to RAID. After a reboot, I hit ctrl-I and entered the RAID utility. I built the RAID and rebooted. Well, to put it nicely, I got a BSOD. I tried various things for the next 3 hours, including using Windows 7's extended partition utility, doing a complete restore to factory on the extended partition, and everything. After I did the restore, I saw that the HDD performance hadn't changed.
Well, I haven't messed with RAID before on a desktop, so this was a learning experience. After some Google searches, I put the computer back in AHCI mode and booted to the clone. This worked just fine. I went to Gateway's website and downloaded the RAID drivers.
I noticed that the driver was named iaStorV.sys, so I did a search for it and found it already installed in the Windows\System32\Drivers folder. I did a registry search for it and found it in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\iaStorV. This made me happy!
Some more Googling later and I figured out that if I changed the REG_DWORD from 3 to 0 that it would enable things to work. I rebooted the computer, went back into the BIOS and changed the SATA controller to use RAID, pointed it to boot from the clone, and it booted right up! No BSOD, no hiccups, no nothing!
This should work going from IDE mode as well. I tried to clone the clone to the RAID, but Acronis didn't like that too much, so I'm doing a full backup of the clone (I needed to do it anyway) and I'm going to try to restore it with the Acronis Resuce media. It's already midnight, and this is one of those things that I'm not going to be able to put down until I'm done with it. Oh well, I guess it's time to get back to work! Good luck getting your stuff working!
HTC EVO 4G, Froyo 2.2 Final release, Unrevoked3 and ROOT
Sprint and HTC released Froyo today as an over-the-air update to all of you out there that haven't already rooted your phone. Well, my EVO was rooted within the first 48 hours of me owning the phone. I recently dropped back to the stock unrooted image only to use Unrevoked3 to root it again.
By having a rooted phone, I can't get the Froyo update over the air, but I can still put it on my phone, and you can too. Here's how to do it.
Step 0: I'm adding this late, but if you don't want to have to search for your apps again, check out AppBrain. Install it before the upgrade, sync your phone up with the website, create a new list on the website and copy all your apps to that as a backup. After you are done with the upgrade, install AppBrain first and sync it back up. All your apps will be there waiting for you.
Step 1: You need to have a phone rooted with Unrevoked3. That's the way I know how to go about it, so go root your phone with Unrevoked3 and come back.
Step 2: Download the rooted Froyo image from here. (Thanks netarchy!!!)
Step 3: Copy the .zip file to the root of your SD card on the phone.
Step 4: Power down the phone and reboot into the boot loader. To do this, make sure the power is off and hold the volume down button while holding down the power button.
Step 5: Once in the boot loader, use the volume rocker to navigate to RECOVERY. Select it by pressing the power button. This will bring you to Clockwork.
Step 6: From ClockworkMod Recovery, select " wipe data/factory reset". Confirm it and do the reset.
Step 7: Select "install zip from sdcard", then "choose zip from sdcard".
Step 8: Navigate to and select "HTC-OTA-3.26.651.6-Final-Froyo-Rooted-Odexed-netarchy-signed.zip". Confirm the install.
Step 9: Once the install is done, navigate back to the top level of Clockwork and select "reboot system now".
And you're done! You should now reboot into Froyo. You can verify this by going to Settings -> About Phone -> Software information. You should see "Android Version 2.2" at the very top!
Once you have updated Froyo, you need to apply a couple of radio updates. You do this the same way you installed the Froyo zip file. After each install, reboot the phone and let it do a full reboot. After it's rebooted, power down and boot into the boot loader to apply the other one. Follow the instructions above (you don't have to wipe data for the radio updates) and you'll be just fine. Here are those two files:
Radio update: 2.15.00.07.28
Wimax Update 26023
Enjoy the speed of Froyo and have fun with your newly rooted phone! For the full thread on xda, click here.
-Greg
Update: I've seen a HUGE spike in traffic in the last hour (that's to be expected) and I can see that a lot of you guys are doing the upgrade (I love Woopra). Leave some comments below and let me know if everything ran OK for you.
Port Forwarding on the Cisco ASA in 8.3 from the ASDM made easy
In my last post I taught you how to forward a port on the ASA 5505 running version 8.3 from the CLI. Some of you prefer to use the ASDM to do you changes, so I guess I'll show you how to do it from there. The ASDM is a bit of a learning curve for someone that's used to the CLI, and most CLI guys hate a GUI with a great passion. I can go either way. I use the ASDM to make some changes simply because I want to learn it and there's some guys coming into the field today that were taught on the GUI rather than a command line.
In this lesson I'm using ASDM version 6.3(1) and ASA version 8.3(1). Since we added a web server in the last post, let's make this one an FTP server. The FTP server's IP is the same as the web server, 10.9.8.7/24 and we're running over the standard FTP port, 21.
First off, we want to start up the ASDM and connect to the ASA. Once there, click on the 
button at the top of the screen, then the 
button near the bottom left, and finally select 
near the top left. You'll now be at a screen that looks something like this:
Click for larger version
Now we need to create a new object, so click on "Add" under Addresses, then "Network Object".
Now we need to fill out our new window. Once you fill out the name, IP address and description, you need to drop down the NAT box and fill it out. Click the "Add Automatic Address Translation Rules" box, leave the type as "static" and set the translated address as the outside interface.
We now need to go to the Advanced menu from the Add Network Object window and setup the port forwarding. The source will be inside, destination is outside. Protocol in this instance is TCP and our port is 21, both real and mapped.
Click "OK" twice and your object will be created as well as the port forward. Now we just need to add the access rule. On the left side of the screen, just above the NAT Rules is your Access Rules. From there we want to click "Add" and "Access Rule".
We need to create the rule on the outside interface, coming from any IP to the FTPServer using FTP as the service.
Once you click OK, your rule is added. You don't have to add a description like I did in the image above this one, I just did that for the hell of it. When you click "Apply" at the bottom of the screen, the ASDM will issue the commands to the ASA. I have preview turned on, so I can always see what commands are being sent to the device before they are actually sent. If you followed all the steps above and you have preview turned on, you'll see the following:
And you'll notice that those are the exact 4 commands that I gave in the last post about doing it from the CLI! Now you can forward any port you want from either the CLI or the ASDM!
On a side note, I know a lot of guys hate the ASDM. When I was writing this post and going through all of this I was kinda upset when I saw that I had 10 pictures for 4 lines of code. The good thing about the ASDM is that you have everything right there at your disposal and you really don't need to know the vernacular of IOS. The drawback is that it will take you longer to get things done at first, but once you get used to it, it can be just as fast.
Port Forwarding on the Cisco ASA in 8.3 from the CLI made easy
So it's been a month and a half since I posted an update, and it's 4:15 am right now. I can't sleep and I found out there's another networking blog out there using the same WP theme as me, so I figured I better put something up here since it was fresh in my mind. Well, now that the niceties are out of the way, let's get to work.
I recently added an ASA 5505 to my home network at the edge. Obviously, when I did, all of my port forwards went to hell because the ASA is now blocking everything. I run a web server on one of my servers here and I like to be able to access it because I keep a lot of tech manuals and other stuff on there. Well, I went about trying to set up port forwarding the old way and learned real quick that this pops up when I do:
ERROR: This syntax of nat command has been deprecated.
Please refer to "help nat" command for more details.
Yeah, that sucks. On the new version of the ASA OS, global has gone the way of the dodo. I did a bunch of searches on Google to figure it out and everything I ran across was very hard to decipher. That's why I'm writing this. You can setup a port forward in 4 quick and easy steps. Just change the things that are underlined to fit your network and you'll be just fine.
In this example, we want to be able to access a web server behind the firewall. We'll assume you are using the standard HTTP port, the web server's internal IP address is 10.9.8.7/24, and that you at least know what you're doing enough to be configuring an ASA in the first place. I'll give you the steps, then I'll explain.
Step 1: Create a new object group for you web server.
asa5505(config)# object network Webserver
Step 2: Add the IP of the web server to the network group.
asa5505(config-network-object)# host 10.9.8.7
Step 3: Forward the port via the NAT command.
asa5505(config-network-object)# nat (inside,outside) static interface service tcp www www
Step 4: Exit back to the root and add the access list
asa5505(config)# access-list outside_access_in permit tcp any object Webserver eq www
That's it! Now, let's explain what's going on here. Cisco has started moving more and more towards use of object groups in their configs. It makes things easier, especially when you have a situation where you have 20 web servers behind the firewall and you want to add 1 more in. Rather than having to rewrite a whole bunch of ACL's, you just add the IP of the new web server into the object group and everything is done for you. After you create the object group (in this instance a network object, you can also create service objects), you add the IP of the specific object (or objects) that you want to point to. So here our web server is 10.9.8.7. If you want to send port 80 to more than 1 IP on your internal network, just add more IP's to that object group.
Now is the fun part. While we're in the object group, we need to NAT port 80 only to that specific object group, hence you're still at "asa5505(config-network-object)#" prompt. Now type "end" to get back to the regular config terminal and we need to open that port in the ACL. Yes, www = 80. You can type either one and you get the same result. If I have to go through and explain NAT, how it works and why I enter in that specific command to forward the port, then there's a possibility that I'd need to send you an invoice for my time because we would be here for a while.
This works for ANY port forward. If you want to RDP into a machine, simply replace port 80 (all those www's you see up there) with 3389. There is one caveat. You can only do one port forward per object group. So let's say that our web server is also an FTP server and you want port 21 to forward as well as port 80. You're going to have to create a whole new object group (object network FTPServer), put the same IP in the group (host 10.9.8.7), do the nat command again (nat (inside,outside) static interface service tcp ftp ftp), exit back to the root of config, and add the access list (access-list outside_access_in permit tcp any object FTPServer eq ftp).
This should get you up and running with you port forwards in no time flat. It is a bit of a pain in the ass to have to create a new object group for every port you want to forward, and maybe there's someone out there that's reading this right now thinking "dude, you don't have to create more than one group! You can just do...". Well, you need to enlighten the world with this knowledge and post it in the comments section. And if you're too scared to do so, shoot me an email to greg(at)gregledet(dot)net.
I'd also like to thank Stefan Fouant for an excellent class today on JUNOS Switching. I learned a lot in his class and you can learn a lot from his website. Check it out and tell him Greg sent ya!
HTC EVO 4G – A week later
I wanted to hold off writing my review of my new phone because I wanted to spend a week with it and learn all about it. I got the HTC EVO 4G on launch day and I immediately fell in love with it. I'm not the only one either, because I ended up ordering my wife one 2 days later.
So far the phone is amazing. Yes, I will admit that there is a bit of a battery life issue, but with a screen this size and the fact that you're going to spend every extra second playing with the damn thing, you're going to burn through some battery. You can get 2 spare batteries and a charger on eBay for about $11, so it's no big deal to carry an extra in your pocket.
Android and HTC Sense are amazing. With the 1Ghz Snapdragon processor, everything runs like a bat out of hell. In fact, I can't say anything bad about the software on this phone. It's very intuitive to use and you pick things up very quickly. It took me all of 3 days to figure everything out, but I'm an experienced user. When it only took my wife the same time to figure the phone out, I knew it wasn't just me. The phone is that easy to use.
My wife told me "I've never been excited about a cell phone before, but I am out this". I feel the same way. Going from the Treo Pro to this is like going from a Yugo to a Ferrari. It's that good. The camera puts out some excellent pictures and now that Qik is up and running, the video calls are really good. I've also tested video calling on Fring with my friend Jef and it seems to work great too. Everything on this phone is snappy and beautiful. From what I've seen about the specs on the new iPhone 4, the EVO 4G should kick it's ass in everything except resolution.
If you're on the fence about getting an EVO, I'd say go for it. You won't be disappointed.
Oil Spill ROV cameras
I've put all the ROV cameras in one place.