Now the fun starts! All the parts arrived and it was time to put them on the test bench to burn things in. This is my first ever dual processor build, so it was definitely a learning experience. Nothing is really different, it's just twice as much. When the motherboard arrived, it was absolutely beautiful.
I couldn't wait to get everything put together and get it on the bench. The RAM had arrived a few days earlier and I knew the processors were supposed to be arriving via USPS later that day. I was antsy with anticipation! Then the letter carrier arrived and my CPUs were ready to go into the board.
Now that I've got the processors and RAM installed, lets put on the coolers, get it mounted to the bench, and get it wired up.
Thank God I sprung for the EATX version of this test bench. For those of you that are curious, this is the Highspeed PC Half-Deck Tech Station XL-ATX and it's a great little test bench. There's no metal parts to come in contact with the motherboard, so no worries about shorting things out.
Now that everything was together, it was time for the smoke test. In case you didn't know, computers actually run on smoke. If the smoke escapes, it stops working, and the first POST of a new computer, especially one with an open-box motherboard and CPUs from eBay, is the time when that smoke is most likely to escape. Luckily, this one passed the smoke test.
I slapped the 6x 6TB drives into a carrier and put the LSI 9211-8i HBA in to start burning everything in. I added a USB fan to keep the HBA cool since there wasn't any airflow on that side of the board. The HDD rack has it's own fan. Getting the HBA flashed to the IT firmware was quite the pain, but I'll save that for it's own post.
First up was to run memtest86 and check the 128GB of ECC DDR3. I ran this for a few days to really beat up the memory, as memtest86 runs over and over and over again until you stop it. After the first pass, I knew I was going to be good because there were no errors found, but I let it run for a while just to be safe. I love this picture because it shows 32 CPUs found and 16 started (It's 16 physical cores with hyper-threading for a total of 32)!
After burning in the machine for a while, it was time to transplant it into its permanent home, the Rosewill rackmount chassis. Problem was, there was already a computer in there.
So, I pulled out the old motherboard (which will actually end up being my new gaming rig) to have a fresh case to start with.
After moving some standoffs around, the motherboard fit in perfectly.
My original plan was to use the onboard SAS ports for the six 3TB drives and use the LSI HBA for the six 6TB drives, then use the onboard SATA3 ports for the two SSDs. I ended up using all 8 onboard SAS ports instead. FreeNAS doesn't care what controller the drives are plugged into. I'm not sure if that was a good idea or not, and I plan on looking into it more. If it turns out it is a bad idea, I'll just move all the 6TB drives to the same controller.
Once everything was put together, it was time to boot it up in the chassis for the first time. I hit the power button and... nothing. The fans spun up for a second, then the whole thing shut down. I had no idea what was going on. The first thing that came to mind was the fact that I couldn't find the second CPU power cable for the EVGA power supply, so I "borrowed" one from a Corsair PSU I had. I went ahead and unplugged all the drives to see if maybe something there was shorted and it wasn't. I grabbed the Corsair PSU and plugged it into the second CPU and the computer booted. Ok, maybe it was the cable...
I pulled the EVGA PSU out, put the Corsair PSU in, kinda redid all the cable management, and hit the power button...
Nothing. WTF??? This thing was working fine on the test bench! I did a little more troubleshooting and figured that if it was working fine on the test bench, I'd just go grab that PSU and use it. Out with the Corsair PSU, in with a Rosewill 1000W that I use for the test bench. I hit the power button and... IT'S ALIVE!
The drives are all recognized, FreeNAS boots up without a problem, and we're good to go. My wife actually did the cable management in the chassis because I was fed up with dealing with it. I was originally going to start with a fresh install of FreeNAS, but since it booted up with no issues, I decided to just stick with the current install, though I found out pretty quick that I needed to delete all the tunables created by autotune as they didn't update to the new hardware. My ARC was still limited to 12GB.
The box has been up and running damn good for over a week now, minus a few reboots with me doing stuff.
I built the new volume with the six 6TB drives and started moving some stuff to that new pool.
So, that's the hardware build of my new FreeNAS server. Next, we'll get into the software part of the whole thing. Even though I already have FreeNAS installed and running on this machine, I'll run through the install procedure using another box and we'll get into the meat and potatoes of getting FreeNAS, Plex, and all the Plex Automation setup.
Recently I've run into two clients that were issues with the Cisco 2960G and 2960S switches. Both clients are using PoE versions of the switch for VoIP applications. They were noticing jitter, packet loss and poor call quality, even though QoS is configured on the switch. After a lot of troubleshooting on the voice side of the house, they came to me to see if I could find anything going on. In digging around in the first customer's network, I noticed that the CLI was pretty slow and did a quick "show processes cpu" and saw that the cpu utilization was around 80%. By sorting the processes, I saw that the Hulc LED process was taking up about 15%. A quick search of the Cisco Bug Toolkit brought up Bug ID CSCtg86211 (you need a CCO account to view), even though that's not 100% correct. It's the only one that explained what's going on.
I had the client open a TAC case and TAC wanted to fight with the client, telling them that the high CPU shouldn't have any effect on the switch performance (really!). I suggested that the client upgrade the switches to the latest version of IOS and once that was done, all the voice quality issues disappeared. Total CPU utilization dropped to below 20%, calls cleared up, everything was beautiful.
Last week, I got an email from one of our project managers asking if I could look into an issue that another client was having. If I hadn't known that this was a different client, I would have thought that she had cut and pasted the exact problems that the first client was having. When I found out that they were using 2960's, I immediately thought of this and sent the client a copy of the bug report and told him to open a TAC case. This is the email I received from him:
I tested the CPU Utilization on all of our Cisco 2960Ss and they ranged between 68-99%. I have a test switch on the bench with nothing connected and it was running at 75%. I updated it with the new code and it dropped to the 20-35 % range. I am going to update some additional switches before I call Cisco. The first question they will probably ask is are you running the latest code.
He's right... Cisco will be wanting to know that. I know that once the new IOS is on the switch, it'll solve his problems. I just wanted to put this out there so you guys don't have to do all the searching that I did when/if you run across the same issues on your end.