I've been working on a plan to upgrade my gaming rig lately, especially since I had a 4790K come available with the upgrade of the FreeNAS box. For Christmas, I got a pair of Gigabyte R9-390X's to upgrade my dual HD7970's. I've been running 32GB of RAM in my 4770K box for quite a while now, so there was nowhere to go from there. The plan was to replace the Gigabyte Z87X-OC and i7-4770K with the ASRock Z97 Extreme6 and i7-4790K from the FreeNAS box. The Z97 board has a PCI 3.0 x4 M.2 slot on it, so I wanted to get the speed increase from using it in addition to everything else. But, here's the problem...
The LGA1150 processors, which means ALL the Haswell processors not considered "Enthusiast / High End"; so the Core i3, i5, and i7s with a 4XXX model number or a G3XXX model number; only have 16 PCIe lanes. The Core i7-5820K (6 core) has 28 lanes and the 5930K (6 core) and 5960X (8 core) both have 40 lanes. Now, let's do some math on what I wanted to go into that computer:
- Two R9-390x's - 32 Lanes
- M.2 4x - 4 Lanes
- Thunderbolt 2 AIC - 4 lanes
Well, I'm no mathematician, but I know 40 lanes when I see them. That only left me one option if I wanted to stick with Haswell, and that's to go with the 5930K. The 5960X is still a $1000 processor and I just wasn't going to drop that kind of coin on a CPU. So, today I went ahead and placed the order and here's the new specs for the new gaming rig:
- Intel Core i7-5930K CPU
- Corsair H110i GTX Liquid CPU Cooler
- ASRock X99 Extreme6/3.1 Motherboard
- 32GB Corsair Dominator Platinum DDR4-2666 (4 x 8GB)
- 2x Gigabyte R9-390X 8GB
- ASRock Thunderbolt 2 AIC
- Samsung 950 PRO 512 GB M.2 4x SSD
- Samsung 850 EVO 1TB SATA6 SSD
- Corsair Obsidian 750D Full Tower Chassis
- Corsair AX1200i PSU
- Corsair Professional Blue Individually Sleeved PSU cables
- 5x Corsair SP140 Blue LED Case Fans (2 front, 2 on the radiator, 1 rear)
- Corsair Link Commander Mini
- 2x Corsair RGB Light kit
No spinning platters in this baby! We're going ALL SSD. Spinning platters are for the NAS. The motherboard has dual gigabit NICs, and my network devices all support link aggregation, so I'll be able to get 2 gigabit network access to the NAS, and that should be more than enough for pulling documents or anything else I need off of it. All this will be displayed on my current triple AOC i2769Vm 27" monitors, which give me nearly 6 linear feet of monitor space. Yes, they are only 1080, but I'm not quite ready to dump out the money for three 27" 4K monitors and they don't make ultra-wide curved monitors big enough for me yet.
Today's order was for the CPU, motherboard, and RAM. Everything else is already here. I'm hoping that everything will be here by the end of the week and I'll be able to finish the build this weekend, so I can finish working on the FreeNAS / Plex Automation series.
Here's a sneak peek of what it'll look like. I was mounting the lights and fans in the case last night, and since the X99 Extreme6 and the Z97 Extreme6 look nearly identical, it'll give you an idea of what we'll be dealing with when it's all said and done.
Now the fun starts! All the parts arrived and it was time to put them on the test bench to burn things in. This is my first ever dual processor build, so it was definitely a learning experience. Nothing is really different, it's just twice as much. When the motherboard arrived, it was absolutely beautiful.
I couldn't wait to get everything put together and get it on the bench. The RAM had arrived a few days earlier and I knew the processors were supposed to be arriving via USPS later that day. I was antsy with anticipation! Then the letter carrier arrived and my CPUs were ready to go into the board.
Now that I've got the processors and RAM installed, lets put on the coolers, get it mounted to the bench, and get it wired up.
Thank God I sprung for the EATX version of this test bench. For those of you that are curious, this is the Highspeed PC Half-Deck Tech Station XL-ATX and it's a great little test bench. There's no metal parts to come in contact with the motherboard, so no worries about shorting things out.
Now that everything was together, it was time for the smoke test. In case you didn't know, computers actually run on smoke. If the smoke escapes, it stops working, and the first POST of a new computer, especially one with an open-box motherboard and CPUs from eBay, is the time when that smoke is most likely to escape. Luckily, this one passed the smoke test.
I slapped the 6x 6TB drives into a carrier and put the LSI 9211-8i HBA in to start burning everything in. I added a USB fan to keep the HBA cool since there wasn't any airflow on that side of the board. The HDD rack has it's own fan. Getting the HBA flashed to the IT firmware was quite the pain, but I'll save that for it's own post.
First up was to run memtest86 and check the 128GB of ECC DDR3. I ran this for a few days to really beat up the memory, as memtest86 runs over and over and over again until you stop it. After the first pass, I knew I was going to be good because there were no errors found, but I let it run for a while just to be safe. I love this picture because it shows 32 CPUs found and 16 started (It's 16 physical cores with hyper-threading for a total of 32)!
After burning in the machine for a while, it was time to transplant it into its permanent home, the Rosewill rackmount chassis. Problem was, there was already a computer in there.
So, I pulled out the old motherboard (which will actually end up being my new gaming rig) to have a fresh case to start with.
After moving some standoffs around, the motherboard fit in perfectly.
My original plan was to use the onboard SAS ports for the six 3TB drives and use the LSI HBA for the six 6TB drives, then use the onboard SATA3 ports for the two SSDs. I ended up using all 8 onboard SAS ports instead. FreeNAS doesn't care what controller the drives are plugged into. I'm not sure if that was a good idea or not, and I plan on looking into it more. If it turns out it is a bad idea, I'll just move all the 6TB drives to the same controller.
Once everything was put together, it was time to boot it up in the chassis for the first time. I hit the power button and... nothing. The fans spun up for a second, then the whole thing shut down. I had no idea what was going on. The first thing that came to mind was the fact that I couldn't find the second CPU power cable for the EVGA power supply, so I "borrowed" one from a Corsair PSU I had. I went ahead and unplugged all the drives to see if maybe something there was shorted and it wasn't. I grabbed the Corsair PSU and plugged it into the second CPU and the computer booted. Ok, maybe it was the cable...
I pulled the EVGA PSU out, put the Corsair PSU in, kinda redid all the cable management, and hit the power button...
Nothing. WTF??? This thing was working fine on the test bench! I did a little more troubleshooting and figured that if it was working fine on the test bench, I'd just go grab that PSU and use it. Out with the Corsair PSU, in with a Rosewill 1000W that I use for the test bench. I hit the power button and... IT'S ALIVE!
The drives are all recognized, FreeNAS boots up without a problem, and we're good to go. My wife actually did the cable management in the chassis because I was fed up with dealing with it. I was originally going to start with a fresh install of FreeNAS, but since it booted up with no issues, I decided to just stick with the current install, though I found out pretty quick that I needed to delete all the tunables created by autotune as they didn't update to the new hardware. My ARC was still limited to 12GB.
The box has been up and running damn good for over a week now, minus a few reboots with me doing stuff.
I built the new volume with the six 6TB drives and started moving some stuff to that new pool.
So, that's the hardware build of my new FreeNAS server. Next, we'll get into the software part of the whole thing. Even though I already have FreeNAS installed and running on this machine, I'll run through the install procedure using another box and we'll get into the meat and potatoes of getting FreeNAS, Plex, and all the Plex Automation setup.
WARNING: The hardware specs you are about to read are NOT needed and are complete overkill for a normal FreeNAS build. It is simply me living by the adage of "anything worth doing is worth overdoing." You can find the FreeNAS hardware recommendations in this thread on the FreeNAS forums. I suggest you spend some time doing your own research into what will be best for you and your situation. I've also gotten a lot of heat from folks on the forum for some of my choices. I'll admit that some choices aren't ideal, but I'm also trying to reuse the hardware I already own as much as possible to lower the cost.
This will be the 3rd (actually the 4th) hardware iteration of my FreeNAS server and it's taken me quite some time to decide on what I wanted to be in this build. When I first decided to build a NAS for my home, I wanted to use some of the hardware still laying around from my bitcoin/litecoin/altcoin mining days. I had sold off many of the GPUs, but still had a few different CPU/Motherboard combos that were collecting dust. This video gives you a very small idea of what things were like back then. After taking a quick inventory of what was available, I decided to go with this:
- Intel G3220 CPU
- ASUS Z87-A Motherboard
- 8GB DDR3-1600
- Thermaltake Commander G42 mid-tower case
- 6x WD Red 3TB HDD in RAIDZ2 (~12TB useable storage space)
- EVGA SuperNOVA 1000G2 80+ Gold PSU
The only thing I needed to buy were four of the hard drives since I already had everything else. Two of the WD Reds were sitting in my ESXi host, unused. After I put everything together and got it running, I realized I needed more RAM due to ZFS's use of RAM for ARC. 32 gigs went in. I then realized that the G3220 wasn't powerful enough to handle multiple Plex streams, so I wanted to upgrade it. When I was swapping it for a Core i7-4790K, I bent some pins on the motherboard, so while waiting for a new motherboard to arrive, I put in an AMD FX-4130 CPU and Gigabyte GA-990FXA-UD3 mobo in order to keep things running. That was technically iteration #2, but it was only that way for about 10 days. At the same time I ordered the new motherboard, I also ordered a rackmount chassis for it.
Iteration #3 is what is running currently. Here's those specs:
- Intel Core i7-4790K CPU
- ASRock Z97 Extreme6 Motherboard (bought because it had 10 SATA ports)
- 32GB DDR3-1600
- Rosewill RSV-L4412 Rackmount Chassis
- 2x A-Data Premier Pro SP900 64GB 2.5" SSD
- 6x WD Red 3TB HDD in RAIDZ2 (~12TB useable storage space)
- EVGA SuperNOVA 1000G2 80+ Gold PSU
I moved to SSDs for the boot device because I was having issues with the USB drives constantly getting errors. I had two 64GB SSDs that were purchased for a previous project and ended up not being used, so I threw those in there and I haven't had any errors on my boot devices since. You DO NOT need SSDs for your boot devices. A couple high quality USB drives will be fine. Even though I have those drives in the computer and mirrored, I can't use that space for anything other than the FreeNAS operating system, so it's wasted. As you can see, I'm currently only using 1GB of space.
I have multiple reasons for creating iteration #4 and for picking the parts I ultimately chose.
- I want to be able to consolidate my ESXi host and my FreeNAS server into one unit
- The ESXi host is also running an i7-4790K maxed out with 32GB of RAM.
- Haswell can't handle more than 32GB of RAM and I need more than that to run the VMs currently installed.
- FreeNAS can act as a VirtualBox host. I don't know how well it works, but we'll soon find out.
- I want to be able to handle anything I can throw at Plex.
- Plex uses a lot of CPU when transcoding. The rough guideline is that you need a passmark score of 2000 for each 1080p/10Mbps stream.
- I recently bought a 4K TV, so I want to be able to handle 4K transcoding. If a 1080p/10Mbps stream takes a score of 2000, then a 2160p/40Mbps stream should take about 4 times that, or a Passmark of 8000.
- I want to be able to use this server for a long time.
- I should build something that has enough horsepower that I don't need to build another one in 3 months. That stuff is starting to get old pretty quick.
- Making it last means making sure I can add more CPU and RAM in the future. The i7-4790K's are the most powerful processors I can use with the Z87/Z97 chipset and 32GB is the most Haswell can handle. I can't upgrade further without changing out motherboards and RAM.
- I don't want to have to worry about running out of storage space anytime soon.
With all these things in mind, I spent some time looking into what would not only fit my need, but also be able to use as much of the gear I already have as possible. I knew I was going to have to buy something that can handle ECC RAM and I wanted dual CPUs for the 4K transcoding. So, without further ado, here's the hardware that will be going into Greg's FreeNAS v 4.0:
- Dual Intel Xeon E5-2660 CPUs (used hardware)
- Dual Supermicro 4U Active CPU Heatsink Cooling for X9 UP/DP Systems SNK-P0050AP4
- SuperMicro X9DR3-F Motherboard (open box)
- 128GB (2x 64GB kits of 4x16GB) Kingston KVR16R11D4K4/64 DDR3-1600 Registered ECC RAM
- 2x A-Data Premier Pro SP900 64GB 2.5" SSD*
- 6x WD Red 3TB HDD*
- 6x WD Red 6TB HDD**
- LSI SAS9921-8i HBA**
- 24 port expansion card for the 9921-8i (don't eBay while drinking, kids)**
- Rosewill RSV-L4412 4u chassis*
- EVGA SuperNOVA 1000G2 PSU
** Purchased before I decided on the CPU/Mobo upgrade
Yes, I'm going with dual Xeons and 128GB of RAM. Complete overkill and I love it. The only "new" hardware is the CPU, motherboard, RAM, and coolers. Everything else was already purchased with the idea to upgrade the old server. The parts have already started arriving and should finish getting here next week, which means I'll probably build it out on a test bench on the weekend of January 16th. The plan is to build the new box with the six 6TB and a spare PSU on the test bench, do some testing and burn-in, then move everything into the rackmount chassis. I'll use my current FreeNAS config on the new server, add a new zpool with the 6TB drives and go from there.
The guys on the FreeNAS forums are giving me a hard time about two things, the chassis and the power supply. They really think I should have redundant power supplies in the server, and while I'll probably look into it, I doubt I'll do it. First off, I have redundant power going to that PSU in the form of dual APC SMX1500RM2UNC UPS systems and a Tripp-Lite PDUMH15ATNET Auto Transfer Switch. Secondly, a redundant 600w power supply isn't cheap. Even if you have dual PSUs, you only have 1 motherboard and 1 set of wires from the PSU chassis, not to mention the backplane of that chassis. You still have multiple single points of failure. As far as the chassis is concerned, they think it's garbage. Here's a couple quotes from the forums:
"I do have to say that building out that much of a server and not going with a better case and redundant power seems like dropping the ball."
"Considering the amount of money you're sinking into this, why not just return or resell the Rosewill clunker and find a nice Supermicro 846 or 847 chassis on eBay? It would be a shame to build a Ferrari powertrain and drop it in a Pinto chassis."
Well, they're not paying for all this hardware, and adding redundant power and one of those Supermicro chassis would add another $1,000 to the cost. If I need to add more hard drives in the future, there are ways of doing that. I could use an HP SAS expander and put another chassis with nothing but hard drives in it, or I can get one of those Supermicro cases at that point and transfer all this hardware into it. I just don't foresee needing more than 12 bays. Also, I'm starting to think that Supermicro must secretly pay the people on the FreeNAS forums. Those guys absolutely LOVE Supermicro hardware. It's the only thing they ever want to talk about. The reason I picked the Supermicro X9 motherboard was because I realized it wouldn't be hard to get support for it from the forums. That's something you might want to keep in mind too. If something doesn't work in your build, you'll wish that you had picked hardware common on the forums, otherwise you'll spend a ton of time trying to figure out the problem.
Well, that's where we stand as of today. I'm thinking about documenting the build on YouTube as well as here, so keep your eyes peeled for links, should I decide to do that. In the meantime, head over to the FreeNAS forums and start reading so you can be informed enough to pick out your own hardware. The decisions you make on hardware will be the most important decisions you make with the whole thing. It can be the difference between a relatively quick and painless setup or an absolute nightmare. Whatever you decide, make damn sure it'll support ECC RAM!!!
With the start of a new year, I've decided to start a series on setting up a FreeNAS server at home along with setting up Plex as a media server and various other applications to help automate Plex. By the time I'm done with this series, you'll be able to setup all the following:
- Install FreeNAS on a bare metal server
- Install the following programs to manage your media
- Plex Media Server
- Transmission - A Bittorrent Client
- Sonarr - Automatically downloads TV Shows
- Couchpotato - Automatically downloads Movies
- PlexPy - Provides in depth monitoring and reporting for Plex
- PlexEmail - Send newsletters to your Plex users
- Setup your own domain using dynamic dns from Dyn.org
- Install and configure nginx to work as a reverse proxy and act as a traffic cop for incoming requests
I've been working on this setup for a few months now, and I've done quite a bit of customization to all these different items to make them work. As this series goes on, I'm going to try to recreate what I've done in a VM so I can have screenshots to show you exactly what you should be doing. In the meantime, I'd suggest you start thinking about what hardware you'd like to use for your build.