GregLedet.net That dude writing stuff about that thing.

23Feb/200

Building a Stratum 1 NTP Server with a Raspberry Pi 4 and Adafruit Ultimate GPS Hat

I'm going to be honest with you. There's a lot of posts on the internet on how to do this, but there's a lot of misinformation out there to. My goal in this post is to give you what you need to get this setup, explain why you need to do what you need to do, and give you the tools you'll need to to study up if you want to know more. In the past week, I've been troubleshooting a GPS hat and I've built an NTP server on a fresh Raspbian build at least 5 times to perfect this, and yesterday I ran through it again writing down the steps as I went. That's where this how-to came from.

There's two trains of thought on how to do this post-RPi2; software UART or hardware UART. Software UART is if you still want to use bluetooth on the RPi. I don't know why you would want to use bluetooth on your NTP server, but some folks might want to. This will not be for them. I'm in the hardware UART crowd.

The Ultimate GPS hat delivers its data over a serial port to GPIO pins 14 and 15 at 9600 bps. On the RPi2, this went directly to the hardware UART, but on the RPi 3 and 4, the hardware UART is taken up by the bluetooth subsystem, and the serial port for pins 14 and 15 is emulated in software. Luckily, we can disable bluetooth and use the hardware UART for the GPS. So, let's get started.

Parts List and Cost:

Raspberry Pi 4 2GB: $34.99 (The 1GB version will work fine, I just had the 2GB version laying around)
Adafruit Industries Ultimate GPS HAT: $44.99
Waterproof GPS Active Antenna 28dB Gain: $14.99 (Optional, but you'll really need this for best results)
Raspberry Pi 4 Power Supply with ON/Off Switch: $9:99
SanDisk Ultra 32GB microSDHC: $7.99
Total Cost: $112.95 (you can usually find things cheaper too). Compare this to commercial NTP servers that range from $1500 to $5000+

I'm going to assume that this is a fresh build and you're doing this headless and over wifi. I also assume you're using Raspbian. The build I used for this how-to was 2020-02-13-raspbian-buster-lite, but unless something major changes with Raspbian, it should work with any build of Raspbian. I also use apt rather than apt-get because I like the little progress bar at the bottom of the screen, but you can use apt-get if you like. I also use nano as my editor. If you're one of those people that's into self harm and you want to use vi, more power to you. Without further ado...

1. Burn the Raspbian image to an SD card
2. Add a blank ssh file and your wpa_supplicant.conf to /boot. (standard stuff for headless RPi access)
3. Put a battery in the hat, attach the hat, plug in your SD card, and either attach an external active GPS antenna or just put the whole RPi outside and power it up.

When I was writing out the instructions for this yesterday, I just plugged my RPi4 in and set it on my back patio for about 30 minutes. You really can't do the later steps until the GPS has a fix. You'll know it has a fix when the red LED stops blinking once per second and starts giving you a brief flash every 15 seconds.

While we're waiting for a fix, go ahead and ssh to raspberrypi.local using your favorite terminal program (I'm on a Mac, so I use terminal, if you're on Windows, I recommend Putty). Let's get this thing updated while waiting...

sudo apt update
sudo apt upgrade

Once we have a GPS fix, we'll move forward. The first thing we want to do is disable the console getty programs. We'll be wanting to use /dev/ttyAMA0, and they're currently using them. While we're at it, we're also going to disable the hciuart service, as it usually attempts to talk to the UART.

sudo systemctl stop serial-getty@ttyAMA0.service
sudo systemctl disable serial-getty@ttyAMA0.service
sudo systemctl disable hciuart

Even though we've stopped the console from starting, we need to stop the kernel from trying to use it. We edit /boot/cmdline.txt and remove the console.

sudo nano /boot/cmdline.txt
remove this: console=serial0,115200

Now we'll need to actually disable bluetooth and take over the hardware UART. This will allow us to use /dev/ttyAMA0 for our GPS. While we're in here, we're also going to enable the PPS pin, which is GPIO pin 4 and disable power saving. We're just doing this now so we don't have to reboot again later.

sudo nano /boot/config.txt
#At the bottom of the file, add the following:

# Use the /dev/ttyAMA0 UART GPS instead of Bluetooth
dtoverlay=disable-bt

# enable GPS PPS
dtoverlay=pps-gpio,gpiopin=4

# Disable power saving
nohz=off

We also need to clean up one more thing before we move on. DHCP can be configured to deliver NTP server info on some networks, but that doesn't work very well with NTP servers themselves. We want to make sure that this doesn't interfere with us, so we'll disable it. If we don't, it could cause your ntp.conf file to be edited or ignored completely.

sudo rm /etc/dhcp/dhclient-exit-hooks.d/ntp
sudo rm /lib/dhcpcd/dhcpcd-hooks/50-ntp.conf

sudo nano /etc/dhcp/dhclient.conf
In the "request" block, remove dhcp6.sntp-servers and ntp-servers

Delete the highlighted options

Finally, we want to change our CPU scaling governor settings to keep the CPU set to the maximum speed for continuous usage. Normally enabling power saving features is a good thing: it saves you power. But when your CPU changes power saving modes, the impact on PPS timing is noticeable. For some reason the NO_HZ kernel mode has a similar bad effect on timekeeping. We disabled nohz earlier in the /boot/config.txt file and to change the scaling governor we need to do the following:

sudo nano /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
replace ondemand with performance

Now that we've edited those files, removed the DHCP configurations, and set our performance level, we need to reboot.

sudo reboot

Give it a couple minutes, then SSH back in and let's check to see if we have communication from the GPS on /dev/ttyAMA0

sudo cat /dev/ttyAMA0

You should see something like this:

sudo cat /dev/ttyAMA0

Great, we have communication between the RPi and the GPS Hat! Awesome! Now, let's add some tools to make this whole thing work.

sudo apt install gpsd gpsd-clients python-gps pps-tools ntp

GPSD is the service we're going to use to decode the NMEA data coming from the GPS. Before it will work, we need to edit it's configuration file. You'll want the options to match the options below. We're not using USB GPS, so we can turn that off, the devices are the /dev/ttyAMA0 that is the UART we stole from bluetooth, and the /dev/pps0 that we requested earlier for pin 4 in the /boot/config.txt file, and the -n option tells GPSD to start talking to the GPS device on startup and not to wait for the first client to attach.

sudo nano /etc/default/gpsd

START_DAEMON="true"
USBAUTO="false"
DEVICES="/dev/ttyAMA0 /dev/pps0"
GPSD_OPTIONS="-n"

After we save the config file, we need to restart the gpsd service so it can pick up the config.

sudo systemctl restart gpsd

Once gpsd is restarted, we'll run gpsmon to see how we're looking.

gpsmon

You should see something like this:

gpsmon

YAY! Your GPS is now passing data and GPSD is processing that data properly, but this is only half the battle. You should see PPS offsets in the gpsmon window, but to verify we have good communication on /dev/pps0, we run the following command.

sudo ppstest /dev/pps0

The output should look something like this:

sudo ppstest /dev/pps0

Great, we have a good PPS signal. The GPS is working, PPS is working, now all that's left to do is to edit the ntp.conf file and add some pretty important stuff. Before we do that, I want to explain a few things as to why we're going to do what we do.

NTP gets precise time from GPSD via a shared memory driver. That shared memory driver uses the magic pseudo-IP address of 127.127.28.X. 127.127.28.0 identifies unit 0 of the ntpd shared-memory driver (NTP0); 127.127.28.1 identifies unit 1 (NTP1). Unit 0 is used for in-band message timestamps and unit 1 for the (more accurate, when available) time derived from combining in-band message timestamps with the out-of-band PPS synchronization pulse. Splitting these notifications allows ntpd to use its normal heuristics to weight them.

Different units - 2 (NTP2) and 3 (NTP3), respectively - must be used when gpsd is not started as root. We've told our GPS HAT to put PPS time on GPIO pin 4, so will also use unit 2 (NTP2) for the PPS time correction. You can verify this by running the command ntpshmmon and it will show you that NTP2 is our primary shared memory clock source. Run that command as sudo, and you should see NTP0 and NTP1 show up as well.

Another thing to note is that even though you're building a highly accurate GPS based stratum 1 NTP server, you're going to want need than one time source. If something happens to the GPS, the antenna breaks, or something else, it's best to have a few sources and let NTP handle the rest. I recommend adding servers that are close to you, and having a few of them available.

Now, we'll get on with editing the ntp.conf file and adding a few NTP servers as well as a log file, our PPS reference and our GPS reference. Your ntp.conf file should look something like this:

sudo nano /etc/ntp.conf

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log

# Leap seconds definition provided by tzdata
leapfile /usr/share/zoneinfo/leap-seconds.list

# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example
server time.nist.gov iburst minpoll 5 maxpoll 5
server tick.usno.navy.mil iburst minpoll 5 maxpoll 5
server 0.us.pool.ntp.org iburst minpoll 5 maxpoll 5
server 1.us.pool.ntp.org iburst minpoll 5 maxpoll 5
server 2.us.pool.ntp.org iburst minpoll 5 maxpoll 5
server 3.us.pool.ntp.org iburst minpoll 5 maxpoll 5

# GPS PPS reference (NTP2)
server 127.127.28.2 minpoll 4 maxpoll 4 prefer
fudge 127.127.28.2 refid PPS

# GPS Serial data reference (NTP0)
server 127.127.28.0 minpoll 4 maxpoll 4
fudge 127.127.28.0 time1 0.500 refid GPS

# pool.ntp.org maps to about 1000 low-stratum NTP servers. Your server will
# pick a different set every time it starts up. Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
#pool 0.debian.pool.ntp.org iburst
#pool 1.debian.pool.ntp.org iburst
#pool 2.debian.pool.ntp.org iburst
#pool 3.debian.pool.ntp.org iburst

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details. The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust

# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255

As you can see, I've added a log file at the top to send logs to /var/log/ntp.log. I also enabled the statsdir to /var/log/ntpstats/. I added 6 NTP servers to make sure I had quite a bit of redundancy, but you can add however many you like. I'd suggest a minimum of 3. I added the minpoll 5 and maxpoll 5 because by default, ntp polls remote servers every 64 seconds, but Linux by default only keeps an ARP table entry for 60 seconds. If the ARP table has flushed the entry for a remote peer or server then when the NTP server sends a request to the remote server an entire ARP cycle will be added to the NTP packet round trip time (RTT). This will throw off the time measurements to servers on the local lan. On a RaspberryPi ARP has been shown to impact the remote offset by up to 600 uSec in some rare cases. The solution is the same for both ntpd and chronyd, add the "maxpoll 5" command to any 'server" or "peer directive. This will cause the maximum polling period to be 32 seconds, well under the 60 second ARP timeout.

Next we added our GPS data, first the PPS reference using the server 127.127.28.2, and we're making PPS our preferred server. Next, we added the GPS signal from 127.127.28.0. We're fudging that one by 500 ms as a start because in my experience, the GPS signal is usually around 500ms off. This will need to be tuned for it to be accurate. More information on that later.

Finally, I remarked out all the standard debian pools, as we're using our own servers. You can leave them in if you want instead of using your own servers.

Now we need to restart the ntp service for it to pick up the config.

sudo systemctl restart ntp

Once NTP restarts, we can check the status by using the ntpq -p (or -pn if you don't want name resolution) command.

ntpq -p

It will take a few moments for NTP to connect to the servers in your list and sort things out. You're looking for the little space just to the left of the name or IP.

(blank) Discarded as not valid
x Discarded by the intersection algorithm as a falseticker
- Discarded by the cluster algorithm as an outlier
+ Included by the combining algorithm
# Backup time source
* System peer (This is what we're looking for)
o Indicates a PPS peer whose driver support is directly compiled into ntpd (NA for us)

Ultimately, you'll end up with something that looks like this:

ntpq -pn

And now, you have a working Stratum 1 NTP Server! Your next steps should be to go ahead and configure your RPi properly by setting localization options, giving it a static IP (you'll definitely want to do this if you're making an NTP server), and anything else I've completely skipped over in the making of this how-to, especially if you haven't done this already. I would not recommend making this a public server as legacy NTP has some security issues with it. There is a hardened version of NTP called NTPSec that is available for Raspbian, but I haven't gotten around to messing with it yet. I would assume that the steps would be the same though.

Update: I just installed NTPSec. It removes NTP, and the ntp.conf file looks like it lives at /etc/ntpsec/ntp.conf now, plus the service is obviously called ntpsec rather than ntp. If you want to make your sever publicly available, I'd suggest using ntpsec rather than regular ntp.

Remember earlier where I said that we were fudging the GPS signal by 500ms, but it needed to be tuned? Yeah, well that's a world all it's own. As it sits, the time you will receive right now will be just fine, but it you want more accurate time, you can fiddle around with it and tune things to become incredibly accurate. Here's a link you can use to learn about that tuning. as it's something that's a little too deep to get into in this post. There's more info in the references below as well about tuning,

GPSD Performance Tuning

There is one tool that comes with gpsd, it's called ntpoffset. It's mentioned in the link above and can be found in /usr/share/doc/gpsd-clients/examples/ for those of you that want to play with it (check out the README in that directory too) . If you're going to try to tune this thing, I would recommend removing the 500ms fudge and letting it settle to get an accurate offset number, at least a day to be safe. I'm doing that right now myself. If you don't mind, please let me know in the comments what your offset comes out to and how long you let it settle for before running the ntpoffset script. You won't have to create the directory or chown it as said in the tuning link above, it will already be there for you. Just run the script and let me know your offset. Also, check periodically to see if your offset is changing. The script gives you an AVERAGE, and it's probably going to change. Remember, once you set the offset (the fudge), that offset is going to change by number of your fudge, so if you set it to 0.500, then let it run for a day and the actual offset needed to be 0.540, the script is going to tell you it's now -40.xxx. If you set it to 0.500 and let it run for a bit, then changed it to 0 and let it run for, say a couple hours and the offset is really 0.540, the ntpoffset script is going to spit out something like -220.xxx. In the image below (which it's incorrect because I removed the fudge from the config and let it run for a short while, so it's lower than it should be), you'll see that the number is -465.353, so my fudge time1 number would be 0.465. In the live ntpq screen, it would be 0.553. If your offset is positive, say if ntpoffset gives you 465.353 (no negative sign), then your fudge time1 would be -0.465. Got it? Told you that it's a world all it's own...

I hope this help clear up some of the confusion out there and help some folks out. Be sure to check out the references below. I couldn't have done this without them.

Resources:

BIPM 2018 Annual Report: Scroll to page 65 for the Time Dissemination Services section. This contains the NTP servers of the National Metrology Laboratories of countries around the world and is a great resource for other stratum 1 NTP servers, many of which are updated directly from atomic sources.
NTPSec.org

References:
Steve Friedl's Unixwiz.net Tech Tips: Building a GPS Time Server with the Raspberry Pi 3
Gary E. Miller and Eric S. Raymond: GPSD Time Service HOWTO
David Taylor @ Satsignal.eu: The Raspberry Pi as a Stratum-1 NTP Server
Ax0n's Den: Stratum-1 NTP Server
Adafruit: Adafruit Ultimate GPS HAT for Raspberry Pi

1Jan/200

Custom PRTG Sensor with Speedtest.Net CLI (Windows)

There's a few different options out there offering insight into how to create a custom speed test sensor to PRTG, but today I'm going to use this one from Nicolai Pederson as my jumping off point. Nicolai was using an .exe file from Github that hadn't been updated in sometime now, and when I started messing with it, I noticed that the speed test really didn't run long enough to give a valid result. Also, Ookla's Speedtest.net recently released their own CLI tool, so I wanted to take what Nicolai did and make it work with the new took from Ookla, which is actually pretty easy. So, we're going to follow his instructions, with a few changes to his .bat file and I'm going to make one change to keep the results consistent.

    1. Download the Speedtest.net CLI app from Ookla.
    2. Copy those files to “C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML”. For sake of simplicity, that's going to be our working directory.
    3. Open up a command prompt, cd to “C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML”, and run "speedtest.exe -L". This is going to give you a list of servers close to you. I would recommend picking the server of your ISP if it's on the list.
    4. Once you have your server picked, make note of server ID. We're going to be using that in our .bat file shortly. In my case, I'm using the Spectrum server, ID 16969.
    5. Open up Notepad and copy the following. We're going to create a .bat file with it.
      1. @ECHO off
        SETLOCAL EnableDelayedExpansion
        SET "Latency="
        SET "Download="
        SET "Upload="
        FOR /F "tokens=4,7,8 delims=," %%A IN ('"C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\speedtest.exe" --accept-license -s 16969 -f csv') DO (
        SET Latency=%%~A
        SET Download=%%~B
        SET Upload=%%~C
        )
        ECHO ^<PRTG^>
        ECHO ^<result^>
        ECHO ^<Channel^>Ping Latency^</Channel^>
        ECHO ^<value^>%Latency%^</value^>
        ECHO ^<Mode^>Absolute^</Mode^>
        ECHO ^<Unit^>TimeResponse^</Unit^>
        ECHO ^<Float^>1^</Float^>
        ECHO ^<ShowChart^>1^</ShowChart^>
        ECHO ^<ShowTable^>1^</ShowTable^>
        ECHO ^</result^>
        ECHO ^<result^>
        ECHO ^<Channel^>Download^</Channel^>
        ECHO ^<value^>%Download%^</value^>
        ECHO ^<Mode^>Absolute^</Mode^>
        ECHO ^<volumeSize^>MegaBit^</volumeSize^>
        ECHO ^<float^>0^</float^>
        ECHO ^<unit^>SpeedNet^</unit^>
        ECHO ^<ShowChart^>1^</ShowChart^>
        ECHO ^<ShowTable^>1^</ShowTable^>
        ECHO ^</result^>
        ECHO ^<result^>
        ECHO ^<Channel^>Upload^</Channel^>
        ECHO ^<value^>%Upload%^</value^>
        ECHO ^<Mode^>Absolute^</Mode^>
        ECHO ^<volumeSize^>MegaBit^</volumeSize^>
        ECHO ^<float^>0^</float^>
        ECHO ^<unit^>SpeedNet^</unit^>
        ECHO ^<ShowChart^>1^</ShowChart^>
        ECHO ^<ShowTable^>1^</ShowTable^>
        ECHO ^</result^>
        ECHO ^</PRTG^>

    6. Replace the server 16969 with the server ID of your choice. The reason we're going to use the same server, preferably from your ISP, is to have consistency with your speed test. If you're using multiple servers, you could get varying results as you don't know what kind of bandwidth each server has. And, if you're using your own ISP, they're a lot more likely to give you truly accurate results and less likely to block you if you run the test a lot.
    7. Save the file as something like speedtest.bat in the working directory, “C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML”. Just make sure you remember what you saved it as.
    8. Go to PRTG and create a new sensor. The sensor type will be "EXE / Script Advanced", then name it and select your "speedtest.bat" for EXE/Script under Sensor Settings.
    9. Once you have the sensor created and you gather some data, go in change the scanning interval. You obviously don't want this thing scanning every 60 seconds or so. I set mine to scan every 6 hours, but you can set yours as you see fit.

So, why did I do this when Nicolai had already done the work? Well, the Github .exe that Nicolai uses only runs for a few seconds and doesn't really run long enough to give an accurate reading, also it tried to use servers that didn't exist anymore. If you check his website, you'll see there's some comments about people complaining that they were getting incorrect results. The "official" Speedtest CLI app solves that problem. Also, the official app can spit out JSON, but PRTG doesn't like the format, and I'm not smart enough to know how to parse the data into a format that it does like, so I had to figure out a way to get the data I wanted into a format that PRTG wanted.

Now, for those of you like me that aren't smart and want to figure out what that .bat file is doing, I'll explain. The speedtest.exe file is spitting out the data in a CSV format (the "-f csv" behind the command is the formatting). The "FOR /F "tokens=4,7,8 delims=,"" in the .bat is a loop telling it that the output is comma delimited, and you want to look at the data that's behind the 4th, 7th, and 8th comma. YOU MAY NEED TO CHANGE THIS! The reason It's set to 4,7,8 on mine is because the output of the actual command comes back with the very first line being "Spectrum - Columbus, OH", and it reads the comma before OH as, well, a comma. If the output of your command doesn't have a comma there, you may have to change it. To find out for sure, you can run the following:

speedtest.exe --accept-license -s #YOURSERVERNUMBER# -f csv

The count commas. If you're not sure what data is where, you can run the following and it will tell you.

speedtest.exe --accept-license -s #YOURSERVERNUMBER# -f csv --output-header

That will tell you what data is in which location. You'll get something like this:

"server name","server id","latency","jitter","packet loss","download","upload","download bytes","upload bytes","share url"

"Spectrum - Columbus, OH","16969","7.495","0.786","N/A","110863914","4621941","1425470568","32103608","https://www.speedtest.net/result/c/73cf23fa-84cd-4473-a816-4154424fd027"

Of course, now that you know what's being parsed and how, you can add more data to this if you want, like packet loss, jitter, download bytes, etc. You just need to follow the example set in the .bat file, make sure you test it out. You can run the .bat from the CLI and see the data or check for errors before creating the sensor. Since I first posted this, I've gone ahead and created an example that pulls all the information from the CLI output except packet loss into one place. You can download that here, and just rename it to .bat to run it. Don't forget to change your server ID too! One more note of changes I made in the .bat file between his and mine; I removed his remarks, added a ~ between "%%~A", etc to remove the quotes from the response in the CSV file, and cleaned up the formatting a bit, and removed the "00" from the upload and download values (they're not needed). I should also note that I spent over 2 hours trying to figure out why I was seeing good, clean data at the CLI, but only zeros in PRTG. Let's just say there's very a reason the "--accept-license" option is set in the command now <grrr....>. Once you're done, you'll end up with a working sensor!

Update 3/11/20: As Roman in the comments found out, if your country or area requires that you accept other types of licenses or data protection regulations (like the GDPR in the EU), you may need to feed that into the command. It took me 2 hours to realize I needed to feed the "--accept-license" option, and it took Roman 3 days to figure out he needed to feed the "--accept-gdpr" option. Whenever you first run the command from the CLI, you will be asked to accept certain things, like the license and possibly the GDPR and anything else. REMEMBER WHAT IT IS YOU ACCEPT. PRTG is going to run this command as a different user, which is why you have to feed the "--accept-license" option to the command; just because you accepted the license doesn't mean PRTG did. If you're getting zero's on your sensor, try to figure out what other options need to be accepted in your area when you issue the command. Then go into the comments below and thank Roman for chasing this down over 3 days so you didn't have to.

7Jul/177

IPv6, Time Warner / Spectrum, and the Juniper SRX.

I've had an IPv6 tunnel from HE.net for quite some time now. Back when I was running the ASA 5505 as my edge, I had to put a router behind it to create the tunnel. Then, when I replaced the ASA with an SRX 220 back in December 2015, I was able to build the tunnel natively on the SRX. Since that time, Time Warner has gotten around to providing IPv6 in my area and I've tried a couple different times to get it working with no luck. Now, I've finally decided that I wasn't going to stop working on it until I got it working, and I've done just that, so it's time to tell you guys how to do it yourself.

First a few caveats... Obviously, Time Warner (now Spectrum) needs to provide IPv6 in your area and that your modem supports it. I don't remember how I found out that they finally had it here, but it was probably a fellow network engineer at TWC that told me. Second, realize that you're going to have to reboot the SRX, so you're going to lose connectivity for a bit. The reason you'll need to reboot is that we need to enable IPv6 flow mode, otherwise the SRX will just drop IPv6 traffic. Let's start with that...

Obviously, ssh into the SRX and enter config mode. Then enter the following command:

set security forwarding-options family inet6 mode flow-based

Then you'll need to reboot with "request system reboot". Once it comes back up, you're ready to move on.

Your ge-0/0/0.0 interface probably looks something this at present:

greg@SRX220H# show interfaces ge-0/0/0.0

description "Uplink to Cable Modem";

family inet {

    dhcp;

}

We're going to need to change the dhcp daemon that you're using on that interface because if we were to continue on with what's coming, you'd get an error. Then we're going to add the ipv6 dhcpv6-client config to the same interface. Here's your commands:

delete interfaces ge-0/0/0 unit 0 family inet dhcp
set interfaces ge-0/0/0 unit 0 family inet dhcp-client
set interfaces ge-0/0/0 unit 0 family inet6 dad-disable
set interfaces ge-0/0/0 unit 0 family inet6 dhcpv6-client client-type statefull
set interfaces ge-0/0/0 unit 0 family inet6 dhcpv6-client client-ia-type ia-na
set interfaces ge-0/0/0 unit 0 family inet6 dhcpv6-client client-ia-type ia-pd
set interfaces ge-0/0/0 unit 0 family inet6 dhcpv6-client client-identifier duid-type duid-ll
set interfaces ge-0/0/0 unit 0 family inet6 dhcpv6-client update-router-advertisement interface vlan.0

Now we need to set our firewall to allow some traffic:

set security zones security-zone untrust interfaces ge-0/0/0.0 host-inbound-traffic system-services dhcpv6
set security zones security-zone untrust interfaces ge-0/0/0.0 host-inbound-traffic protocols router-discovery

That should be pretty self explanatory. You need to allow dhcpv6 through the firewall for all this to work, and we're going to use router-discovery to figure things out. Once you commit that, the SRX should ask TWC for an IPv6 address. Let's check to see if we got one...

greg@SRX220H# run show dhcpv6 client binding

IP/prefix                       Expires     State      ClientType    Interface       Client DUID

2607:fcc8:ffc0:5:14c9:b140:XXXX:XXXX/128 600553 BOUND  STATEFUL      ge-0/0/0.0      LL0x3-54:e0:32:ec:XX:XX

2605:a000:XXXX:XXXX::/64        600553      BOUND      STATEFUL      ge-0/0/0.0      LL0x3-54:e0:32:ec:XX:XX

It looks like we have an address! Now, we need to add a route... we'll find our next hop by running the previous command and adding detail:

greg@SRX220H# run show dhcpv6 client binding detail

Client Interface: ge-0/0/0.0

     Hardware Address:             54:e0:32:ec:XX:XX

     State:                        BOUND(DHCPV6_CLIENT_STATE_BOUND)

     ClientType:                   STATEFUL

     Lease Expires:                2017-07-14 08:01:52 EDT

     Lease Expires in:             600551 seconds

     Lease Start:                  2017-07-07 08:01:52 EDT

     Bind Type:                    IA_NA IA_PD

     Client DUID:                  LL0x3-54:e0:32:ec:XX:XX

     Rapid Commit:                 Off

     Server Ip Address:            fe80::201:5cff:fe78:XXXX

     Client IP Address:            2607:fcc8:ffc0:5:14c9:b140:XXXX:XXXX/128

     Client IP Prefix:             2605:a000:XXXX:XXXX::/64

DHCP options:

    Name: server-identifier, Value: LL_TIME0x1-0x1d7c50b0-00:50:56:XX:XX:XX

Yes, the lease started about an hour before I posted this. I was so excited that I had to post immediately! Anyway, we're looking for that Server IP Address. Once we have that, let's add a static route to it.

set routing-options rib inet6.0 static route ::/0 qualified-next-hop fe80::201:5cff:fe78:XXXX interface ge-0/0/0.0

The qualified-next-hop is going to give you a lot more control over a standard next-hop. Commit the config. Once everything is committed, it's time to test, so we'll ping Google's DNS server.

# run ping 2001:4860:4860::8888

You should get a response. IPv6 is now working! W00T! In order to get your network clients talking to the internet on IPv6, you'll have to configure them to use IPv6. As you can see up above in the dhcpv6 client binding detail, there's a "Client IP Prefix". That's the prefix assigned to you. If you do a "run show interfaces vlan.0 terse", you'll see that it now has an inet6 address that looks like 2605:a000:XXXX:XXXX:1::1/80. That's going to be your IPv6 router / gateway address. You can statically assign IP's by just counting up from that last ::1, so assign 2605:a000:XXXX:XXXX:1::2/80 to your workstation and try to ping 2001:4860:4860::8888. If you get a response, you're good to go.

So, that's the commands I had to enter to get IPv6 working on my SRX. YMMV depending on TWC's configuration in your area, but this should get you pretty damn close.

19Jan/160

Upgrading the gaming rig, for what feels like the 4173rd time.

I've been working on a plan to upgrade my gaming rig lately, especially since I had a 4790K come available with the upgrade of the FreeNAS box. For Christmas, I got a pair of Gigabyte R9-390X's to upgrade my dual HD7970's. I've been running 32GB of RAM in my 4770K box for quite a while now, so there was nowhere to go from there. The plan was to replace the Gigabyte Z87X-OC and i7-4770K with the ASRock Z97 Extreme6 and i7-4790K from the FreeNAS box. The Z97 board has a PCI 3.0 x4 M.2 slot on it, so I wanted to get the speed increase from using it in addition to everything else. But, here's the problem...

The LGA1150 processors, which means ALL the Haswell processors not considered "Enthusiast / High End"; so the Core i3, i5, and i7s with a 4XXX model number or a G3XXX model number; only have 16 PCIe lanes. The Core i7-5820K (6 core) has 28 lanes and the 5930K (6 core) and 5960X (8 core) both have 40 lanes. Now, let's do some math on what I wanted to go into that computer:

  • Two R9-390x's - 32 Lanes
  • M.2 4x - 4 Lanes
  • Thunderbolt 2 AIC - 4 lanes

Well, I'm no mathematician, but I know 40 lanes when I see them. That only left me one option if I wanted to stick with Haswell, and that's to go with the 5930K. The 5960X is still a $1000 processor and I just wasn't going to drop that kind of coin on a CPU. So, today I went ahead and placed the order and here's the new specs for the new gaming rig:

  • Intel Core i7-5930K CPU
  • Corsair H110i GTX Liquid CPU Cooler
  • ASRock X99 Extreme6/3.1 Motherboard
  • 32GB Corsair Dominator Platinum DDR4-2666 (4 x 8GB)
  • 2x Gigabyte R9-390X 8GB
  • ASRock Thunderbolt 2 AIC
  • Samsung 950 PRO 512 GB M.2 4x SSD
  • Samsung 850 EVO 1TB SATA6 SSD
  • Corsair Obsidian 750D Full Tower Chassis
  • Corsair AX1200i PSU
  • Corsair Professional Blue Individually Sleeved PSU cables
  • 5x Corsair SP140 Blue LED Case Fans (2 front, 2 on the radiator, 1 rear)
  • Corsair Link Commander Mini
  • 2x Corsair RGB Light kit

No spinning platters in this baby! We're going ALL SSD. Spinning platters are for the NAS. The motherboard has dual gigabit NICs, and my network devices all support link aggregation, so I'll be able to get 2 gigabit network access to the NAS, and that should be more than enough for pulling documents or anything else I need off of it. All this will be displayed on my current triple AOC i2769Vm 27" monitors, which give me nearly 6 linear feet of monitor space. Yes, they are only 1080, but I'm not quite ready to dump out the money for three 27" 4K monitors and they don't make ultra-wide curved monitors big enough for me yet.

Today's order was for the CPU, motherboard, and RAM. Everything else is already here. I'm hoping that everything will be here by the end of the week and I'll be able to finish the build this weekend, so I can finish working on the FreeNAS / Plex Automation series.

Here's a sneak peek of what it'll look like. I was mounting the lights and fans in the case last night, and since the X99 Extreme6 and the Z97 Extreme6 look nearly identical, it'll give you an idea of what we'll be dealing with when it's all said and done.

IMG_5327

Filed under: Computers No Comments
22Dec/150

Home NAS Refresh

I think that, in this day and age, everyone should have a NAS at their house. For those of you that don't know what I'm talking about, NAS stands for 'Network Attached Storage'. A NAS is handy for storing all sorts of things, primarily backups of your computers and your media. In my case, I have a lot of movies and TV shows for my various media players. I also have a ton of photos and videos from over the years, as well as from my drones. Having a large NAS means that I don't have delete anything. My NAS also acts as a server for various other things that I'll get into in another post.

For your NAS to be effective, it needs to have lots of space and have enough room to expand. You also need to have an effective operating system running the NAS. For this build, I'm going to use FreeNAS. I had been planning to build this thing for a while, but didn't get around to finally getting everything setup and running until July 31, 2015. Since then it's been running pretty stable, but I used an Intel G3220 and 8GB of RAM when I first put it together and I've outgrown that processor and RAM, so it's time for an upgrade. Here's the hardware list of everything that's going into the machine:

  • Intel Core i7-4790K CPU
  • ASRock Z97 EXTREME6 ATX LGA1150 Motherboard
  • G.Skill Ripjaws X Series 32GB (4 x 8GB) DDR3-1600 Memory
  • 6x WD Red 3TB 3.5" 5400RPM HDD
  • Rosewill R​SV-L4412 -​ 4U Rackmo​unt Server​ C​hassis, 12​ SATA / SA​S Hot-swap​ Drives
  • EVGA SuperNOVA 1000G2 1000W 80+ Gold Certified Fully-Modular ATX Power Supply

The only thing that's carrying over from the previous build are the 6 WD Red 3TB hard drives and the actual FreeNAS install. I was going to just upgrade the CPU and the RAM, but some pins got bent on the ASUS Z87-A motherboard I had, so it needed to get upgraded too. I also figured that while I was at it, I'd put it in a nice rackmount chassis too.

The build went rather smooth. I pulled the hardware out of the old mid-tower case and moved it into the rackmount chassis. I had originally planned on using some M.2 SSDs for boot drives, but ran into some issues. First, the drives I bought weren't compatible with the Ultra M.2 slot on the motherboard. Secondly, the other M.2 slot ate two of my SATA ports on the motherboard. Because I didn't bother to read the manual, it took me quite a while to figure out why those two drives weren't being seen by the BIOS. Ultimately, I got everything put together and all 6 drives were being recognized. FreeNAS booted right up without any issues. I'll probably pick up an Ultra M.2 SSD in the future to use as L2ARC since it's so freaking FAST.

More info will be posted soon on how I'm going to automate my media collection and sharing.

20Feb/130

Windows IP conflict when there is no conflict

Just had an interesting problem with a customer that seems a bit obscure, so I figured I would write it down to help someone else. All of the other solutions to this issue focus solely on there being a problem on the Windows side, which may not necessarily be the case.

Situation: customer is setting up a Windows 2008 R2 server in a VMware cluster, on a VLAN that is sitting behind a firewall. The firewall is is the gateway for the VLAN (say 192.168.34.1). When configuring the network interface on the server, picking ANY IP address in the 192.168.34.0/24 network results in the error message “Windows had detected an IP address conflict”. This happens even if there are no other devices on the VLAN aside from the firewall.

The issue? There was a static (identity) NAT entry in the Cisco ASA firewall for 192.168.34.0/24. By default, Cisco firewalls will proxy ARP for NAT entries.

  • (8.3(1), 8.3(2), and 8.4(1)) The default behavior for identity NAT has proxy ARP disabled. You cannot configure this setting.
  • (8.4(2) and later) The default behavior for identity NAT has proxy ARP enabled, matching other static NAT rules. You can disable proxy ARP if desired.

This is desirable behavior for a firewall on the edge of the network because the upstream router needs to know where to send traffic for NAT’ed hosts. For internal firewalls this can cause issues, especially with 8.4 code where you need to setup identity NAT to exempt devices from NAT.

The solution? Add “no-proxy-arp” to the end of your identity NAT statements:

nat (inside,outside) source static obj_Internal obj_Internal no-proxy-arp route-lookup

The other (less desirable) solution is to disable the ARP-checking functionality in Windows, but this means it won’t be able to detect a legitimate IP conflict. You can do this through a quick registry hack: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters, create a DWORD named “ArpRetryCount” with a value of “0?.

9Feb/124

New App! Square Fee Calculator

I recently signed up for the Square service to accept credit cards on my iPhone. They have two different ways of charging fees: 2.75% per transaction when you swipe the card using their reader or 3.5% + %0.15 per transaction if you enter the card number manually. Since I've already written mobile fee calculators for both PayPal and Etsy, I decided to use the same code and make one for Square. All 3 are designed to fit perfectly on an iPhone screen and since they are very lightweight, they run pretty quick. If you use any of these services, these calculators will come in very handy!

PayPal Fee Calculator
Etsy Fee Calculator
Square Fee Calculator

4Aug/100

How to enable AHCI/RAID mode in Windows 7 without reinstalling

I recently got a wild hair up my ass to add a RAID to my desktop. My desktop is a Gateway FX6840-23 and it came with a 1TB drive. I bought an identical drive and thought that I'd put then in RAID 0 for the increased performance, seeing as my Experience Index was only 5.9 due to a slow HDD (all other indexes were in the mid-7's, and the drive is a 7200 RPM unit).

Digging around the BIOS I saw that the SATA controller was using AHCI mode. I cloned my current drive to another 1TB drive I had (yeah, I have 3 -1TB drives, a 500GB, and a 1.5 TB), rebooting into the BIOS and changed it to RAID. After a reboot, I hit ctrl-I and entered the RAID utility. I built the RAID and rebooted. Well, to put it nicely, I got a BSOD. I tried various things for the next 3 hours, including using Windows 7's extended partition utility, doing a complete restore to factory on the extended partition, and everything. After I did the restore, I saw that the HDD performance hadn't changed.

Well, I haven't messed with RAID before on a desktop, so this was a learning experience. After some Google searches, I put the computer back in AHCI mode and booted to the clone. This worked just fine. I went to Gateway's website and downloaded the RAID drivers.

I noticed that the driver was named iaStorV.sys, so I did a search for it and found it already installed in the Windows\System32\Drivers folder. I did a registry search for it and found it in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\iaStorV. This made me happy!

Some more Googling later and I figured out that if I changed the REG_DWORD from 3 to 0 that it would enable things to work. I rebooted the computer, went back into the BIOS and changed the SATA controller to use RAID, pointed it to boot from the clone, and it booted right up! No BSOD, no hiccups, no nothing!

This should work going from IDE mode as well. I tried to clone the clone to the RAID, but Acronis didn't like that too much, so I'm doing a full backup of the clone (I needed to do it anyway) and I'm going to try to restore it with the Acronis Resuce media. It's already midnight, and this is one of those things that I'm not going to be able to put down until I'm done with it. Oh well, I guess it's time to get back to work! Good luck getting your stuff working!