GregLedet.net That dude writing stuff about that thing.

1Jan/200

Custom PRTG Sensor with Speedtest.Net CLI (Windows)

There's a few different options out there offering insight into how to create a custom speed test sensor to PRTG, but today I'm going to use this one from Nicolai Pederson as my jumping off point. Nicolai was using an .exe file from Github that hadn't been updated in sometime now, and when I started messing with it, I noticed that the speed test really didn't run long enough to give a valid result. Also, Ookla's Speedtest.net recently released their own CLI tool, so I wanted to take what Nicolai did and make it work with the new took from Ookla, which is actually pretty easy. So, we're going to follow his instructions, with a few changes to his .bat file and I'm going to make one change to keep the results consistent.

    1. Download the Speedtest.net CLI app from Ookla.
    2. Copy those files to “C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML”. For sake of simplicity, that's going to be our working directory.
    3. Open up a command prompt, cd to “C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML”, and run "speedtest.exe -L". This is going to give you a list of servers close to you. I would recommend picking the server of your ISP if it's on the list.
    4. Once you have your server picked, make note of server ID. We're going to be using that in our .bat file shortly. In my case, I'm using the Spectrum server, ID 16969.
    5. Open up Notepad and copy the following. We're going to create a .bat file with it.
      1. @ECHO off
        SETLOCAL EnableDelayedExpansion
        SET "Latency="
        SET "Download="
        SET "Upload="
        FOR /F "tokens=4,7,8 delims=," %%A IN ('"C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\speedtest.exe" --accept-license -s 16969 -f csv') DO (
        SET Latency=%%~A
        SET Download=%%~B
        SET Upload=%%~C
        )
        ECHO ^<PRTG^>
        ECHO ^<result^>
        ECHO ^<Channel^>Ping Latency^</Channel^>
        ECHO ^<value^>%Latency%^</value^>
        ECHO ^<Mode^>Absolute^</Mode^>
        ECHO ^<Unit^>TimeResponse^</Unit^>
        ECHO ^<Float^>1^</Float^>
        ECHO ^<ShowChart^>1^</ShowChart^>
        ECHO ^<ShowTable^>1^</ShowTable^>
        ECHO ^</result^>
        ECHO ^<result^>
        ECHO ^<Channel^>Download^</Channel^>
        ECHO ^<value^>%Download%^</value^>
        ECHO ^<Mode^>Absolute^</Mode^>
        ECHO ^<volumeSize^>MegaBit^</volumeSize^>
        ECHO ^<float^>0^</float^>
        ECHO ^<unit^>SpeedNet^</unit^>
        ECHO ^<ShowChart^>1^</ShowChart^>
        ECHO ^<ShowTable^>1^</ShowTable^>
        ECHO ^</result^>
        ECHO ^<result^>
        ECHO ^<Channel^>Upload^</Channel^>
        ECHO ^<value^>%Upload%^</value^>
        ECHO ^<Mode^>Absolute^</Mode^>
        ECHO ^<volumeSize^>MegaBit^</volumeSize^>
        ECHO ^<float^>0^</float^>
        ECHO ^<unit^>SpeedNet^</unit^>
        ECHO ^<ShowChart^>1^</ShowChart^>
        ECHO ^<ShowTable^>1^</ShowTable^>
        ECHO ^</result^>
        ECHO ^</PRTG^>

    6. Replace the server 16969 with the server ID of your choice. The reason we're going to use the same server, preferably from your ISP, is to have consistency with your speed test. If you're using multiple servers, you could get varying results as you don't know what kind of bandwidth each server has. And, if you're using your own ISP, they're a lot more likely to give you truly accurate results and less likely to block you if you run the test a lot.
    7. Save the file as something like speedtest.bat in the working directory, “C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML”. Just make sure you remember what you saved it as.
    8. Go to PRTG and create a new sensor. The sensor type will be "EXE / Script Advanced", then name it and select your "speedtest.bat" for EXE/Script under Sensor Settings.
    9. Once you have the sensor created and you gather some data, go in change the scanning interval. You obviously don't want this thing scanning every 60 seconds or so. I set mine to scan every 6 hours, but you can set yours as you see fit.

So, why did I do this when Nicolai had already done the work? Well, the Github .exe that Nicolai uses only runs for a few seconds and doesn't really run long enough to give an accurate reading, also it tried to use servers that didn't exist anymore. If you check his website, you'll see there's some comments about people complaining that they were getting incorrect results. The "official" Speedtest CLI app solves that problem. Also, the official app can spit out JSON, but PRTG doesn't like the format, and I'm not smart enough to know how to parse the data into a format that it does like, so I had to figure out a way to get the data I wanted into a format that PRTG wanted.

Now, for those of you like me that aren't smart and want to figure out what that .bat file is doing, I'll explain. The speedtest.exe file is spitting out the data in a CSV format (the "-f csv" behind the command is the formatting). The "FOR /F "tokens=4,7,8 delims=,"" in the .bat is a loop telling it that the output is comma delimited, and you want to look at the data that's behind the 4th, 7th, and 8th comma. YOU MAY NEED TO CHANGE THIS! The reason It's set to 4,7,8 on mine is because the output of the actual command comes back with the very first line being "Spectrum - Columbus, OH", and it reads the comma before OH as, well, a comma. If the output of your command doesn't have a comma there, you may have to change it. To find out for sure, you can run the following:

speedtest.exe --accept-license -s #YOURSERVERNUMBER# -f csv

The count commas. If you're not sure what data is where, you can run the following and it will tell you.

speedtest.exe --accept-license -s #YOURSERVERNUMBER# -f csv --output-header

That will tell you what data is in which location. You'll get something like this:

"server name","server id","latency","jitter","packet loss","download","upload","download bytes","upload bytes","share url"

"Spectrum - Columbus, OH","16969","7.495","0.786","N/A","110863914","4621941","1425470568","32103608","https://www.speedtest.net/result/c/73cf23fa-84cd-4473-a816-4154424fd027"

Of course, now that you know what's being parsed and how, you can add more data to this if you want, like packet loss, jitter, download bytes, etc. You just need to follow the example set in the .bat file, make sure you test it out. You can run the .bat from the CLI and see the data or check for errors before creating the sensor. Since I first posted this, I've gone ahead and created an example that pulls all the information from the CLI output except packet loss into one place. You can download that here, and just rename it to .bat to run it. Don't forget to change your server ID too! One more note of changes I made in the .bat file between his and mine; I removed his remarks, added a ~ between "%%~A", etc to remove the quotes from the response in the CSV file, and cleaned up the formatting a bit, and removed the "00" from the upload and download values (they're not needed). I should also note that I spent over 2 hours trying to figure out why I was seeing good, clean data at the CLI, but only zeros in PRTG. Let's just say there's very a reason the "--accept-license" option is set in the command now <grrr....>. Once you're done, you'll end up with a working sensor!

Update 3/11/20: As Roman in the comments found out, if your country or area requires that you accept other types of licenses or data protection regulations (like the GDPR in the EU), you may need to feed that into the command. It took me 2 hours to realize I needed to feed the "--accept-license" option, and it took Roman 3 days to figure out he needed to feed the "--accept-gdpr" option. Whenever you first run the command from the CLI, you will be asked to accept certain things, like the license and possibly the GDPR and anything else. REMEMBER WHAT IT IS YOU ACCEPT. PRTG is going to run this command as a different user, which is why you have to feed the "--accept-license" option to the command; just because you accepted the license doesn't mean PRTG did. If you're getting zero's on your sensor, try to figure out what other options need to be accepted in your area when you issue the command. Then go into the comments below and thank Roman for chasing this down over 3 days so you didn't have to.

22Dec/150

Home NAS Refresh

I think that, in this day and age, everyone should have a NAS at their house. For those of you that don't know what I'm talking about, NAS stands for 'Network Attached Storage'. A NAS is handy for storing all sorts of things, primarily backups of your computers and your media. In my case, I have a lot of movies and TV shows for my various media players. I also have a ton of photos and videos from over the years, as well as from my drones. Having a large NAS means that I don't have delete anything. My NAS also acts as a server for various other things that I'll get into in another post.

For your NAS to be effective, it needs to have lots of space and have enough room to expand. You also need to have an effective operating system running the NAS. For this build, I'm going to use FreeNAS. I had been planning to build this thing for a while, but didn't get around to finally getting everything setup and running until July 31, 2015. Since then it's been running pretty stable, but I used an Intel G3220 and 8GB of RAM when I first put it together and I've outgrown that processor and RAM, so it's time for an upgrade. Here's the hardware list of everything that's going into the machine:

  • Intel Core i7-4790K CPU
  • ASRock Z97 EXTREME6 ATX LGA1150 Motherboard
  • G.Skill Ripjaws X Series 32GB (4 x 8GB) DDR3-1600 Memory
  • 6x WD Red 3TB 3.5" 5400RPM HDD
  • Rosewill R​SV-L4412 -​ 4U Rackmo​unt Server​ C​hassis, 12​ SATA / SA​S Hot-swap​ Drives
  • EVGA SuperNOVA 1000G2 1000W 80+ Gold Certified Fully-Modular ATX Power Supply

The only thing that's carrying over from the previous build are the 6 WD Red 3TB hard drives and the actual FreeNAS install. I was going to just upgrade the CPU and the RAM, but some pins got bent on the ASUS Z87-A motherboard I had, so it needed to get upgraded too. I also figured that while I was at it, I'd put it in a nice rackmount chassis too.

The build went rather smooth. I pulled the hardware out of the old mid-tower case and moved it into the rackmount chassis. I had originally planned on using some M.2 SSDs for boot drives, but ran into some issues. First, the drives I bought weren't compatible with the Ultra M.2 slot on the motherboard. Secondly, the other M.2 slot ate two of my SATA ports on the motherboard. Because I didn't bother to read the manual, it took me quite a while to figure out why those two drives weren't being seen by the BIOS. Ultimately, I got everything put together and all 6 drives were being recognized. FreeNAS booted right up without any issues. I'll probably pick up an Ultra M.2 SSD in the future to use as L2ARC since it's so freaking FAST.

More info will be posted soon on how I'm going to automate my media collection and sharing.