Configuring Unraid for constant data ingest

How to configure unraid for constant downloads

Hey guys, this is my 4th blog post in this series of how to configure unraid by.....well, someone who probably shouldnt be writing this, along with my terrible grammar mistakes im only so-so when it comes to Unraid, SpaceInvader One is probably the go to guy on Unraid setup guides (i am not affiliated with him, just a big fan). Alright, done selling my self short, the purpose of this guide is to maximize my download speeds, and lessen up the work load that my array has been doing this whole time. I am moving all of my Ingest processing onto my cache in hopes that the faster read and write speeds mean my cpu has to do less work over all and can have more idle time.

so, lets get to it......

What i am running:

NZBGet

Sonarr

Radarr

Specifics needed for this setup:

A large Cache pool (mine is 2 - 1tb drives in Raid0)

Enough RAM to allocate a gig or two extra to NZBGet

I'm going to setup some 'Ingest' folders in my cache pool, I am doing this for two main reasons.

I want all the downloads, checking, and unpacking to happen as fast as it possibly can and SSD's are great for this. The second reason, quick read speeds, I want Radarr and Sonarr to pull from my SSD's / Cache only when the file is ready.

So lets get some folders going, you can land on which ever structure you want, this is just how i did it.

I will be using the command line interface or CLI for short to get this done.

CD - Changes directory "CD /mnt" -This command brings you to the dir /mnt
MKDIR - Creates a directory / Folder - "MKDIR Ingest"
Tree - Shows the directory tree from the location you are.

You can also use midnight commander "MC" if you want to work with a GUI

So lets begin.

Using the commands above we need to create the following directory tree.

cd /mnt/cache/

mkdir Ingest

cd Ingest

mkdir Downloads

cd Downloads

mkdir Complete

mkdir Incomplete

cd Complete

mkdir Movies

mkdir Shows

cd mnt/cache/Ingest

tree

*recreated in windows for a visual view*

Ok we have our folder structure setup, lets jump into the docker config for NzbGet and add a new path with the following settings 

This is case sensitive, its best to click in the host path line and click your way to the downloads folder.

ok save and let the docker rebuild, and lets jump into NzbGet and adjust some Path and Category settings.

Starting with paths we want to set the following.

The only settings to worry about right now are the MainDir DestDir and InterDir

Setting the Paths to the cache directory has the added benefits of faster que / logging creation for NzbGet. The main point is to define the MainDir variable so the variables below can make use of it.

Next lets move on to the Category settings.

Here is how i setup Movies

You should only need to set the DestDir

And here are my Shows settings

You should only need to set the DestDir

Lastly in the Download Queue section i upped the amounts for both Article Cache to 1024mb and WriteBuffer to a 1024kb

Now everything that NzbGet downloads is going to be pulled straight onto the cache drive and when its done checking and unpacking, Sonarr and Radarr will grab the files from here.

If you plan on using the Drone factory, you may want to hot inside of the docker settings for both Radarr and Sonarr and add a path to the Completed Movies or Shows folder, and inside of the web page for either assign that path variable in the drone factory location field.

Where to look for used servers?

Where to look for used servers

So maybe your sitting at your desk realizing all these external hard drives aren't cutting it for you, or maybe your tired of sharing resources on your primary computer to have Plex and it's siblings Radarr and Sonarr running at full tilt. Maybe you have something to share with the world and want to get a website hosted, or an FTP site.

There are plenty of reasons for deciding to jump into the world of servers, and the first step for many people making that move is to purchase a used enterprise grade server machine. 

Solid choice, these servers are built to last and you can usually find a great deal on them.....but where do you find them?

A lot of people use ebay or a site that parses ebay like LabGopher.

Labgopher is a great resource to use, and even if you can't find the server you want, you can use it to narrow down your searches at other places by seeing whats common and how powerful it is with the handy 'Passmark column' - though it is only as accurate as what the seller provides and some times it doesn't parse the version level of a cpu accurately.

But we do have some other options...

US Based

TheServerStore

EntraSource

/r/HomelabSales

HomeLabHero

SaveMyServer

TechMikeNY

MetServers

Xbyte

OrangeComputers

ServerMonkey

UnixSurplus

HomeLabTech

Expanding your storage array

So you want to connect some drives to your server! We have plenty of options here, i hope you saved a PCIe slot!

Lets Begin, first lets go over the methods of connecting drives internally, you may already be familiar with the most common method, Sata ports.

But, often enough, we don't have enough of them, Your first answer to that might be to search for a Sata Expansion Card, which is great, you are on the right path, but really quickly you will start to see just how deep this rabbit hole goes...

Maybe eventually you made it through some solid channels and got some good advice, Perhaps someone suggested you look into LSI branded cards, or maybe some HP Expander cards, depending on your use case.

Now instead of going over the VAST list of options we have, i'll go over what i believe to be the most common scenarios...

We have two main categories to look at, 

connecting drives internally, and connecting your server via cable to a seperate storage array / disk shelf/DAS (externally) ....or maybe you need to do both.

I'm a big fan of software raid, connecting Just a Bunch Of Disks (JBOD) together....no really, JBOD is a real term used....

Ok so you want to connect....4.....8.....12....16 drives internally, Lets take a look at how we can do that.

Before we dig into the meat and PCB, lets take a look at.....Cable options!

So more often than not you are looking at using one of these mini sas to sata cables for internal connections, they allow you to connect one port to your...as of yet unmentioned LSI HBA card, to 4 physical hard drives.

They can be found HERE! and are very affordable.

So lets keep this in mind, one mini SAS connector will allow us to make 4 sata connections.

Ok, now we can dive into the controller cards, LSI and later on, HP cards to be exact.

LSI SAS 9211-4i 

LSI SAS 9211-8i 

LSI SAS 9201-16i

Pay attention to the last section of numbers and letters, a good rule of thumb is to use this number to understand how many drives each card can handle....4, 8, and 16, each with 1, 2, and 4 mini sas ports on the card, remember, one mini sas allows 4 sata drive connections. You can see the silver port on the right hand side of the cards below.

4i

8i

16i

Now maybe you are wondering what that little "i" at the end stands for, and if you guessed Internal, you'd be correct!

Up until now we have been focusing on purely internal connections...now i guess it is true if you had one of these cards in Machine-A and got a long enough cable and decided to route it out the back....or front of Machine-A and into a second machine....or Machine-B and connect the 4 sata ports to drives within this second machine......you can, but i don't recommend it. A safer way to do this would be to look into some cards with an "e" designation for external.

LSI SAS 9200-8e

LSI SAS 9201-16e

LSI SAS 9202-16e

These cards are going to allow you to connect one physical machines hard drive array to another physical machine, and allow you to treat them as if they were internal drive on the machine your connecting to.

These cards would be placed in your "Compute" server, the machine you want to access the drives from, and with the use of a cable, ran to another machine that has a similar....but not the same HBA card or controller. 

Some common uses would be NetApp gear like the DS4243 or DS4246, 

If you plan to go in that route, make sure to get the correct cable - QSFP+ (SFF-8436) / Mini SAS (SFF-8088)

I recommend going the NetApp route, the used hardware can be found really cheaply on ebay, as of now, around $150-$200 and can support 24 drives.

but maybe you want to get your hands dirty and build your own disk shelf, in that case i would suggest using something like this..

HP SAS Expander

and

LSI SAS 9212-4i4e (4 internal sata ports...not mini sas)

Id recommend the HP SAS expander, it can be found for very cheap, and it can support up to 28 internal drives, as well as allow you to run a cable from said card, to another physical machine and access those drives, this is great for creating your own Direct Attach Storage or DAS for short.

With the two cards above, the general layout would be to install one into a machine (your disk shelf / DAS). To do that, all you really need is a motherboard to power the card and drives, a power supply, and some mini sas to sata, or sata cables if you you use the LSI card. Connect all your drives, and give them power.

On the "Compute" server end (separate physical device), you would install one of the above LSI SAS 9200-16e\8e\4e (External) port cards and run a cable between the two with an SFF-8088 cable.

Just be careful to get the HP SAS Expander with the green pcb board, the older yellow ones can not be firmware upgraded.

This was a very scaled down introduction into HBA / Raid / SAS cards/expanders, there are many options to use and many different scenarios to work through but i believe this covers the top 3, internal connections, external connections and both internal and external connections.

-Jared

Unraid Build for Plex Automation

Hardware List

The Unraid Host
aka
Building Skyros

Rosewill RSV-L4312 - 4u Rackmount Chassis

AMD Ryzen 5 3600X 6-Core, 12-Thread

G.SKILL TridentZ RGB Series 32GB (2 x 16GB) DDR4 3200Mhz

MSI X570-A PRO Motherboard

EVGA 650 GQ, 80+ GOLD 650W

LSI 9200 series External HBA

QSFP+ / Mini SAS Cable

Disk Shelf

NetApp DS4246

I won't be going over the specifics of building out the physical machine, the important parts to take away from this will be that im using an externally connectable LSI HBA card, using the cable linked just below it, and the disk Shelf. With this combination, you can run the single cable from the back of the Unraid host to the back of the NetApp DiskShelf and see the drives in a JBOD (just a bunch of disks) format.

Below is a quick picture of the resulting setup. One "Compute" server running Unraid, connected to a disk shelf.

Docker Setup

I will be focusing on the following:

Plex

Sonarr

Radarr

Nzbget

To start, this can be done several ways, this is what i landed on as being the fastest way to ingest new media.

Setup a few directories how ever you see fit, this is how mine is.

Under a share i created named Media i have the following:

You can ignore any folder not associated to a Movie or TV Show genre such as transcode, tmp, nzb etc

Some folders to house the retrieved media, as well as a complete, and an incomplete folder, the complete folder having sub folders for movies and shows.

Lets start with NZBGET, this is what a lot of your dockers will be talking to. Here is my container settings page:

Before i go any further let me make it clear, for usenet services, you need both an indexing service (to find the media) and a server (to download the media)

There are guides on reddit and lists suggesting which sites to use. Personally i use NZBGEEK as an indexer and Newsgroup Ninja as a server.

Lets continue..

Configure your Server, port, and connection type, i recommend TLS/SSL connections

For the connection count, i set it to one, determined the average speed, and then figured out how many connections at average speed would i need to saturate my network and used that number, you can adjust the live speed later on.

And now the Paths settings

I believe the above settings will be ignored while the below categories are used.

I have yet to configure my NzbDir, Queue, Temp and Web dir, which is why you see those folders earlier in one of the images, i plan to move these to clean up my Media folder.

Now we need to setup Categories, this is how we will differentiate Movies from TV Shows.

Movies:

TV Shows:

Please google around for other minor adjustments that can speed up Nzbget.

Next, Sonarr, nothing special as far as setup goes, here is my container settings page.

Sonarr and Radarr both have built in support for Nzbget so the setup is easy, the main thing to focus on is making sure the category name you gave to Nzbget is the same one typed in the Sonarr settings, for some reason i went with all lower case, but i believe default is to have the first letter uppercase.

Sonarr Indexer settings

I also have my drone factory setup, though it isn't needed.

Now we will do a similar process in Radarr

I know its probably not correct, but i have my Drone Factory turned on....i don't recommend doing this.

I also have Remote mappings setup, thought i don't believe they are needed in my case, i will include them though.
The Host is the IP of where i want to connect, the remote path that NZBGET saves files, and the local path of how this docker / server will reach the remote path.  This will be needed in scenarios where your container paths have different directories defined, or using an external machine to download media.

All you need to do now is make sure everything is turned on and add some movies and shows to your list to download.

That is it for this post, i will continue with setting up Plex at another time.