Add 10gbps ethernet post
|
@ -0,0 +1,193 @@
|
|||
+++
|
||||
date = 2020-11-18
|
||||
title = "10GB Ethernet with Proxmox and Ryzentosh 3700X with macos Catalina"
|
||||
tags = ["proxmox", "nas", "storage", "homelab", "10gb", "ryzentosh", "hackintosh"]
|
||||
description = "Yeah. This just happened. I now have setup 10GB Ethernet between my Ryzentosh and Proxmox NAS / Home server."
|
||||
+++
|
||||
|
||||
_Please read my previous post on my DIY NAS here: [1](https://www.devroom.io/2020/02/28/building-a-diy-home-server-with-freenas/) and [2](https://www.devroom.io/2020/11/12/the-big-diy-nas-update/)._
|
||||
|
||||
_10 gigabit_. That is _10,000 megabit_.
|
||||
|
||||
I come from a time when fast internet meant you had ISDN. For reference, that's 192kbps, or roughly 24KB/s.
|
||||
|
||||
For the past eight years I've had CAT-5E / CAT 6 installed through-out my house and have enjoyed LAN speeds of 1000mbps or 1gbps. At full throttle that ways in at about 125 Megabyte/s.
|
||||
|
||||
Now, gigabit ethernet is fine for almost everything. Unless you either transfer large files or if you use that NAS as a direct access disk on a workstation.
|
||||
|
||||
The latter is pretty much my use-case. I make music and the occasional video, and storing _and_ manipulating all this over a 1gbps line is not so fun.
|
||||
|
||||
Yes, I _do_ have an SSD, yes I do have spinning rust in my workstation. The problem is that they lack redundancy and extra work to backup. Everything that's on my NAS profits
|
||||
from the redudancy of my ZFS pool, periodical ZFS snapshots, decent drive health monitoring (thanks, Proxmox!) and automated backups to another set of disks and the cloud.
|
||||
|
||||
When I found out that there's a _really_ cheap option to add 10 gigabit networking between my Ryzentosh and Proxmox NAS - I went for it.
|
||||
|
||||
Total cost of this project:
|
||||
|
||||
* $40 - 2x Solarflare SFN5122F shipped. yes, that is _two_ network cards, with _two_ 10 gbit ports each. For $40. Including shipping.
|
||||
* $20 - A SFP+ 10gbe copper cable (1 meter).
|
||||
|
||||
That. Is. It.
|
||||
|
||||
![](/img/nas-10gbe-solarflare.jpg)
|
||||
|
||||
## Proxmox setup
|
||||
|
||||
Okay, let's start with the easy part. Proxmox. Or Debian Linux, more specifically. It's just plug and play. ;-) Plug the card in any PCIe x8 slot and you're
|
||||
good to go.
|
||||
|
||||
![](/img/nas-10gbe-proxmox.png)
|
||||
|
||||
Next, you only need to configure your NIC. It's a good idea to create a `Linux Bridge` in Proxmox. This will allow you to assign this interface to containers as well,
|
||||
as I'll come back to in a minute.
|
||||
|
||||
You can pick any IP address you like in a private range. You don't need a gateway - well I didn't - because I have a point-to-point connection. Litteraly a cable
|
||||
between the two systems.
|
||||
|
||||
![](/img/nas-10gbe-proxmox-mtu.png)
|
||||
|
||||
Be sure to enable _advanced_ options and set `MTU` to `9000` - jumbo frames. This will enable you to reach those juicy 10gbps speeds later on.
|
||||
|
||||
## Hackintosh / Open Core setup
|
||||
|
||||
[Download Solarflare 10GbE Mac Drivers.zip](http://localhost:3000/files/macos/solarflare-10gbe-mac-drivers.zip)
|
||||
|
||||
_Solarflare is no longer supporting macos and has since removed these drivers from their website. The drivers they made still work in Catalina._
|
||||
|
||||
This is where things get a bit more tricky. I can't vouch for this working on a _real_ Mac, but then again, adding (non-Apple) hardware is not what you're
|
||||
supposed to do with these anyway, right? ;-)
|
||||
|
||||
Any modern hackintosh should run [Open Core] these days. [Open Core] is awesome. It takes a bit of work to setup, but your mac will continue working after that,
|
||||
updates and everything. So, the next bit assumes you're familiar with Open Core.
|
||||
|
||||
* Download the archive above. It contains a _signed_ installer for macos 10.9. Mount that disk image. Don't bother using the installer.
|
||||
* Use "View contents" on the installer and fine `Archive.pax`. Unpack that and find your kext and some `Application Support` stuff.
|
||||
* Drag folder under `Application Support` to `/Library/Application Support` if you need it.
|
||||
* Mount your `EFI` partition and drop the kext in the folder with your other kexts.
|
||||
* Then update your `config.plist` to load the kext - you can copy paste any other entry and change the filename. But again, if you're already running [Open Core], you probably know the drill.
|
||||
* Reboot
|
||||
|
||||
Next up, all you need to do is configure your NIC under Network Preferences. Again, pick an IP address in the same subnet as your server.
|
||||
|
||||
![](/img/nas-10gbe-network-1.png)
|
||||
|
||||
Make sure to select "Jumbo Frames (9000)" under "Hardware" here as well. Both sides need to have them enabled (or disabled).
|
||||
|
||||
![](/img/nas-10gbe-network-2.png)
|
||||
|
||||
## Testing speed
|
||||
|
||||
Okay, time to do some benchmarks. Let's break out `iperf3`.
|
||||
|
||||
* On Proxmox: `iperf3 -s` to start the server
|
||||
* On the Ryzentosh: `iperf3 -c 172.16.0.20 -f g`
|
||||
* Wait 10 seconds (for 11.2 GIGABYTES to be transferred)
|
||||
|
||||
```
|
||||
# iperf3 -c 172.16.0.20 -f g
|
||||
Connecting to host 172.16.0.20, port 5201
|
||||
[ 5] local 172.16.0.10 port 50113 connected to 172.16.0.20 port 5201
|
||||
[ ID] Interval Transfer Bitrate
|
||||
[ 5] 0.00-1.00 sec 1.12 GBytes 9.63 Gbits/sec
|
||||
[ 5] 1.00-2.00 sec 1.12 GBytes 9.64 Gbits/sec
|
||||
[ 5] 2.00-3.00 sec 1.12 GBytes 9.63 Gbits/sec
|
||||
[ 5] 3.00-4.00 sec 1.12 GBytes 9.63 Gbits/sec
|
||||
[ 5] 4.00-5.00 sec 1.12 GBytes 9.64 Gbits/sec
|
||||
[ 5] 5.00-6.00 sec 1.12 GBytes 9.63 Gbits/sec
|
||||
[ 5] 6.00-7.00 sec 1.12 GBytes 9.63 Gbits/sec
|
||||
[ 5] 7.00-8.00 sec 1.12 GBytes 9.63 Gbits/sec
|
||||
[ 5] 8.00-9.00 sec 1.12 GBytes 9.64 Gbits/sec
|
||||
[ 5] 9.00-10.00 sec 1.12 GBytes 9.64 Gbits/sec
|
||||
- - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
[ ID] Interval Transfer Bitrate
|
||||
[ 5] 0.00-10.00 sec 11.2 GBytes 9.64 Gbits/sec sender
|
||||
[ 5] 0.00-10.00 sec 11.2 GBytes 9.63 Gbits/sec receiver
|
||||
|
||||
iperf Done.
|
||||
```
|
||||
|
||||
You can also adjust the output formatting to give you bitrates in GBytes/sec instead:
|
||||
|
||||
```
|
||||
# iperf3 -c 172.16.0.20 -f Gbytes 1|2 ↵ 1005 18:50:53
|
||||
Connecting to host 172.16.0.20, port 5201
|
||||
[ 5] local 172.16.0.10 port 50155 connected to 172.16.0.20 port 5201
|
||||
[ ID] Interval Transfer Bitrate
|
||||
[ 5] 0.00-1.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 1.00-2.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 2.00-3.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 3.00-4.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 4.00-5.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 5.00-6.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 6.00-7.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 7.00-8.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 8.00-9.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
[ 5] 9.00-10.00 sec 1.12 GBytes 1.12 GBytes/sec
|
||||
- - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
[ ID] Interval Transfer Bitrate
|
||||
[ 5] 0.00-10.00 sec 11.2 GBytes 1.12 GBytes/sec sender
|
||||
[ 5] 0.00-10.00 sec 11.2 GBytes 1.12 GBytes/sec receiver
|
||||
```
|
||||
|
||||
That is 1.12GB or 1120MB per _**SECOND**_. Imaging that. That's a bit more than a full CD of data. Heck, that's about 1000 floppy disks per second. Okay, I'm claiming that. 1000fdps. :-)
|
||||
|
||||
## Samba
|
||||
|
||||
I run Samba in a LXC container on Proxmox. There's now only a few things to do for Samba to use that fat pipe of networking goodness.
|
||||
|
||||
First up, add another Network device. Pick another IP address in the private subnet and use the Linux Bridge you created earlier. You can add this _in addition_ to the network device you
|
||||
probably already have for your old and slow gigabit LAN.
|
||||
|
||||
![](/img/nas-10gbe-proxmox-lxc.png)
|
||||
|
||||
You may need to check `/etc/network/interfaces`. Proxmox probably updated it for you already, but it's missing the `MTU` setting. So, just add it and restart networking (or you're container, it's fast anyway) ;-)
|
||||
|
||||
```
|
||||
iface eth1 inet static
|
||||
address 172.16.0.22/16
|
||||
mtu 9000
|
||||
```
|
||||
|
||||
If you did it all correctly you should see:
|
||||
|
||||
```
|
||||
# ip link list
|
||||
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
|
||||
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
|
||||
2: eth0@if82: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
|
||||
link/ether 8a:b3:db:74:b9:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0
|
||||
3: eth1@if86: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
|
||||
link/ether ea:49:ef:01:82:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0
|
||||
```
|
||||
|
||||
Then you need to configure Samba to listen on both your gigabit and 10 gigabit interfaces. Check `/etc/samba/smb.conf`, there's probably a comment in there
|
||||
already about the `interfaces` flag. I simply added the following to enable both my interfaces.
|
||||
|
||||
`interfaces = 172.16.0.22/16 10.0.2.245/16`
|
||||
|
||||
Restart samba or reboot. And you're good to go.
|
||||
|
||||
## Accessing 10,000mbps Samba shares
|
||||
|
||||
You have to routes to your NAS / Samba shares now. One over your old gigabit network and one over your fast 10gbps point-to-point connection. Samba can handle both, so
|
||||
it's only a matter of accessing it correctly from macos.
|
||||
|
||||
Open Finder and press ⌘-K (or select Go -> Connect to Server...). Then you'll need to enter the proper URL for your Samba share like so. Remember to use the
|
||||
IP for your container - not the IP for your Proxmox host.
|
||||
|
||||
![](/img/nas-10gbe-samba-mount.png)
|
||||
|
||||
## That's all
|
||||
|
||||
That's all there is to it. Mount those shares and enjoy blistering fast transfer speeds. Your HDD's are once again the bottleneck now.
|
||||
|
||||
No, I'm not upgrading to a full SSD ZFS pool...
|
||||
|
||||
_**Let me help you**_
|
||||
|
||||
Are you planning on building a home server or NAS, but aren't you sure about the components you picked or if Proxmox / FreeNAS is right for you?
|
||||
|
||||
[Hop on over to BuyMeACoffee and plan a 30 minute call with me to go over the details!](https://www.buymeacoffee.com/ariejan/e/10602)
|
||||
|
||||
|
||||
[Open Core]: https://github.com/acidanthera/OpenCorePkg
|
BIN
static/files/macos/solarflare-10gbe-mac-drivers.zip
Normal file
BIN
static/img/nas-10gbe-network-1.png
Normal file
After Width: | Height: | Size: 141 KiB |
BIN
static/img/nas-10gbe-network-2.png
Normal file
After Width: | Height: | Size: 135 KiB |
BIN
static/img/nas-10gbe-proxmox-lxc.png
Normal file
After Width: | Height: | Size: 37 KiB |
BIN
static/img/nas-10gbe-proxmox-mtu.png
Normal file
After Width: | Height: | Size: 30 KiB |
BIN
static/img/nas-10gbe-proxmox.png
Normal file
After Width: | Height: | Size: 46 KiB |
BIN
static/img/nas-10gbe-samba-mount.png
Normal file
After Width: | Height: | Size: 63 KiB |
BIN
static/img/nas-10gbe-solarflare.jpg
Normal file
After Width: | Height: | Size: 3.4 MiB |