My internet connection is getting upgraded to 10 Gbit next week. I’m going to start out with the rental router from the ISP, but my goal is to replace it with a home-built router since I host a bunch of stuff and want to separate my out home Wi-Fi, etc onto VLANs. I’m currently using the good old Ubiquiti USG4. I don’t need anything fancy like high-speed VPN tunnels (just enough to run SSH though), just routing IPv6 and IPv4 tunneling (MAP-E with a static IP) as the new connection is IPv6 native.
After doing a bit of research the Lenovo ThinkCenter M720q has caught my eye. There are tons of them available locally and people online seem to have good luck using them for router duties.
The one thing I have not figured out is what CPU option I should go for? There’s the Celeron G4900T (2 core), Core i3 8100T (4 core), and Core i5 (6 core). The former two are pretty close in price but the latter costs twice as much as anything else.
Doing research I get really conflicting results, with half of people saying that just routing IP even 10 Gbit is a piece of cake for any decently modern CPU and others saying they experienced bottlenecks.
I’ve also seen comments mentioning that the BSD-based routing platforms like pfSense are worse for performance than Linux-based ones like OpenWRT due to the lack of multi-threading in the former, I don’t know if this is true.
Does anyone here have any experience routing 10 Gbit on commodity hardware and can share their experiences?
You must log in or register to comment.
Here are some results if anyone comes across this thread in the future.
The baseline result I need to achieve is a speedtest result of 7.5 Gbit that the ISP’s rental router gives me.
I ended up picking up:
- Lenovo ThinkCentre Tiny M720q, Core i3 8100T, 8 GB RAM ($70)
- Huawei SP310 (Intel X520-DA2/82599) dual 10 Gbit NIC ($20)
- 16x PCIe riser + Network Card Bracket ($20)
Initially I installed pfSense. I ran iperf3 to just get an initial sanity check that the PCIe card/wiring was working right but was getting results between 3-7 Gbit with the CPU pegging at 50%. Some quick googling returned results like “you can’t run iperf on pfSense!” and “pfSense isn’t a router, why do people keep using it as a router, it’s a firewall!”, so I decided to switch to OpenWRT since the Linux side of things always seems to make more sense.
On OpenWRT, iperf easily hit 9 Gbit with like the CPU at 95% idle.
It took like 2 hours to configure the weird IPIP6 tunnel my ISP uses for IPv4, but once it was set up, the machine has no trouble routing the same 7.5 Gbit speedtest the ISP router managed, with the CPU usage at 78% idle (the remainder in “sirq”)
Power consumption:
- ISP router draws a solid 16W both when it’s idle and when there is 7.5 Gbit of traffic
- The M720q draws 16W when idle and up to 29W when there is 7.5Gbit of traffic. This is with two copper SFP+
- This is without tweaking any power saving options in BIOS etc
I am saddened to see that this thread had no mention of how many horses it takes to run a router. What do y’all think? Would one be enough? It would need to work in shifts to keep up time at 100%. Maybe 3 to be safe?
This is why I came here. I think you’d need at least three. One to work while the other sleeps, and a spare in case one gets injured.
3 horses = 3 horsepower, which translates to a whopping 393.6 Duckpower.
Honestly, why are we still using horses as the standard here? Ducks are clearly the superior metric. So if you’re like me and prefer a more feathered approach, just remember:
3 horses = 3 horsepower = 393.6 ducks You’re welcome.
(PS: Just imagine 393.6 ducks handling 10Gb… now that’s efficiency.)
(quoting from wikipedia) In 2023 a group of engineers modified a dynometer to be able to measure how much horsepower a horse can produce. This horse was measured to 5.7 hp (4.3 kW)
Hmm, is that waddling or flying power? Swimming?
Also, the only reason for the 3 horsepower is so the others can rest, so we’d probably need far fewer than 393.6 ducks, I think we could get away with <100, provided we can manage their sleep cycles properly.
Alright, let’s get into the nitty-gritty of Duckpower.
First, let’s settle the “waddling vs. flying vs. swimming” debate. Horses aren’t big on flying, so we’re talking waddling power here. Until someone locates a Pegasus, we’re limited to the traditional land-bound horsepower. If you want swimming power, I guess you’d need to measure a seahorse?
Now, here’s where it gets serious: according to the brilliant minds at Art of Engineering, we can calculate Duckpower using a clever formula. They took the mass of a duck, compared it to a horse, and ran it through Kleiber’s Law. The answer? One horsepower = 131.2 Duckpower. So, back to our math:
3 horsepower = 3 x 131.2 Duckpower = 393.6 ducks waddling their hearts out.
But wait! We probably don’t need all 393.6 ducks if we give them some solid shift schedules. Horses only get 3 HP so two can rest; following this logic, we’d only need around 100 well-rested ducks, provided they get naps and stay hydrated.
So, let’s optimize our duck workforce with a shift schedule. Assuming we only need 100 ducks, here’s the plan:
Duckpower Shift Schedule:
Total Ducks: 100
Working Ducks per Shift: 25
Shift Duration: 2 hours on, 6 hours off (plenty of time for snacks and naps)
In a day, we’d run 4 shifts like this:
-
Shift 1: 25 ducks start strong at 8:00 AM, waddling with purpose.
-
Shift 2: Fresh 25 ducks take over at 10:00 AM while Shift 1 ducks hit the ducky lounge for snacks and a nap.
-
Shift 3: At 12:00 PM, another 25 ducks clock in to keep those wheels turning.
-
Shift 4: Finally, at 2:00 PM, the last 25 ducks take over while the others catch up on R&R.
With this cycle, each duck works only 2 hours out of every 8, staying energized, waddling at peak efficiency, and ready for action.
TL;DR: 3 horsepower = 393.6 ducks waddling but if we set up a 4-shift system, we can pull this off with only 100 ducks working 2 hours each, plus snack breaks.
-
Considering a typical horse produces about 1-14 HP*, it’s possible to run the horse for several hours at a time, change a battery, and run an efficient router off of it each day.
*sauce: https://www.sciencefocus.com/science/how-much-horsepower-does-a-horse-have
We also need to consider the practical aspects. Who mucks after the horses? Who feeds them? Do we need a stall? Does it need to be air conditioned in the summer/winter?
Horsepower is a very rough “average” of work output over a given period of time. It doesn’t really account for spikes in load. For that we’ll have have to consider the torque. So the real question is, how many foot/pounds or newton/meters does OP need to handle 10 gigs of throughput?
Switches and routers are pretty low-power, so we could probably get away with some form of body heat -> electricity thing. Or a battery and put the horse on a treadmill every so often.
So we built big tanks with a lever system like a piston on your car. Fill the left piston with water. With a small hole in the middle. For flow. If you do the math right you get the horse to walk up the stairs and stand on that piston(header really) the water drops slowly all day forced out of the hole spinning a turbine translating to electricity, preferably a battery. Horse never has to go down stairs thankfully, just back up the stairs to the other side. Moving from one side~ 3 meters every 12 hours should do it.
Basically, horse bed one side. Horse day lounge area other
Ooh, sounds quite practical. Would work even better with a cow, and you’ll get milk out of the deal too!
Neither 10G multiport routers nor L3 wirespeed switches are low power. We’re looking at 100+ W to multiple hundred watts. In 1U these are rather screamy.
I think it was 66~ watts for my Layer 3 8 port 10Gb switch, and Router in use.
That’s pretty good. Which models are these?
Here are two somewhat reasonable routers that support 10G (via 2 SFP+ ports):
Both have max power draw under 50W, though I don’t know what they’d actually draw (would depend on how much traffic and whatnot).
And here’s a switch with 2 SFP+ ports with max draw of 11W: https://mikrotik.com/product/css610_8g_2s_in.
Interesting. Wonder how much horsepower they have in the L3 department.
If you connect via 10gbit PCIe extension cards it is often a question of how many PCIe channels the CPU has and if the mainboard you are using has these connected directly to the CPU or needs to pass them through the mainboard chipset which is much slower.
For a dual port card, you will want an 8 lane PCIe 3.0 slot connected to the CPU. Almost any desktop CPU will have enough lanes since you won’t be using a graphics card. You can get by with a 4 lane slot, but you won’t be able to max out both ports bidirectionally at the same time.
These ThinkCenter M720q machines I’m looking at all seem to have a single PCIe 3.0 8x card slot, regardless of the CPU, and that seems to be all that the Mellanox ConnectX cards need according to their spec sheets, so hopefully that is good.
I have 10Gbit and hunted that whale. But I didn’t build my own router. Electricity is $0.51 Kw/h. Ouch.
First, 10Gbit hardware is more available now than years ago, so you have more options. I started off with the router my ISP gave me. It worked, but it was 1Gbit. Not going to do for me. Plus, basic function was paywalled. Booooo! Snagged a broken Asus router and got it working great.
With IDS/IPS enabled, I get about 3.5Gbps. There is newer router tech today that looks interesting with fewer bottlenecks that would have been nice years ago, but not worth the upgrade right now.
My desktop hits about 2Gbps downloading Steam games/updates, but my partners desktop lags behind with SATA SSD storage. Definitely need NVME with that speed.
I will say my experience with 10Gbit Ethernet cards is not positive. I have a lot of intermittent disconnections and there are a lot of bugs vs 1Gbit switches. They do not like sharing with 2.5Gbit devices. I keep my server on 1Gbit connections. It’s plenty fast for my needs though.
The low power consumption is one of the reasons I was attracted to the ThinkCenter M720q devices. It definitely wouldn’t be worth it if I had to build some tower PC or run a Xeon server!
The ISP router I’m getting is 10 Gbit (on WAN and one LAN port, the rest are 1 Gbit), but the configuration seems limited and it’s a $5/mo rental tacked onto the bill.
I think I can live without IDS/IPS, in all the time I used it on UniFi, it never gave me any actionable info, so hopefully that helps me with performance.
That’s interesting about the 10Gbit ethernet cards. Is that with something like a Mellanox or some other card? My NAS is going to be stuck on 2.5 Gbit since it’s just a Synology.
$0.51/kWh?! Where on earth is that?!
Hah, that’s rookie numbers man…we just hit $1.2USD/kWh the other day during the “dinner rush” between 5pm-9pm
That’s insane! I pay a flat US$0.11/kWh, and if I wanted to go peak/off-peak it would be $0.15/0.06!
Yeah it’s pretty crazy…prices vary by the hour, and that was only the single most expensive hour in that period though, and it was way above normal. Normally it peaks around 0.35USD/kWh with normal daytime prices around 0.2USD/kWh and nighttime prices around 0.1USD/kWh.
My total electricity consumption in the 5-9pm period is only around 2kWh though, so despite high prices it’s not much money.
Core i3 is fine, celeron can route, but you don’t have as much headroom, or room for firewall rules, etc. Recommend Intel x520 or mellanox cx3 or newer, though the cx2 is perfectly fine.
The bs about bsd being slower is maybe 15 years old at best?
Bsd is a monster for routing.
Run 25gbe routing, still can get by on your 4 core, but I throw some serious xeons at it anyway.
Thanks for the Intel x520 recommendation, those are looking like a much better deal right now than the Mellanox cards I was looking at.
Glad to hear it about the BSD networking!
I’m still trying to avoid the Xeons for power consumption reasons, hehe, although it would be a lot more fun for sure!
Your uplink capabilities are way different than your actuality. Get the service first, do some measurements, then start planning.
Yeah I’m not ordering anything until I have the connection up and running, which is why I opted to rent the ISP router to begin with, but looking at results online that others on the same ISP have posted, I can probably expect up to around 7 Gbit real-world so I’ve been thinking that I will at least want something better than the standard 1 Gbit or even 2.5 Gbit stuff out there, hence why I’m trying to research what the hardware requirements actually are!
You need at least one 10G port which is a pain on the Lenovo. There is a 10G passively cooled Mikrotik with sufficient power available.
Very interesting, thanks for the links
I don’t know the answer, but I do know I’d at least start off looking for hardware with a dedicated ASIC for routing, not general-purpose PC hardware doing routing with the CPU.
The problem is that it all looks really $$$, even on the used market
But do those specialized hardwares also work with stuff like opnsense?
many people just buy junk like this https://www.amazon.com/Mikrotik-Router-Switch-CRS305-CRS305-1G-4S/dp/B08437RDM1 it’s cheaper in the long run.
You will need a good 10Gb nic, I have been using Intel nic’s if you use a Intrusion Prevention System that can eat away at the CPU, also more RAM helps like 8GB’s or more for IPS, I use 16GB’s for IPS + ZFS and a nice Switch can help a lot as it can do DNS and the works, more or less i use a firewall box to a Switch and use a Layer 3 Switch for routing, some can do 20+Gb’s routing.
Mikrotik make good hardware, what are you on about?
they do make some good hardware, just this one they cheaped out on and used 16MB’s of Storage, it really hobbled the device.
The product you linked is a cheap fanless 10G layer 2 switch. It’s ok for the price, as fanless 10G enterprise switches are hard to get used.
There are suitable 10G capable Mikrotik routers however. This one, for instance: https://www.amazon.de/MikroTik-RB5009UPr-S-IN/dp/B0BBW159WW If you want wirespeed 10G routing on two or more ports it’s going to get expensive and/or noisy fast. A good compromise is a single 10G port router in a router on a stick mode used with a cheap 10G layer 2 switch.
The one i linked is both a router and or switch, you can get cheap switches for like $300 that do really well.
Wire speed L2 in hardware is cheap, but layer 3 is not (and is typically limited to few k routes for campus type of switches). I have a Quanta LB 10G and a Brocade ICX 1G/10G/40G switches for lab use, which are hot and screamy but were cheap used. I would not trust software L3 implementations to not drop packets at high rates.
deleted by creator
I can only look at Mikrotik gear in jealousy since they don’t have a reseller here, so all that’s available are overpriced, un-warrantied gray imports…
