Build a Farm: Hardware considerations -

Null

Ooperator
Patriarch
kiwifarms.net
I've tried every optimization. Here's how fucked it is.

I've spent all morning implementing an X-Accel-Redirect header in XenForo. I learned about this during Infinity Next.

Basically, if you use PHP for sending images, you're going to have a bad time. Literally the exact fucking opposite of web scale. For some reason, XenForo does this. It's become apparent XenForo is not optimized for large sites and especially not for large sites that want to cache content.

So I set to work getting this feature in XF2 and by the end of it, proxy.php is so fast that the new iowait bottleneck is on nginx and the entire server is down, effectively DoS'd by early morning traffic.

fuck this, I'm going to buy a server tomorrow and hope the merch run finances it, we can't wait anymore
 

DNA_JACKED

kiwifarms.net
I've tried every optimization. Here's how fucked it is.

I've spent all morning implementing an X-Accel-Redirect header in XenForo. I learned about this during Infinity Next.

Basically, if you use PHP for sending images, you're going to have a bad time. Literally the exact fucking opposite of web scale. For some reason, XenForo does this. It's become apparent XenForo is not optimized for large sites and especially not for large sites that want to cache content.

So I set to work getting this feature in XF2 and by the end of it, proxy.php is so fast that the new iowait bottleneck is on nginx and the entire server is down, effectively DoS'd by early morning traffic.

fuck this, I'm going to buy a server tomorrow and hope the merch run finances it, we can't wait anymore
I still think a Xeon E-2288G is going to be your best bet for a new server CPU, with a 3.7 base and 5.0 turbo. There is a 4.0 base xeon 6 core option as well, but that about as fast you are going to get single core performance in a server today.
 

Null

Ooperator
Patriarch
kiwifarms.net
I still think a Xeon E-2288G is going to be your best bet for a new server CPU, with a 3.7 base and 5.0 turbo. There is a 4.0 base xeon 6 core option as well, but that about as fast you are going to get single core performance in a server today.
There are some 1U dual sockets that are available.
 

Slav Power

Tag jes.
kiwifarms.net
If there was some kind of forum software that would be much more optimized and SEO friendly, how hard would it be to port the entire forum to it, if such thing would be even possible in the first place?
 

DNA_JACKED

kiwifarms.net
Don't mobos with 2 sockets instance a single OS and stack?
Yes they do. I was running off your comment you wanted high single threaded performance for PHP, and all the dual socket servers I could find new for under $5K have 8 core CPUs that top out at only 4-4.2 GHz boost, with lower base clocks, compared to the 5GHz boost features on Xeons in single socket servers.

Of course, a single dual socket server could host twin VMs, one for each of your current servers. There are plenty of free VM applications that would work for hosting, and it would make system restore easier then running on the metal does.

I'd still recommend a R240 from dell, with a Xeon E-2288G, which will give you the single threaded performance you want for PHP, possibly buy two R240s for less then you can get a single dual socket server, then use your current box for hosting VM instances that you could bounce between the two new servers for better uptime, specially in case of hardware failure or sudden spike in load.
 
Last edited:
  • Informative
Reactions: JoshPlz

Null

Ooperator
Patriarch
kiwifarms.net
Yes they do. I was running off your comment you wanted high single threaded performance for PHP, and all the dual socket servers I could find new for under $5K have 8 core CPUs that top out at only 4-4.2 GHz boost, with lower base clocks, compared to the 5GHz boost features on Xeons in single socket servers.
I can't find any processors over 4GHz that aren't $10k. What website are you using?

I'm most mystified by what HD setup to get. I don't really know the difference between NVMe / SAS / SATA. I thought I wanted to get 2 RAID10s, one of 4x256GB SSD and another of 4x2TB HDD with HW RAID, but I have no fucking clue anymore.
 

DNA_JACKED

kiwifarms.net
I can't find any processors over 4GHz that aren't $10k. What website are you using?

I'm most mystified by what HD setup to get. I don't really know the difference between NVMe / SAS / SATA. I thought I wanted to get 2 RAID10s, one of 4x256GB SSD and another of 4x2TB HDD with HW RAID, but I have no fucking clue anymore.
None of Intel's xeon processors come with a base clock over 4.0 GHz. They all rely on boost to hit those higher clocks. Server configuration sites dont do a good job of explaining what boost clocks look like, I've been looking up CPU boost stats from intel's website individually. Xeons with metal names (xeon bronze/silver/gold) are more aimed at parallel tasks, and favor cache over clock rates. Stick to numbered xeons (Xeon E-2288G for instance) for higher per core performance.

If you are not bandwidth limited on the storage space, NVMe will be an expensive waste of time. The big difference is speed. SAS/SATA use the same protocol, you can use SATA drive on a SAS port (but not the other way around). SAS/SATA is limited to about 550MB/s in real world use per drive, NVMe can hit as high as PCIe allows, PCIe NVMe hits around 3-3.4GB/s max, where PCIe NVMe hits right now around 6.5GB/s.

I would recommend you stick with HDDs for storage, given how large the site is and how expensive SSDs are at larger capacities. A 2TB SSD will run around $200-250 for a consumer model, figure around $100 for a 1TB consumer model, higher prices for enterprise drives.

Most servers dont have native NVMe drive support. The name of the game today is using NAS full of HDDs that host VMs that run on host servers outfitted with a single SSD to hold the host OS. If we are running on a single HDD now for hosting data storage, even a RAID 5 of 4 HDDs would be a massive boost to performance. If you felt like spending soem cash, 4x2TB SATA III SSDs would provide upwards of 1.5GB/s performance in RAID 5, but the constant read/writes will wear them out quicker. I have no idea what your daily read/write rates are on the current machines.
 

Null

Ooperator
Patriarch
kiwifarms.net
Xeons with metal names (xeon bronze/silver/gold) are more aimed at parallel tasks, and favor cache over clock rates. Stick to numbered xeons (Xeon E-2288G for instance) for higher per core performance.
God, is that why the processors are so expensive and suck? Thanks for that
 
  • Agree
Reactions: JoshPlz

DNA_JACKED

kiwifarms.net
God, is that why the processors are so expensive and suck? Thanks for that
Yes, precious metal Xeons come in higher core count and higher cache varients, along with a quad channel memory controller. Precious metal Xeons are based on the same tech that goes into Intel's HDET line.

Their numbered Xeons are amped up versions of the arch/chipset tech used in their consumer i3/5/7 line. These servers are even compatible with consumer chips, the dell x240 can be had with celeron, core i3, or xeon chips. These have dual channel memory controllers and smaller cache setups meant for quicker access.

For your purposes, stick with numbered Xeons.
 
  • Like
Reactions: JoshPlz

DNA_JACKED

kiwifarms.net
Do Xeons rule out all dual socket mobos? i.e. I can't have two E-2288G?
Yes. The consumer based Xeon chipsets are single socket only. If there is a dual socket version, I havent seen it, it would require twin chipsets and custom firmware that Intel would not support.

Also, here is a breakdown of numbered xeon clock speeds, turbo speeds, and turbo bins per cores used:

table1.png
table2.PNG
 

Coffee Anon

kiwifarms.net
Monk determined that the CPU is not the bottleneck, but the HDD seemed to be. If it's an issue with need for quick random access (i.e. requiring low-latency, not necessarily high-throughput), then a high capacity SSD might be the only solution required.
The Samsung 860 PRO has a good review:

(4 TB model is $930 on Amazon)

(2 TB model is $490)

The EVO model has a comparitively bad review, even though it's cheaper, so I wouldn't go with that.
To the point where I wonder if they got a faulty drive.

The PRO also comes with a limited 5 year warranty which is nice.

Note that this is still considered a CONSOOMER drive, not ENTERPRISE™, but those high end SAS SSDs can be pricey if you want top tier performance.

($3540 for 1.6 TB! Hard to believe. I wonder if this is some sort of scammy reseller getting 300% markup?)
 
  • Like
Reactions: Oban Kamz

DNA_JACKED

kiwifarms.net
Monk determined that the CPU is not the bottleneck, but the HDD seemed to be. If it's an issue with need for quick random access (i.e. requiring low-latency, not necessarily high-throughput), then a high capacity SSD might be the only solution required.
The Samsung 860 PRO has a good review:

(4 TB model is $930 on Amazon)

(2 TB model is $490)

The EVO model has a comparitively bad review, even though it's cheaper, so I wouldn't go with that.
To the point where I wonder if they got a faulty drive.

The PRO also comes with a limited 5 year warranty which is nice.

Note that this is still considered a CONSOOMER drive, not ENTERPRISE™, but those high end SAS SSDs can be pricey if you want top tier performance.

($3540 for 1.6 TB! Hard to believe. I wonder if this is some sort of scammy reseller getting 300% markup?)
How much data do these drives write monthly? If the random read is way too low on a HDD, but the total amount written is also comparatively low, the crucial MX 500 is half the price and also features a 5 year warranty (but has 1/3rd the TBW rating, for what thats worth, so if we write massive amounts of data the 860 pro would likely be better).

MX 500 2TB $209: https://www.amazon.com/Crucial-MX500-NAND-SATA-Internal/dp/B078C515QL/ref=sxin_2_ac_d_pm?ac_md=4-2-QWJvdmUgJDIwMA==-ac_d_pm&cv_ct_cx=crucial+MX500&keywords=crucial+MX500&pd_rd_i=B078C515QL&pd_rd_r=36a47d67-4feb-45b1-abd5-e70a26dd22f4&pd_rd_w=Zr0YA&pd_rd_wg=q7qXT&pf_rd_p=0e223c60-bcf8-4663-98f3-da892fbd4372&pf_rd_r=B36NPRDKDZRH757Z33MW&psc=1&qid=1581012098&sr=1-3-22d05c05-1231-4126-b7c4-3e7a9c0027d0
 
  • Like
Reactions: Oban Kamz

Null

Ooperator
Patriarch
kiwifarms.net
Discuss

Thinkmate® RAX-1208-SH 1U Chassis - 8x Hot-Swap 2.5" SATA/SAS3 - 600W Single Power
Intel® C246 Chipset - 6x SATA3 - 2x M.2 - Dual Intel® 1-Gigabit Ethernet (RJ45)
Six-Core Intel® Xeon® Processor E-2286G 4.0GHz 12MB Cache (95W)
2 x 16GB PC4-21300 2666MHz DDR4 ECC UDIMM

LSI MegaRAID 9341-8i SAS 12Gb/s PCIe 3.0 8-Port Controller
4 x 240GB Intel® SSD D3-S4510 Series 2.5" SATA 6.0Gb/s Solid State Drive
4 x 1.8TB SAS3 12.0Gb/s 10000RPM - 2.5" - Seagate Exos 10E2400 Series (512e/4Kn)

1581013202677.png


CONFIGURED PRICE:
$4,160.00

($377/mo)
 

DNA_JACKED

kiwifarms.net
Discuss

Thinkmate® RAX-1208-SH 1U Chassis - 8x Hot-Swap 2.5" SATA/SAS3 - 600W Single Power
Intel® C246 Chipset - 6x SATA3 - 2x M.2 - Dual Intel® 1-Gigabit Ethernet (RJ45)
Six-Core Intel® Xeon® Processor E-2286G 4.0GHz 12MB Cache (95W)
2 x 16GB PC4-21300 2666MHz DDR4 ECC UDIMM

LSI MegaRAID 9341-8i SAS 12Gb/s PCIe 3.0 8-Port Controller
4 x 240GB Intel® SSD D3-S4510 Series 2.5" SATA 6.0Gb/s Solid State Drive
4 x 1.8TB SAS3 12.0Gb/s 10000RPM - 2.5" - Seagate Exos 10E2400 Series (512e/4Kn)

View attachment 1133069

CONFIGURED PRICE:
$4,160.00

($377/mo)
Looks like a good setup.

Are you building this from Thinkmate’s site, or elsewhere? I don’t know... it’s solid enough, I just feel as if it’s egregiously overpriced, even at that monthly price point.
You have to figure the server itself, from someone like Dell, would be 1100-1200$, with only 8GB of RAM and 1 HDD. This thing has 32GB, quad SSDs and quad 10K HDDs with a hardware SAS RAID controller. That ain't cheap.
Thinkmate. What would you suggest? I checked ebay for that processor and didn't find much.
I'd be amazed if you could find them at all, given these chips came out 3 months ago.
 
Tags
None