[zfs-discuss] Cheap arm64 development board

Gordan Bobic gordan.bobic at gmail.com
Wed Dec 28 07:03:55 EST 2016

On Wed, Dec 28, 2016 at 11:44 AM, Durval Menezes <durval.menezes at gmail.com>

> Hi Gordan,
> On Wed, Dec 28, 2016 at 9:33 AM, Gordan Bobic via zfs-discuss <
> zfs-discuss at list.zfsonlinux.org> wrote:
>> On Wed, Dec 28, 2016 at 11:21 AM, Durval Menezes <
>> durval.menezes at gmail.com> wrote:
>>> Hello,
>>> On Wed, Dec 28, 2016 at 8:45 AM, Gordan Bobic via zfs-discuss <
>>> zfs-discuss at list.zfsonlinux.org> wrote:
>>>> On Wed, Dec 28, 2016 at 10:38 AM, Romain Dolbeau via zfs-discuss <
>>>> zfs-discuss at list.zfsonlinux.org> wrote:
>>>>> 2016-12-28 6:28 GMT+01:00 thekingmen via zfs-discuss
>>>>> <zfs-discuss at list.zfsonlinux.org>:
>>>>> > I found an arm 64 bit development board with ample ram that does not
>>>>> cost
>>>>> > 300$ like all the others. The pine64 is only 29$usd.
>>>>> There's a lot of choice of A53s with neither ECC nor SATA for
>>>>> $30-$100. They might not be a good fit for ZFS, since ECC is
>>>>> considered by many a must...
>>>>> Since the AMD A1100 SoC is nowhere to be found, and the X-Gene boards
>>>>> are quite expensive (but at least they exist), the only one I found so
>>>>> far is a development board:
>>>>> <https://www.solid-run.com/marvell-armada-family/armada-8040
>>>>> -community-board/>.
>>>>> More expensive than the plethora of basic Aarch64 SBC, but you get
>>>>> A72s, ECC, SATA and 10 GbE. At least that's the claim, I haven't got
>>>>> one yet.
>>> Good find, that looks like an awesome aarch64 board. They only seem to
>>>> list u-boot, though. Let's hope that Tianocore UEFI firmware also becomes
>>>> available when the product ships.
>> To correct myself, they DO list UEFI. Let's hope it's sensibly priced.
>>> +1. Albeit it has way too much network interfaces and not enough USB 3
>>> (nor SATA ports)...
>> I'm pretty sure there are 3 SATA ports visible in the picture.
> I saw them. But 3 SATA is hardly enough... 4 would be the bare minimum,
> unless one goes the SATA-multiplier way, and I would not be the first to
> test SATA multiplier compatibility on an ARM board with an untested
> controller...

PMP compatibility is patchy at best on x86, I haven't found ARM to be
particularly worse. I'm not sure about the very latest generation of Intel
SATA controllers, but they traditionally haven't supported FBS, only CBS,
which are unusably slow for most loads.

AMD SB600 PMP support was also at best CBS, IIRC.

Marvell 9123 (IIRC) had a _bizzare_ problem where one PMP would work (2
ports), but 2 would not. At most you could have 1 disk on one port, and a
PMP with however many disks that supports on the other, but combining it
with a JMB 6Gb SATA PMP would error out all over the place under load,
while the Silicon Image 3Gbit PMP would work fine (both FBS, though, which
is at least something).

The most reliable combination of controller and PMP I have found is Silicon
Image 3132 (yes, I know it's painfully slow, it tops out at something like
170MB/s across both ports), and a Silicon Image 3Gbit PMP. Yes, the
bandwidth is poor, but at least it's FBS and for non-sequential workloads
with spinning rust, 170MB/s is actually quite a lot.

And there is that PCIe port on the board that you could drop a nice 24-port
SAS controller into. :-)

> I would gladly exchange those 4 network interfaces for a single 10Gbit
>>> copper-only ethernet
>> I didn't think 10Gb copper is a "thing" yet, mainly because of cost and
>> power requirements.
> I have used it successfully with Sun, er Oracle T4-2 SPARC-based hardware
> a couple of years ago, connected to Brocade switches. And it also fell back
> to 1Gbps with no issues.

I know it's possible, but the cost of just the NICs would probably double
the target price of the board. And then there's the cost of the switches.
If you want tons of network bandwidth on the cheap, 2nd hand Infiniband kit
is the way forward. It's dirt cheap, as long as you can live with the 50ft
maximum cabling distance without repeaters (and then it gets _very_

>> and the other three in additional USB3 ports (or, now that we're dreaming
>>> anyway, a pair of  Thunderbolt 3 ports). OTOH it has a PCIEx4 port, so a
>>> 4-port USB3 expansion port (or a pair of Thunderbolts) perhaps aren't too
>>> far-fetched...
>>> Two very important things are still unsaid for this board, tho: ETA and
>>> $$$..
>> Well, to put it into perspective:
>> This has twice as many cores that are 20% faster, and it has 8x the DIMM
>> slots, and 2x the PCIe slots:
>> https://www.xcase.co.uk/products/gigabyte-mp30-ar0-with-
>> appliedmicro-x-gene1-processor
> Ouch, GBP 450 plus VAT for just the mobo plus CPU.
> This is a lot of dough just to get kinky with ARMs... At this price I
> would rather stay orthodoxally x86 and get one of the latest, low-power
> Xeons... it would be great if Intel would make a Xeon-based, low priced
> NUC...

Don't underestimate what Xeons cost, though. I have a maxed out G8 HP
microserver, and although the machine itself was only about £100, putting
in a top of the line CPU (E3 1290v2) was about £300 extra on top, and
between it's 4 cores, it gets beaten by the 8 X-Gene cores on the MP30 ARM
board in server tasks and non-FPU heavy workloads (e.g. compiling). The
Xeon wins by a large margin on heavily optimized SIMD workloads (e.g.
libx264 encoding), but that's a pretty narrow use case. And the X-Gene has
a TDP of just over half.

So while it's not for _everyone_, the TCO/performance is already in ARM
chips' favour.

> So I would expect the solid-run board to be somewhere in the < £200 price
>> bracket.
> Considering it has ECC, that isn't too unreasonable, provided it comes
> with the processor (as I understand it does) and only needs the RAM to
> start singing the ZFS tune...

Yes, the CPU looks to be soldered onto the board. For me a DIMM socket
instead of RAM soldered onto the motherboard is a big selling point. With
the 32-biit ARMs it wasn't as big a deal when most of them supported at a
push 3.5GB of usable RAM, and the decent boards had the full complement
soldered on. Not to mention that it actually appears to be in ATX form
factor, which is also a big win for those of us who like being able to use
standard cases and PSUs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20161228/10ee6cb7/attachment.html>

More information about the zfs-discuss mailing list