Choosing an open source EtherCAT master for embedded Linux
On the controller side (industrial PC, SBC…), EtherCAT keeps a simple promise: an ordinary Ethernet interface is enough, with no fieldbus card and no proprietary hardware. That is often the argument for picking this bus over another one, and it is the first thing the choice of MainDevice can preserve or break.
On a Linux controller, that choice is therefore not primarily a choice of library. It is a choice about what goes into the image, about what will have to be rebuilt on every kernel upgrade, and about the hardware the implementation still allows.
Two open source implementations are commonly used today, SOEM and IgH EtherLab. A third one, EtherCrab, is much more recent. All three are compared here, and the commercial products are listed at the end of the article.
Vocabulary
- MainDevice, SubDevice
Official terminology from the EtherCAT Technology Group since its renaming: MainDevice replaces master, SubDevice replaces slave. Both vocabularies coexist, and the documentation and source code of the implementations still use master and slave extensively. This article uses the current terms, except in the title.
- Kernel module, user space
A MainDevice implemented as a kernel module runs inside the Linux kernel. In user space, it runs as an ordinary process, in the threads of the application. The difference is paid for mostly at the time the image is built.
- Working Counter (WKC)
Counter incremented by every SubDevice that has processed the frame. Its value on return tells how many participants actually answered, and serves as a first diagnostic.
- Distributed Clocks (DC)
Hardware synchronisation of the bus. Each SubDevice aligns its clock on a reference clock, which requires the frame to arrive before a sync pulse, and not simply on time on average.
- Process image
Memory area exchanged cyclically, in which the application writes its setpoints and reads back states. Its layout is described by the PDO mapping.
Before choosing: the kernel and the network card
No implementation makes up for a badly chosen kernel, so two questions come before the comparison itself.
Does the silicon vendor BSP allow PREEMPT_RT? Since 6.12, PREEMPT_RT
has been part of the mainline kernel, which removes the old trade-off between
"up to date kernel" and "real-time kernel". But many BSPs are frozen on an
older version, sometimes with no RT patch available. Under Yocto, the
linux-yocto-rt recipe covers the generic case. A vendor kernel, on the other
hand, is worth checking before committing to it.
Which network card? A USB to Ethernet adapter has to be ruled out for the bus: USB polling jitter makes the measurements unusable. The trap is that some industrial PCs and SBCs on the market actually have their Ethernet ports behind an internal USB controller.
A native interface is required, directly on PCIe for instance, and it has to be dedicated to the bus, not shared with application traffic.
The MainDevice, on the other hand, does not care about the medium: a segment can run on copper or on fibre through a converter, with no consequence for it or for the application.
Three criteria
They are ordered by cost: the first one commits you for the maintenance of the whole product life, the second one commits the hardware, the third one is discovered during integration when it has not been checked beforehand.
What goes into the image
A MainDevice implemented as a kernel module is out-of-tree: it is built
against the kernel sources of the image and has to be rebuilt on every version
upgrade. Under Yocto, that means a module recipe tied to kernel-module-*, a
loading step to declare, and a hard dependency between two components that move
at different paces.
In user space, the MainDevice is a library or a dependency of the application. It knows nothing about the kernel version and is not rebuilt when that version changes.
User space does not make the system better in real time, it makes maintenance more predictable. On a product maintained for ten years, with security fixes to ship, the difference is paid for on every upgrade.
The network path, and what it demands of the hardware
The classic argument against user space is about the journey of a frame between
the card and the application. The historical path goes through a raw socket
(AF_PACKET), and its cost is paid on the way back from the softirq to user
space and on the system call exit.
Two more recent kernel mechanisms shorten that journey.
- io_uring
Submissions and completions go through rings shared between the kernel and the application, which removes one system call per operation.
- XDP and
AF_XDP An eBPF program intercepts the frame at driver level, before the
skbis allocated, and redirects it to anAF_XDPsocket read from user space. The network stack is bypassed, and the copy can be avoided.
These mechanisms narrow the historical gap between user space and kernel module, the very gap that justified the integration cost of an out-of-tree module.
They are not free, though. XDP requires libxdp, libbpf and careful
configuration, and above all, in native mode, a network driver that supports
it, which many do not. Generic mode works everywhere, but goes back through
the skb and loses most of the benefit.
Worth remembering when choosing hardware: the constraint on the controller does
not weigh on IgH and its native drivers only, it comes back as soon as you want
AF_XDP in user space. The AF_PACKET path, on the other hand, remains
available on any card.
What the implementation covers of the protocol
A MainDevice does not implement all of EtherCAT, and the scope is checked against the list of SubDevices to be driven.
Cyclic operation is not enough: configuring the SubDevices, updating their firmware and tunnelling protocols all go through separate mailbox protocols.
- CoE, CANopen over EtherCAT
Configuration through SDOs.
- FoE, File over EtherCAT
Firmware update over the bus.
- EoE, Ethernet over EtherCAT
Tunnelling Ethernet to a SubDevice, which some vendor configuration tools require. Typically to carry TCP/IP traffic through to a SubDevice.
- SoE, Servo over EtherCAT
Drive profile, an alternative to the DS402 profile carried over CoE.
Serial interface terminals, for instance, are configured through CoE: baud rate, parity and frame format are SDO objects. Without CoE, they stay inert.
The check comes down to three steps: list the SubDevices you have selected, note for each of them the mailbox protocols its use requires, then compare that list against the features each implementation announces. A missing feature is rarely worked around at application level.
One last point if custom boards are to be developed: the EtherCAT specification is obtained through membership of the EtherCAT Technology Group, which also issues the vendor ID. Both the membership and the vendor ID are free of charge, but the specification is only available on those terms.
The three implementations
SOEM
SOEM is a C library in user space, proven and widely deployed.
Integration. A recipe to write, simple because it has no kernel dependency:
neither Buildroot nor the common OpenEmbedded layers package it, only the ROS
layers do. From a Rust application, access goes through FFI (Foreign Function
Interface), so through an unsafe layer to write and maintain, which becomes
a component of the project in its own right.
Licence. It changed along the way. The 1.x versions are under GPLv2 with a linking exception, which allows linking with proprietary code. Version 2.0 is under GPLv3 or a commercial licence from RT-Labs, without that exception. For a closed product, the version you pick determines the licensing model.
The blocking architecture of SOEM also forces manual memory management and explicit locking.
IgH EtherLab
IgH EtherCAT Master is implemented as a kernel module, with the best real-time characteristics of the three. The code is on GitLab, and the documentation is published as a PDF.
Integration. It is the most expensive one, but it is packaged: Buildroot
provides igh-ethercat, which builds the module against the target kernel with
one option per native driver, and OpenEmbedded has an equivalent recipe in
meta-tanowrt. The
module is rebuilt on every kernel version upgrade.
IgH offers two families of network drivers. The native drivers, for specific chips (Intel I210 to I226, e1000e, r8169, among others), drive the card without interrupts and give the best performance, but they follow the kernel at the same pace as the module and constrain the choice of card. The generic driver works with any card supported by Linux, at the cost of going through the network stack.
With a native driver, the image therefore holds three coupled components to validate together: kernel, module and driver. With the generic driver, two.
Licence. The kernel modules are under GPLv2, the user space library under LGPLv2.1: a proprietary application can link against it dynamically. The constraint is on the maintenance of the module, not on the distribution of the application code.
EtherCrab
EtherCrab is a MainDevice written in Rust, running in user space, with no C dependency and no kernel module, under the MIT or Apache 2.0 licence. The API is documented on docs.rs.
The main developer covers several design points on his blog.
Integration. It only makes sense in a Rust application. The MainDevice is a
plain dependency there, declared in Cargo.toml and compiled with the
application. Nothing shows up separately in the image. Under Yocto, Rust has
been supported since 3.4: the cargo class handles cross compilation, and
cargo-bitbake generates a recipe that pins the crate index for
reproducibility. The io_uring and XDP paths described above are exposed as
build options, through Cargo features.
Reservation. The crate is at 0.x and its interface is still moving. On feature coverage, it announces CoE, autoconfiguration of SubDevices from their EEPROM and Distributed Clocks with drift compensation; EoE, FoE and SoE are not listed among the announced features.
The version is pinned in Cargo.toml, with the option of pointing at a git
revision or a local path to fix a defect without waiting for a release, which
remains the best recourse against a missing feature.
With those reservations stated, I have put it to work at several clients, on distributions built with Yocto.
Comparison
| EtherCrab | SOEM | IgH EtherLab | |
|---|---|---|---|
Execution space | user space | user space | kernel module |
In the image | nothing separate | library | module + network driver |
Kernel upgrade | no effect | no effect | module and driver to rebuild |
Constrains the network card | no with | no | no with the generic driver, yes with a native driver |
Integration from Rust | native | FFI | FFI |
Licence | MIT / Apache 2.0 | GPLv2 + exception (1.x), GPLv3 or commercial (2.0) | GPLv2 kernel, LGPLv2.1 user space |
Maturity | 0.x | proven | proven |
Summary
EtherCrab for a Rust application: nothing separate in the image, no kernel dependency, permissive licence. In exchange, a crate at 0.x whose version has to be pinned, and a protocol coverage to check before committing.
SOEM on an existing C codebase, with a simple recipe and no coupling to the kernel. Check first which version, and therefore which licence, you are committing to.
IgH EtherLab when the tightest cyclic behaviour is what matters and the team accepts maintaining a module at the pace of the kernel, plus a native driver alongside it if performance demands one, with the constraint on the card that comes with it.
In all three cases, the choice is not enough: determinism does not come from the MainDevice alone, it comes mostly from the system that runs it. Core isolation, interface settings, the design of the application loop, all of that remains to be done once the implementation has been selected.
The commercial products
They fall outside the scope of this comparison, but they exist and are worth knowing about. The list is not exhaustive.
- Acontis EC-Master
Commercial MainDevice, under a proprietary licence with a per-device royalty.
- Beckhoff TwinCAT
Beckhoff now offers a TwinCAT runtime for Linux: it is no longer limited to Windows and TwinCAT/BSD. That runtime does run on the Beckhoff Linux distribution, derived from Debian and shipped with a real-time kernel, first on the CX82x0 and CX9240 ARM embedded PCs, then progressively on their other IPCs.
TwinCAT is therefore not something you add to an image built with Yocto or Buildroot: the distribution and the hardware are adopted along with it. The decision is made at ecosystem level, not at recipe level, and the application does not own the cycle there, it talks to the runtime.
Note that the boundary is not watertight: SOEM 2.0 also offers a commercial licence alongside GPLv3.