Performance

Experiments have moved more than 1 GB/s through midas, from many readout boards over 10 Gb/s Ethernet into the SYSTEM buffer, and from there to disk. The standard midas programs are rarely the limit. Getting there depends more on how the readout hardware, the network, the frontends and the computer are set up.

Where the time goes

Data passes through several steps on its way to disk: the readout hardware, the network, the frontend, an event buffer, often an event builder and another event buffer, then mlogger and the disk. Each step copies the data at least once, and the copies share the computer's memory bandwidth.

• Memory bandwidth is usually the first limit. If the computer can copy memory at 14 GB/s, it can make about 14 copies of a 1 GB/s data stream per second, for everything: the network card, the socket reads, the event buffers, compression and the disk. Avoid unnecessary copies, and choose computers with fast, multi-channel memory.
• CPU time is the next limit. Unpacking, checking and sorting data costs CPU time in the frontends and the event builder, and the logger needs CPU time for compression. Add up the CPU use of all the programs on the computer, not just the one you are optimising.
• Small units of data are expensive. Many costs are per packet, per system call, per lock or per event, not per byte. Most of the advice below comes down to handling data in larger batches.

To find the bottleneck, look at the CPU use of each program (e.g. with top), the event rates on the Status page, and how full each event buffer is on the Buffers page. A buffer that is always full means its readers are too slow. A buffer that is always empty means its writers are too slow.

Reading out the hardware

• Do expensive processing in the hardware. Zero suppression and similar data reduction are much cheaper in an FPGA than in the frontend. Embedded processors (e.g. ARM processors in FPGA chips) often can't keep up with even a 1 Gb/s link, so send the data straight from the FPGA logic to the network if you can.
• Plan for lost UDP packets. Most readout boards send UDP, as it is fast and much simpler to implement in an FPGA than TCP. But UDP packets are lost whenever any step can't keep up, and nothing resends them. Number the packets in the hardware, and mark the start and end of each event, so that the frontend can detect lost packets. The rest of this list is mostly about not losing them.
• Use jumbo frames. The per-packet overhead of 9000-byte packets is almost 10 times lower than that of 1500-byte packets.
• Use Ethernet flow control, or limit the traffic yourself. When many boards send data at once, a 10 Gb/s link can receive more data than it can pass on. Either enable Ethernet flow control everywhere (in the readout boards, every switch and the receiving network card), or only arm the trigger again once the previous event has been completely received. Not all switches have working flow control, so check that yours does (e.g. with the pause frame counters of the network cards).
• Choose the network hardware carefully. Some network cards and switches can't handle full-speed traffic, even if their data sheets say they can. Experiments have had good results with Intel network cards.
• Make the receive queues large. Increase the socket receive buffer size, and the queue sizes of the network card.

Writing frontends for high rates

• Read the hardware in a dedicated thread. If the same thread also writes to midas, it stops reading while it waits for the event buffer, writes statistics to the ODB or checks alarms, and the hardware or network loses data. Read the hardware in one thread, and send the events to midas from another, with a queue between them (e.g. a std::deque protected by a mutex, or a midas ring buffer). The TMFE framework was designed for this, and its poll thread runs independently of the rest of the frontend. mfe frontends can stall for a long time between calls to your readout function.
• Pass data between threads in batches. Locking a mutex for every network packet costs more than the packet itself at high rates. Collect many packets (e.g. 1000) before putting them in the queue, and take them out in batches too.
• Put many small pieces of data into one midas event. Each midas event has a fixed cost, so one event with 1000 network packets as separate banks is much cheaper than 1000 events.
• Use system calls that handle many packets at once, e.g. recvmmsg() to read many UDP packets in one call.
• Avoid memory allocations and copies for every piece of data. bm_send_event_sg() and bm_send_event_vec() send an event made of several separate pieces of memory, without first copying them into one block. In TMFE, use the EqSendEvent() variants that take a list of pieces.
• Split the work across several programs. One frontend can only use so much CPU time. Several frontends, each reading part of the hardware into its own event buffer, can be combined by an event builder. The event builder may need several threads too, e.g. one per input buffer. Separate programs are also much easier to debug than one large multithreaded program.

Python frontends

The Python frontend framework is slower than the C++ ones, because the data has to be converted between Python and C++. To make it as fast as possible:

• Store bank data in numpy arrays of the matching type. The data is then already in the format midas needs, and is copied without conversion. For example, use an np.int16 array for a TID_SHORT bank. midas.tid_np_formats gives the numpy type for each TID_xxx type. With Python lists, or numpy arrays of a different type, every value is converted, which can be about 20 times slower for large events.
• Set the equipment's period to 0, in /Equipment/<name>/Common/Period, so that your functions are called as often as possible.
• Return a list of events from your readout function, to send several events in one call.

Cache sizes

Writing an event to an event buffer locks the buffer, which can take a long time if other programs are using it. To reduce the number of locks, a frontend can collect events in a write cache, and copy the whole cache to the event buffer when it is full. This makes a big difference for high rates of small events. For large events it only adds an extra copy.

A few rules apply to the write cache size:

• The smallest write cache is 10 MB. Smaller sizes are increased to 10 MB, with an error in the message log.
• The largest write cache is a third of the size of the event buffer. Larger sizes are reduced, again with an error in the message log. So for a large write cache, increase the size of the event buffer too.
• Events larger than a tenth of the cache size are written straight to the event buffer, without using the cache.
• Events wait in the cache until it is full or flushed. So with a large cache and a low data rate, other programs (e.g. an analyzer or the Event Dump page) see the events late.

Programs that read many small events can use a read cache in the same way, with bm_set_cache_size(). mlogger uses a 1 MB read cache.

A frontend running on another computer sends its events to mserver over the network, and mserver writes them to the event buffer. The network connection then decouples the frontend from the event buffer, as long as the network buffers are large enough and the event buffer isn't congested.

Event buffer sizes

Event buffers decouple the programs that write events from the programs that read them, e.g. a frontend from mlogger. The larger a buffer, the longer the writers can continue while a reader is slow. A buffer should hold at least two of the largest events, and ideally 10 to 100 events. See Event buffers: configuring for how to change the buffer sizes and the maximum event size.

Writing to disk

mlogger can write more than 1 GB/s to a fast disk array, e.g. several hard disks in a RAID or ZFS array. To get the most out of it:

• Choose the compression carefully. Compression costs CPU time, which mlogger then has to share with the frontends and event builder. lz4 is the fastest compression, and is the default.
• Don't expect events to reach the file immediately. Events can wait in the event buffer, the compression buffers and the operating system's disk cache, so it can take seconds before an event is in the file. This is normal, and doesn't slow down the data taking.
• Reduce the amount of data early, in the hardware or the frontend, rather than writing data that will be thrown away later.