Event data format

Introduction

A midas data file contains midas events. The structure of these of events is detailed on this page. The same structure is used both for passing events between different programs (using event buffers) and for writing the data to disk.

Midas data files are written by the mlogger program and end in the .mid extension. Files can optionally be compressed using gzip, lz4, bzip2 or pbzip2 (thus ending in .mid.gz etc).

Two event formats are supported by the frontends, although the vast majority of cases use the "midas" format:

Midas format events
FIXED format events

Note that a frontend cannot write data directly into ROOT format. A conversion to ROOT may be done by an analyzer (such as manalyzer) or by user-written scripts. Older versions of the data logger mlogger could also write ROOT files, but this was removed in midas-2026-07-a.

Overall file format

A midas file is simply a concatenation of midas events. There are no special headers or lookup tables, but it may contain a couple of special events (e.g. a dump of the ODB content).

Note that this means that if you want to jump to a certain event in a midas file, you need to parse the header of each event until you find the data of interest (i.e. read the header of the first event, decode the size of the event, jump forward that many bytes, read the header of the second event, ...). If your midas file is compressed, this means you must decompress the entire file up to and including your event of interest. This may be time-consuming (especially for gzip compression) and is a major reason why you should keep each midas file a reasonable size using subruns.

Midas format event

The midas event format is a variable length event format. It uses "banks" as subsets of an event. A bank is composed of a bank header followed by the data.

Structure of midas event showing Event and Bank headers with data banks.

Event header

Each event carries a 16-byte header that identifies which program generated the event, when it was written etc.

Definitions of the event header structure are found in:

Language	Structure/function	File
C++	`EVENT_HEADER` and associated macros	`include/midas.h`
C++	`TMEvent`	`midasio/midasio.h`
Python	`EventHeader`	`python/midas/event.py`
JS	`bkToObj()`	`resources/midas.js`

It has following structure:

Event ID: The event ID identifies which program produced the event (aka the type of the event). Care should be taken to ensure that all frontends in your experiment use different Event IDs, so you can more easily identify/filter your data during analysis. It is a number in the range 0-65535, and is the same for all events from an equipment in a frontend. The Event IDs 0x8000, 0x8001 and 0x8002 are reserved for special events.
Trigger mask: The trigger mask can be used to describe the sub-type of an event. For example, your experiment may trigger data readout for "physics events", "calibration events", "clock events" etc. Setting bits in the trigger mask to represent the source allows easier filtering of the data in mlogger or during data analysis. It is a number in the range 0-65535 (but is best used as a 16-bit mask), and can vary event-to-event.
Serial number: The serial number starts at 0 and is incremented by the frontend for each event that is written during a run. It is a number in the range 0-4294967295.
Time stamp: The time stamp is written by the frontend before an event is read out. It uses the time() function which returns the time in seconds since 1.1.1970 00:00:00 UTC.
Event data size: The event data size contains the size of the event in bytes excluding the header.

Byte-ordering

Event headers are always kept in the byte ordering of the local machine. If events are sent over the network between computers with different byte ordering, the event header is swapped automatically, but not the event contents.

If the byte ordering of the contents of a complete event has to be swapped, the bk_swap() function can be used.

Data area

The data area of each event is split into "banks". Each bank contains an identifying name, and then a data array. The data array may be of any type (integer, float, character etc).

Each bank can only contain one data type (e.g. an array of integers).
Different banks in the same event may have different types.

Each event contains a "global bank header" followed by one or more data banks. Each data bank has its own header followed by its data array.

The definitions of the global bank header and bank header are found in:

Language	Global bank header	Per-bank header	File
C++	`BANK_HEADER`	`BANK`/`BANK32`/`BANK32A`	`include/midas.h`
C++	`TMEvent`	`TMBank`	`midasio/midasio.h`
Python	`Event`	`Bank`	`python/midas/event.py`
JS	`bkToObj()`	`bkToObj()`	`resources/midas.js`

Global bank header

There is one global bank header per event. It contains the following fields:

All bank size: Size in bytes of the following data banks including their bank names
Flags: The lowest 4 bits indicate the version of the bank structures - this is currently "1" and used for endian detection (byte ordering). The 5th bit indicates that the bank header uses 4 bytes to record the bank size (BANK32 format). The 6th bit indicates that the bank data is 64-bit aligned (BANK32A format).

Midas data bank

Each data bank contains a header defined by the BANK, BANK32 or BANK32A structures. They all contain the same 3 fields, just with different alignments.

It has the following structure:

Bank name: Four characters for the name of each bank. Each bank in an event must have a unique name.
Bank type: One of the TID_xxx data types, showing how to parse the data that follows.
Bank length: Size in bytes of the following data.

Following the header is the data in the format designated by the Bank type parameter (e.g. integer or float). A separate midas bank must be created for each data type needed.

Bank alignment

The data area of a bank must have a size of multiples of 8 bytes. This is required to align the next bank header on an 8-byte boundary in memory, which allows faster access by the CPU. If a bank size is a few bytes short of a multiple of 8 bytes, the size is rounded up to the next multiple of 8 bytes and the remaining bytes in the bank may contain random data. Since midas event headers, bank headers (except the BANK32 bank header) and banks are all multiples of 8 bytes long, all data structures inside a midas event is 8-byte aligned for optimal CPU access. The BANK32 header is 12 bytes long, causing the following data area to not be 8-byte aligned. If 8-byte alignment is important, the BANK32A header must be used instead, which pads an extra 4 bytes to each bank header.

Example of midas event

Below is a dump of a midas event containing banks coloured to match the structure in Figure 1. This has been obtained from a midas data file using the mdump application. Colours match those in the diagram above.



Evid:000d- Mask:0000- Serial:0- Time:0x4c7a6869- Dsize:48/0x30

------------------------ Event# 2 --------------------------------

#banks:1 - Bank list:-SDAS-

Bank:SDAS Length: 32(I*1)/8(I*4)/8(Type)Type:Real*4 (FMT machine dependent)

 1-> 4.000e+00 1.000e+01 1.000e+00 3.400e+00 3.400e+00 3.400e+00 3.400e+00 3.400e+00

------------------------ Event# 3 --------------------------------

Evid:0001- Mask:0000- Serial:0- Time:0x4c7a686b- Dsize:344/0x158

#banks:2 - Bank list:-MPETMCPP-

Bank:MPET Length: 304(I*1)/76(I*4)/76(Type) Type:Unsigned Integer*4

 1-> 0x80010000 0x00000002 0x10010000 0x00004e21 0x80020000 0x00000002 0x20020000 0x000015f4

 9-> 0x20020000 0x00001660 0x20020000 0x0000185f 0x20020000 0x0000191e 0x20020000 0x000019d6

17-> 0x40020000 0x00001a37 0x20020000 0x00001a77 0x20020000 0x00001ba2 0x10020000 0x00004e22

25-> 0x80030000 0x00000002 0x20030000 0x00001637 0x20030000 0x000018d1 0x20030000 0x000019bc

33-> 0x20030000 0x00001b35 0x20030000 0x00001bb2 0x10030000 0x00004e21 0x80040000 0x00000002

41-> 0x10040000 0x00004e22 0x80050000 0x00000002 0x20050000 0x000013c5 0x20050000 0x000017f2

49-> 0x20050000 0x0000185f 0x20050000 0x00001976 0x20050000 0x00001aa8 0x10050000 0x00004e21

57-> 0x80060000 0x00000002 0x20060000 0x000015c3 0x20060000 0x000018d8 0x20060000 0x0000198d

65-> 0x20060000 0x00001ac4 0x10060000 0x00004e22 0x80070000 0x00000002 0x20070000 0x00001747

73-> 0x20070000 0x000019ae 0x10070000 0x00004e21

Bank:MCPP Length: 16(I*1)/4(I*4)/4(Type) Type:Unsigned Integer*4

 1-> 0x00005e4c 0x0000352d 0x00006453 0x00006d5b

FIXED format event

The "FIXED" format event has historically been used to map C structs to event data. The event consists of an event header, followed by the data payload. Most midas tools just treat it as arbitrary data, but it is possible to let the data logger know how to interpret the bytes, for example to convert data to ROOT files. See the frontends documentation for details about how to enable this.

Special events

There are three types of special events: begin-of-run, end-of-run and message events.

In all cases, the events contain an EVENT_HEADER, and then proceed directly to the data payload. There are no banks or BANK_HEADER.

Begin-of-run and end-of-run events

If file-level ODB dumps are enabled, two extra events will be added to each midas file (as the first and last events, respectively):

The begin-of-run (BOR) event has an EventID of 0x8000.
The end-of-run (EOR) event has an EventID of 0x8001.

In both cases, the data area of the event contains a JSON or XML representation of the ODB at the time the event was written. The format (JSON/XML) is specified using mlogger's ODB dump format setting.

Note that although the events are called "begin-of-run" and "end-of-run", they are actually written for every subrun! This is so that analysis code can find any relevant ODB settings in the same midas file as the data it is analysing.

These events have a trigger mask equal to 0x494d (aka 'MI'), and a serial number equal to the current run number.

Message events

If mlogger has been configured to log messages to data files, any messages written to the message log will also be stored in the .mid file.

The fields of a message event have the following meanings:

Field	Meaning / value
Event ID	0x8002
Trigger mask	Message type (e.g. Error == `MT_ERROR` == 1)
Time stamp	Time the message was sent
Data area	The message content as an ASCII string

Deprecated features

Tape format

Writing to tape is not maintained. Files are usually written to disk, and may later be archived onto tape or other storage devices using normal file transfer methods.

Events are written to disk files without any reformatting. For tapes, however, a fixed block size is used. The block size TAPE_BUFFER_SIZE is defined in midas.h and usually 32kB.

A tape can therefore be identified as a midas formatted tape.

The routine tape_copy() in the utility mtape.c is an example of how to read a tape in midas format.