Data logging: midas files

mlogger reads data from an event buffer and writes it to disk. The data is generally produced by frontends, but in principle any midas client can write events to a buffer. mlogger can also handle special types of "event" including messages and storing a full copy of the ODB at the start/end of each file.

Data format primer

Details of the event data format are available in Event data format. The key thing to note for now is that midas does not care about the content of the event - it just provides a standard way to tag metadata such as:

• event ID - which program produced the event
• trigger mask - an extra identifier set by the program that generated the event (often used to distinguish the trigger source - physics-like event, periodic event, calibration event etc)
• timestamp - the time the event was generated

Multiple channels

mlogger can handle multiple "channels" of data. The channels can read from different event buffers, and even filter the data based on the event ID and trigger mask.

The vast majority of experiments just use a single logging channel, recording all events that get written to the SYSTEM buffer. This is the default configuration.

To add an extra channel, create more entries in /Logger/Channels. Although the default channel is called 0, you do not have to just use numbers for your channel names. You can even rename the default channel if desired. To add more channels, you should copy/paste an existing channel, using either the web interface or the command-line:

odbedit -c 'copy "/Logger/Channels/0" "/Logger/Channels/new_channel"'

The Statistics subdirectory contains information about the amount of data written

Whether data should get written

The global boolean ODB key /Logger/Write data determines whether to write data to disk or not. This only affects event data - the history system has its own keys for determining whether data gets written.

There are also per-channel ODB keys /Logger/Channels/xxx/Active that can disable a channel even if the global Write data is true.

Data sources

Frontends

The main data logger to midas files is midas events written to an event buffer. Frontends are the main producers of such events, but in principle any midas client can write events to a buffer.

You must specify which buffer mlogger should read from, and can filter events based on event header metadata.

• /Logger/Channels/xxx/Buffer specifies the buffer to read from. The default is SYSTEM.
• /Logger/Channels/xxx/Event ID specifies which equipment's events to log. The default value of -1 means "no filtering".
• /Logger/Channels/xxx/Trigger mask specifies a bitmask to compare against each event's trigger mask. If any bit matches, the event is logged. The default value of -1 means "no filtering".

Each logging channel can only read from a single buffer.

Messages

Midas messages are automatically saved to the midas message log text files. However you can also log them to midas data files if desired.

The ODB key /Logger/Channels/xxx/Log messages enables this, and allows you to specify which message types get logged. mlogger will automatically connect to the SYSMSG buffer if you ask for messages to be logged in data files.

Details of how message events appear in the file are found in Event data format.

ODB at begin/end of file

mlogger can automatically save a copy of the ODB to a text file at the start of each run.

It can also store a copy at the start and end of each midas file. This is particularly useful when you are analyzing data, as all your configuration parameters are recorded in the same file as the raw data itself.

• /Logger/Channels/xxx/ODB dump is a boolean that says whether to save a copy of the ODB in each midas file
• /Logger/Channels/xxx/ODB dump format specifies the format to use - json or xml.

Details of how message events appear in the file are found in Event data format. Note that although documentation (and the acronyms BOR/EOR) refer to these as begin-of-run and end-of-run dumps, they are actually written at the begin/end of each subrun.

Where to write the data

The channel Output determines what happens to the data - valid options are FILE and FTP. The default is FILE.

Writing to disk

Setting /Logger/Channels/xxx/Output to FILE results in midas files getting written to disk.

The location where midas files get written is based on:

• the global directory specified in /Logger/Data dir
• the per-channel sub-directory specified in /Logger/Channels/xxx/Subdir format. This can use a strftime format to create new subdirectories based on date/time. This can help you avoid having thousands of files in the same directory, which can be unwieldy to handle.
• the per-channel filename specified in /Logger/Channels/xxx/Filename. This can use %d placeholders to replace the run and/or subrun.
• whether any compression is enabled using /Logger/Channels/xxx/Compress. This automatically appends a compression suffix such as .lz4 or .gz.

/path/to/dir/2026_12_31/run01234_0000.mid.lz4

Writing to FTP

Setting /Logger/Channels/xxx/Output to FTP results in midas files getting written to an FTP server. mlogger streams the data (i.e. the FTP connection remains open while data is being acquired, rather than writing data to a temporary local file and then copying the final file via FTP).

The /Logger/Channels/xxx/Filename key specifies all the FTP connection details. Subdir format and Data dir are not used for FTP channels.

The Filename can take one of two formats:

ftp_host,port,user,password,remote_directory,filename_format
ftp_host,port,user,password,remote_directory,filename_format,mode

# Run number substitution only, no explicit permissions
myhost.my.domain,21,john,password,/usr/users/data,run%05d.mid

# Run and subrun number substitution, with explicit permissions
myhost.my.domain,21,john,password,/usr/users/data,run%05d_%04d.mid,660

Data compression

The /Logger/Channels/xxx/Compress key allows you to compress a file so it takes up less space on disk (at the cost of being slower to read the file later, as you need to then decompress it).

Valid compression algorithms are gzip, lz4, bzip2 and pbzip2. The default is lz4, and this will be sufficient for most users. You can disable compression by setting the Compress key to none.

Algorithm	File extension	Is the default	Supported writing to disk	Supported writing over FTP
none	N/A
lz4	.lz4
gzip	.gz
bzip2	.bz2
pbzip2	.bz2

There are extra options available for the gzip, bzip2 and pbzip2 algorithms to fine-tune the compression level:

• lz4 always uses the default compression level.
• The gzip compression level can be set using the Gzip compression key. Valid numbers are 1-9, with 1 giving the fastest speed, and 9 giving maximum compression. 0 means no compression! A common value is 6.
• The bzip2 compression level can be set using the Bzip2 compression key. Valid numbers are 1-9, with 1 giving the fastest speed, and 9 giving maximum compression. 0 will result in the system default (probably 9).
• pbzip2 has 3 settings.
  • Pbzip2 compression sets the compression block size. Valid numbers are 1-9, with 1 giving the fastest speed, and 9 giving maximum compression. 0 will result in the system default (probably 9).
  • Pbzip2 num cpu sets the number of CPUs to use for the parallel compression code.
  • Pbzip2 options is a string containing any other parameters you wish to pass to the pbzip2 command-line program. The -z and -c options are already set. -p is also already set if Pbzip2 num cpu is > 0.

Checksums for data corruption checks

Checksums are small strings of data that let you check if a file has been corrupted. They don't let you fix the corruption, but are useful for verifying data integrity.

Midas provides checksums for both the pre-compressed data and compressed data. Both support the same checksum algorithms: CRC32C, SHA256, SHA512 and ZLIB. You can also disable writing of checksum files by setting an algorithm of NONE.

• /Logger/Channels/xxx/Data checksum specifies the pre-compressed checksum algorithm
• /Logger/Channels/xxx/File checksum specifies the post-compressed checksum algorithm

The checksums are written to small text files containing the checksum itself, the size of the file in bytes, and the filename:

run30805.mid.crc32c      # Data checksum
run30805.mid.lz4         # Data file
run30805.mid.lz4.crc32c  # File checksum

8285ea51 334379 /path/to/run30805.mid.lz4

The available algorithms are:

Checksum	Is the default	File extension	Tool to verify with
NONE		N/A	N/A
CRC32C		.crc32c	cat <file> | crc32c_sum
SHA256		.sha256	sha256 -q <file>
SHA512		.sha512	sha512 -q <file>
ZLIB		.crc32zlib	crc32 <file>

# See that midas wrote 3 files for this run. 
# The checksum files are very small; the .mid.lz4 is the real data.
$ ls run30805.mid*
run30805.mid.crc32c
run30805.mid.lz4
run30805.mid.lz4.crc32c

# Compute checksum of .mid.lz4 file
$ cat run30805.mid.lz4 | crc32c_sum
8285ea51

# Note it matches the first entry in the .mid.lz4.crc32c file
$ cat run30805.mid.lz4.crc32c
8285ea51 334379 /path/to/run30805.mid.lz4

# The 2nd entry in the .mid.lz4.crc32c file matches the file size of the .mid.lz4 file
$ ls -l run30805.mid.lz4
-r--r--r--  1 ben  staff  334379  1 Jun 14:42 run30805.mid.lz4

# Decompress the .lz4 and compute the checksum of the .mid file
$ unlz4 run30805.mid.lz4
$ cat run30805.mid | crc32c_sum
ee53b826

# Note it matches the first entry in the .mid.crc32c file
$ cat run30805.mid.crc32c
ee53b826 10451573 /Users/ben/DAQ/test_expt/run30805.mid

Run and subrun duration

mlogger can automatically stop a run after a certain time period has elapsed, and can split data from one run across multiple files (called subruns). More advanced run control logic can be implemented using the sequencer.

Variable	Stop run	Move to next subrun
Time
Data size (bytes)
Number of events

Run duration

mlogger can automatically stop a run based on 3 criteria:

• /Logger/Run duration specifies a time in seconds
• /Logger/Channels/xxx/Byte limit specifies a total data size in bytes
• /Logger/Channels/xxx/Event limit specifies a total number of events

In all cases, a value of 0 means no limit. For the per-channel settings, the first channel to reach the limit will cause the run to stop.

Auto restart

You can request that mlogger automatically restart the run if it is stopped due to one of the above criteria. /Logger/Auto restart enables this feature, and /Logger/Auto restart delay specifies a time in seconds to wait between one run stopping and the next one starting.

mlogger will not automatically restart a run if it is stopped for any other reason (e.g. the user stopping a run manually, or the disk getting full).

Subrun duration

mlogger can automatically start a new subrun based on 3 criteria:

• /Logger/Subrun duration specifies a time in seconds
• /Logger/Channels/xxx/Subrun Byte limit specifies a file size in bytes
• /Logger/Next subrun is a boolean that a user (or code) can set to request a new subrun

A duration or size limit of 0 means no limit. Subruns start at 0 and increment each time a new file is opened.

Note that if you are using subruns, you should ensure that your filename format includes a placeholder for both the run number and subrun number!

In practice, the vast majority of experiments use size-based limits on their subruns.

Note that the checks are performed only AFTER an event has been written. That means that a new subrun is only started once the limit has been exceeded.

Analyzing midas files

Analysis of midas files generally requires one to write some code, and this is covered in Data analysis.

Statistics and disk level

The ODB directory /Logger/Channels/xxx/Statistics contains details about the amount of data written and the current disk fullness. Full details of all the keys can be found in the ODB reference.

Stopping the run if disk is getting full

If mlogger is writing files to disk (not via FTP), the amount of space left on the disk will be checked, and the fraction of the disk that is full (from 0 for empty to 1 for full) will be written to /Logger/Channels/xxx/Statistics/Disk level.

A fullness bar is shown on the main status webpage.

If the Type is Disk (not anything else) and the Output is FILE (not FTP), then the run will stop automatically when the disk is getting full. The limits are hardcoded:

• if the disk is > 100GB, the run will stop when there is less than 1GB free
• if the disk is 10-100GB, the run will stop when there is less than 100MB free
• if the disk is < 10GB, the run will stop when there is less than 10MB free

These checks are only performed if a channel is active and data-writing is enabled.