Skip to content

See data analysis for how to analyse midas data, and data format for the structure of midas files. This page contains the full API reference of the midas.file_reader module.

midas.file_reader is pure Python, so it can read midas files on a machine without a midas installation. It reads compressed files (.gz, .bz2 and .lz4) as well as plain .mid files. Reading .lz4 files needs the lz4 Python package (pip install lz4).

Printing every event in a file

import midas.file_reader

my_file = midas.file_reader.MidasFile("/path/to/file.mid.lz4")

for event in my_file:
    event.dump()

Reading a value from the ODB dump at the start of the run

import midas.file_reader

my_file = midas.file_reader.MidasFile("/path/to/file.mid.lz4")
odb = my_file.get_bor_odb_dump().data

run_number = odb["Runinfo"]["Run number"]

There is a longer example in $MIDASSYS/python/examples/file_reader.py.

midas.file_reader

Tools to read a midas file.

The midas file format is documented at https://midas.triumf.ca/MidasWiki/index.php/Event_Structure#MIDAS_Format_Event. Each event has a "header" containing metadata, then a series of names "banks" containing the actual data. The banks can be of different formats (e.g. lists of floats/doubles/ints).

The tools in this files can read in midas events, and automatically convert the data in the banks to appropriate python data types (so you get a tuple of ints, rather than just a raw set of bytes).

We can read on files in .mid. .mid.gz and .mid.lz4 format. lz4 support is not present in the standard python libraries, but can be installed using pip (e.g. pip install lz4). See the README file to learn more about pip.

Basic usage

import midas.file_reader

# Open our file
mfile = midas.file_reader.MidasFile("040644.mid")

# We can simply iterate over all events in the file
for event in mfile:
    bank_names = ", ".join(b.name for b in event.banks.values())
    print("Event # %s of type ID %s contains banks %s" % (event.header.serial_number, event.header.event_id, bank_names))

More complex usage, which can allow you to skip events more efficiently

import midas.file_reader

# Open our file
mfile = midas.file_reader.MidasFile("040644.mid")

# Here we choose to just read in the header of each event, and will read
# the body (the actual banks) later.
while mfile.read_next_event_header():
    header = mfile.event.header

    if header.is_midas_internal_event():
        # Skip over events that contain midas messages or ODB dumps
        continue

    print("Overall size of event # %s of type ID %s is %d bytes" % (header.serial_number, header.event_id, header.event_data_size_bytes))

    if not mfile.read_this_event_body():
        raise RuntimeError("Unexpectedly failed to read body of event!")

    # Loop over the banks of data in this event and print information about them
    for name, bank in mfile.event.banks.items():
        # The `bank.data` member is automatically converted to appropriate python data types.
        # Here we're just figuring out what that type is to print it to screen. Normally
        # you already know what to expect for each bank, and could just the tuple of floats,
        # for example.

        if isinstance(bank.data, tuple) and len(bank.data):
            # A tuple of ints/floats/etc (a tuple is like a fixed-length list)
            type_str = "tuple of %s containing %d elements" % (type(bank.data[0]).__name__, len(bank.data))
        elif isinstance(bank.data, tuple):
            # A tuple of length zero
            type_str = "empty tuple"
        elif isinstance(bank.data, str):
            # Of the original data was a list of chars, we convert to a string.
            type_str = "string of length %d" % len(bank.data)
        else:
            # Some data types we just leave as a set of bytes.
            type_str = type(bank.data[0]).__name__

        print("  - bank %s contains %d bytes of data. Python data type: %s" % (name, bank.size_bytes, type_str))

Reading ODB information

import midas.file_reader

mfile = midas.file_reader.MidasFile("040129.mid")

try:
    # Try to find the special midas event that contains an ODB dump.
    odb = mfile.get_bor_odb_dump()

    # The full ODB is stored as a nested dict withing the `odb.data` member.
    run_number = odb.data["Runinfo"]["Run number"]
    print("We are looking at a file from run number %s" % run_number)
except RuntimeError:
    # No ODB dump found (mlogger was probably configured to not dump
    # the ODB at the start of each subrun).
    print("No begin-of-run ODB dump found")

midas.file_reader.MidasFile

Provides access to a midas file - either raw (.mid), gzipped (.mid.gz) or lz4 (.mid.lz4).

Attributes:

  • file (file-like object) –

    The file we're reading

  • event (Event) –

    The event we've just read

  • next_event_offset (int) –

    Position in file where the next event starts

  • this_event_payload_offset (int) –

    Sometimes we just read the event header, not the full data. This member is where the data of the current event starts.

  • use_numpy (bool) –

    Whether to use numpy when extracting bank contents (so bank data is a numpy array rather than a standard python tuple)

__init__(path, use_numpy=False, ignore_bad_bytes_in_odb=True)

Open a midas file.

Parameters:

  • path (str) –

    Path to the file

  • use_numpy (bool, default: False ) –

    Whether to use numpy when extracting bank contents (so bank data is a numpy array rather than a standard python tuple)

  • ignore_bad_bytes_in_odb (bool, default: True ) –

    Whether to silently skip any non-UTF8 bytes in the begin/end-of-run ODB dumps. If False, you'll get strict checking and may raise an error if invalid UTF-8 byte sequences are present in the ODB.

get_bor_odb_dump()

Return the begin-of-run ODB dump as a midas.file_reader.Odb object.

Raises:

get_eor_odb_dump()

Return the end-of-run ODB dump as a midas.file_reader.Odb object.

Raises:

get_event_count(include_midas_special_events=False)

Count the number of events in this file.

Parameters:

  • include_midas_special_events (bool, default: False ) –

    Whether to include midas' internal events in the count (begin-of-run, message and end-of-run events).

Returns:

  • int –

    Number of events

get_next_event_with_bank(bank_name)

Find the next event that contain a bank with the specified name.

Returns:

  • Event –

    or None of no such event found.

jump_to_start()

Rewind to the start of the file.

open(path)

Open a midas file.

Parameters:

  • path (str) –

    Path to midas file. Can be raw, gz or lz4 compressed.

read_next_event()

Read the header and content of the next event. May be slow if there is a lot of data.

Returns:

  • Event –

    or None if no more events left.

read_next_event_header()

Just read the header/metadata of the next event. If you read it and think it's interesting, you can then call read_this_event_body() to grab the actual data. If the event isn't interesting, then you saved yourself a lot of time by not loading a bunch of data you don't care about.

Returns:

  • Event –

    (with only the header populated), or None if no more events left.

read_this_event_body()

Read the data of the current event (that you've already read the header info of).

Populates event.banks or event.non_bank_data (depending on the event type).

Returns:

  • Event –

    with both the header and body populated

midas.file_reader.Odb

Helps read an XML/JSON representation of an ODB, and convert it to a python dict.

Attributes:

  • written_time (datetime) –

    Time the ODB dump was written (only if the dump was written as XML)

  • data (dict) –

    The actual ODB structure

__init__(odb_string=None, ignore_bad_bytes_in_odb=True)

Initialize an ODB object py parsing an ODB dump.

Parameters:

  • odb_string (str, default: None ) –

    Either XML or JSON representation of an ODB dump.

  • ignore_bad_bytes_in_odb (bool, default: True ) –

    See MidasFile.init