Skip to content

Large experiments

This page describes what to do when a large experiment reaches one of midas's built-in limits: the number of programs, hotlinks or event requests. It covers other ways of organising the experiment, and how to raise the limit if you need to.

For all the limits, see Limits.

Before you raise a limit

Some limits are fixed when midas is compiled, and can only be raised by changing the midas source code and re-compiling it. Some of these changes have knock-on effects, and care is required.

The midas developers' advice is: do you really need to raise the limit?

  • Is there a different way of organising your experiment, so that you don't have to raise the limit? Each section below suggests some
  • If you do raise the limit, will it make the experiment harder to use, or slower?
  • You'll need to make the same source code changes each time you update midas, and each time you install it on another computer
Why the sizes in the assert() statements are hard-coded

Raising a limit changes the size of some data structures, so you also need to change some assert() statements that check the size of a structure against a fixed number. Users sometimes ask why these numbers are hard-coded, rather than calculated (e.g. instead of assert(sizeof(DATABASE_CLIENT) == 2112), use assert(sizeof(DATABASE_CLIENT) == 64+sizeof(OPEN_RECORD)*MAX_OPEN_RECORDS)).

When you change these sizes, your midas is no longer binary-compatible with standard midas, and the asserts tell you so. This isn't a small step: you have to re-compile every midas program, and if you miss one and run it with your changed midas, it will break the shared memory, with no safety net.

Calculating the sizes from the constants would defeat the purpose of the checks, which is to catch the sizes of critical data structures being different from what's expected, whether by mistake or because of a mis-compilation.

More than 64 programs

Midas allows 64 programs (clients) to be connected to an experiment. This includes the midas programs (mlogger, mhttpd etc.) as well as your frontends. Experiments with many pieces of hardware may need more frontends than this.

Downsides

The downsides of allowing more programs are:

  • The status page will become unwieldy, and hard to read
  • Each program regularly checks that all the other programs are still running, so the number of checks will grow large
  • Some resources can only be used by one program at a time (e.g. only one program can write to an event buffer at once). With more programs, they'll spend more time waiting for each other
  • The ODB and the event buffers in shared memory have to be recreated
  • It's a delicate operation: you must make sure that all your programs are re-compiled, and that your changes to midas aren't lost when you update it

Alternatives

There are two main ways to avoid needing more programs:

  1. Run some of the programs outside midas. If you have several copies of the same frontend (e.g. each doing the same work, but reading a different digitizer), you could run those outside midas. One midas program forwards commands to them (e.g. "start a run with these settings"), collects their data and writes it to an event buffer. You'd need ZeroMQ or your own protocol for the communication between the midas and non-midas programs.
  2. Split the experiment into sub-experiments. If your experiment breaks down into parts, this does the same thing more neatly. Say you have a TPC, an ECAL and an MCAL. You could have four midas experiments: one for each part, and a global one that ties them together. The global one has three frontends, each of which talks to one of the other experiments (forwarding run starts etc.). Each part is a real midas experiment, so it can be run on its own for debugging. If the TPC isn't working, you can take it out of the main experiment and carry on with the ECAL and MCAL, while experts work on the TPC experiment on its own. The T2K experiment did this for its ND280 detector, and its code could be added to midas if it would be useful for other experiments.

If you do decide to allow more programs:

  1. Make sure you have a JSON backup of the ODB (run odbedit -c "save odb.json" if you're not sure).
  2. Increase MAX_CLIENTS in include/midas.h.
  3. In db_validate_sizes() in src/odb.cxx, increase the expected sizes of BUFFER_HEADER and DATABASE_HEADER. Look at the structure definitions to work out the new sizes.
  4. Stop all the programs connected to the experiment.
  5. Re-compile midas, then all your frontends and other programs.
  6. Follow the instructions to repair the ODB from a JSON backup, because the existing ODB shared memory is no longer valid.
  7. Remove the shared memory files of the event buffers (e.g. .SYSTEM.SHM in the experiment directory), so that they're recreated (see Increasing buffer sizes).

Midas allows 256 ODB hotlinks in each program. Most programs stay far below this, because one hotlink can watch a whole ODB directory, and be told whenever any value inside it changes (however deep).

The program that most often needs more is mlogger. If you have many frontends and history events, mlogger may report that it can't open a hotlink for all of them.

Why the limit is for each program

The hotlinks are stored in the ODB's shared memory, so that every program can see them. Each connected program has its own table of up to 256 hotlinks there. When a program changes a value, it looks through all the programs' tables for hotlinks on that key (or on a directory containing it), and sends a short message to each program that has one. So the experiment as a whole can have up to 64 × 256 hotlinks, but one program can only open 256. See Hotlink implementation for the details.

Downsides

The downsides of allowing more hotlinks are:

  • The ODB in shared memory has to be recreated
  • It's a delicate operation: you must make sure that all your programs are re-compiled, and that your changes to midas aren't lost when you update it

Alternatives

  • If your own program has too many hotlinks, change it to watch a higher-level directory, rather than many subdirectories
  • If mlogger has too many, you may be able to combine several history events into one (e.g. change your frontend to write large arrays in /Equipment/xxx/Variables, rather than many single values)

If you do decide to allow more hotlinks:

  1. Make sure you have a JSON backup of the ODB (run odbedit -c "save odb.json" if you're not sure).
  2. Increase MAX_OPEN_RECORDS in include/midas.h.
  3. In db_validate_sizes() in src/odb.cxx, increase the expected sizes of DATABASE_CLIENT and DATABASE_HEADER. Look at the structure definitions to work out the new sizes.
  4. Stop all the programs connected to the experiment.
  5. Re-compile midas, then all your frontends and other programs.
  6. Follow the instructions to repair the ODB from a JSON backup, because the existing ODB shared memory is no longer valid.

More than 10 event requests in a program

Midas allows 10 event requests in each program. This may not be enough for an event builder that combines data from many event buffers.

Downsides

The downsides of allowing more event requests are:

  • The event buffers in shared memory have to be recreated
  • It's a delicate operation: you must make sure that all your programs are re-compiled, and that your changes to midas aren't lost when you update it

Alternatives

Change your frontends so that they all write to the same event buffer. The event builder can then read all the data from that one buffer. Give each frontend's events their own event ID or trigger mask, so that other programs can still request the events of only one frontend.

If you do decide to allow more event requests:

  1. Increase MAX_EVENT_REQUESTS in include/midas.h.
  2. In db_validate_sizes() in src/odb.cxx, increase the expected sizes of BUFFER_CLIENT and BUFFER_HEADER. Look at the structure definitions to work out the new sizes.
  3. Stop all the programs connected to the experiment.
  4. Re-compile midas, then all your frontends and other programs.
  5. Remove the shared memory files of the event buffers (e.g. .SYSTEM.SHM in the experiment directory), so that they're recreated (see Increasing buffer sizes).

The ODB isn't affected by this change, so you don't need to recreate it.