Skip to content

Writing long-running programs

There is no central/master program in midas - all clients participate in periodic checks of whether alarms need to be raised, detect if other programs have crashed, and more. Programs also need to detect if other programs have communicated with them, if any ODB keys they were watching have changed, if run transitions are occurring and so on.

All these checks are handled by cm_yield() in C++ and client.communicate() in Python.

There is also a separate watchdog system, which programs use to tell other clients that they are still alive and running.

Calling the periodic tasks

All long-running programs in midas MUST repeatedly call cm_yield() (C++) or client.communicate() (Python) in a loop.

Exceptions to this rule are C++/Python frontends and Python sequencer scripts, where the frameworks automatically call these functions for you.

In both C++ and Python, an argument is provided with a timeout in milliseconds. The functions will return immediately if an event is available, or once the timeout has expired if there is no work to do.

while (true) {
  INT status = cm_yield(1000);

  if (status == RPC_SHUTDOWN || status == SS_ABORT) { //(1)!
    cm_disconnect_experiment();
    exit(0);
  }
}
  1. We've been told to shutdown (probably because the user clicked the "Stop program" button on the Programs webpage)
while True:
  client.communicate(1000) #(1)!
  1. Program will automatically clean up and exit if we're told to shutdown

Note that if you have registered callback functions (e.g. for handling run transitions or watching for ODB changes), those functions are called from within cm_yield()/client.communicate().

Full list of periodic checks
  • Flush messages that have been buffered
  • Flush unsent ODB changes
  • Check for new events in event buffers
  • Check alarm conditions
  • Handle RPC communications with other clients (both sending and receiving)
    • Run transitions
    • ODB hotlinks
    • Shutdown requests
    • Custom RPC calls
  • Check the watchdog timeouts of other clients

Watchdog timeout

A common problem in DAQ systems is the possible crash of a client that was requesting data from an event buffer. This can cause the whole system to hang up, and may require a restart of the DAQ causing a loss of both time and, eventually, precious data. In order to address this problem, a watchdog scheme has been implemented.

Your program should periodically notify other clients that it is still running. If your program does not check in for a long time, other programs doing their checks will assume that your program is unresponsive, revoke its access to the experiment (including removing any event requests and ODB hotlinks), and send the wayward program a command to shutdown.

Checking-in

The simplest way to periodically check-in is to launch a separate watchdog thread:

INT status = cm_start_watchdog_thread();

In Python, the separate watchdog thread is started automatically when you create your MidasClient object. This means that you can easily use a MidasClient object interactively (in a python command-line session or Jupyter notebook), as the background thread continues to run even when the prompt is waiting for more input from the user.

If you don't want to run a separate thread, you can manually set the flags, but there are very few cases where this makes sense on modern machines:

DWORD now = ss_millitime();
bm_update_last_activity(now);
db_update_last_activity(now);

There is no need to check-in manually in Python, as we are always running a separate watchdog thread.

Changing the timeout

The default timeout is 10s (10000ms) and can be permanently changed for each program using the ODB keys /Programs/xxx/Watchdog timeout (specified in ms).

If you are not using a separate watchdog thread and are about to perform a task that takes a long time, you can temporarily change the watchdog timeout using code. Setting a watchdog timeout of 0 disables the checks; this may be useful if you are debugging a program and don't want it to get disconnected from the experiment while you operate interactively.

// Get the original settings
BOOL call_watchdog = FALSE;
uint32_t original_timeout_ms = 0;
INT status = cm_get_watchdog_params(&call_watchdog, &original_timeout_ms);

// Set a new timeout
uint32_t new_timeout_ms = 60000;
status = cm_set_watchdog_params(call_watchdog, new_timeout_ms);

// Do your time-consuming task

// Revert to the original settings
status = cm_set_watchdog_params(call_watchdog, original_timeout_ms);

Doing work when the program exits

You may need to do some work just before your program disconnects from midas, e.g. to put some hardware back into a safe state. Frontends have their own exit function for this (see Frontend exit).

Do the work before calling cm_disconnect_experiment(). This includes when cm_yield() returns RPC_SHUTDOWN or SS_ABORT (see Calling the periodic tasks).

Call register_disconnect_callback with the function to run.

To be sure that the function is called, create your client using a "context manager" (a with statement):

# GOOD - using a context manager (1)
with midas.client.MidasClient("my_client") as client:
    client.register_disconnect_callback(my_cleanup_function)
    # Do your work

# BAD - not using a context manager (2)
client = midas.client.MidasClient("my_client")
client.register_disconnect_callback(my_cleanup_function)
  1. my_cleanup_function() is always called, even if there's an uncaught exception or exit() is called.
  2. my_cleanup_function() may NOT be called!
Technical explanation

Without a context manager, the best we can do is to disconnect from midas in the destructor of the MidasClient object. But Python does not guarantee that an object's __del__() function will be called in a timely manner (or even at all!). It depends on your Python version, Python implementation, and whether your code does anything unusual (e.g. creating circular references).

With a context manager, we can disconnect from midas in the __exit__() function, which Python DOES guarantee will be called.