Skip to content

Programs: configuring

This page covers how to configure how programs are started.

For the concepts and day-to-day use, see Programs: overview.

/Programs in the ODB

Each program gets its own entry in the ODB in /Programs/xxx. This directory contains both settings and status information. For example, mlogger's information is in /Programs/Logger.

You can access each program's ODB directory using the ODB web interface, or by clicking the program's name from the Programs webpage.

The two most important settings are Required and Start command.

Required programs

Setting /Programs/xxx/Required to y has two impacts:

  1. the program will always appear on the Programs webpage, even when it is not running
  2. if /Experiment/Prevent start on required progs is y, then the run will not start if the program is not running

Start command

Setting the ODB key /Programs/xxx/Start command allows you to start a program from the Programs webpage (or from code).

The first time you start a program, the Start command is populated with the command-line you used to invoke the program. You may need to adjust the command for it to work correctly!. For example, if your initial invocation used a relative path (e.g. ../bin/my_fe.exe), you may want to specify the full path in the Start command (e.g. ~/my_package/bin/my_fe.exe).

Environment of launched programs

Programs are started in the same environment as the program that is launching them. If using the web interface, this is the environment that the mhttpd program is currently running in.

Impacts of this are:

  • you cannot guarantee what the "current directory" is, so should use absolute paths rather than relative paths
  • if you stop and restart mhttpd, the environment might have changed
  • if you edit your .bashrc file (or similar) to define new environment variables, they may not appear for your spawned programs until you relaunch mhttpd in the new environment

If you don't want to restart mhttpd to pick up a new environment, you can instead source a script as part of your start command:

Example start command that first sources a script

source ~/.bashrc && ~/my_package/bin/my_fe.exe

If you write code that launches a program, and that code is running on a remote machine, the program is launched on the remote machine rather than the main midas host!

Note that the command used to start the program does not need to remain alive - midas checks whether programs are still running by having them check-in periodically. So your start command could interface with systemd, or spawn a program as daemon (-D flag) - as long as the program that connects to midas checks-in, it will appear as "running" on the Programs webpage.

Starting a program on a remote host

If you are running on multiple machines, your start command can include an ssh to the remote node.

You will need to have password-less ssh access to the remote machine for this to work.

Example start command for a remote host

ssh my.other.host "~/my_package/bin/my_fe.exe -i 4"

In this case, the Programs webpage informs you which host the program is running on.

Checking if programs are still running

All midas clients periodically check-in to report that they're still running. If a program doesn't check-in for a long time, other programs will notice this. The wayward program will lose its access to the experiment, and be asked to shut down. More details are available in Writing long-running programs.

The ODB key /Programs/xxx/Watchdog timeout specifies how long (in ms) this program can go without checking in, before being declared dead. The default is 10000ms (10s).

Alarm if program isn't running

You can raise an alarm if a program isn't running. This is configured by setting the /Programs/xxx/Alarm class to the name of a valid alarm class.

The Check interval key specifies how long a program must be absent for before the alarm gets raised. This helps prevent false alarms when a user is deliberately restarting a program. The default value is 180000ms (3 minutes).

The First failed key is used internally to track how long the program has been stopped for. It is a UNIX timestamp with 1s precision, and gets reset to 0 whenever the program checks-in correctly.

Note that these alarms are only active if the alarm system is active.

Automatically starting/stopping programs

The keys Auto start, Auto stop and Auto restart in /Programs/xxx allow you to start programs automatically.

  • Auto start and Auto stop will start/stop a program at the beginning/end of a run.
  • Auto restart will restart a program at any time if it isn't running.

If Auto start is set to y, the program's "start command" will be executed at the beginning of the run start transition.

'Auto start' happens unconditionally!

If Auto start is set to y, the "start command" is executed unconditionally! Because there are no checks or tests, the "start command" will be executed even if the program is already running. Depending on how the program is coded, you may either:

  • end up with two instances of the program running at the same time
  • kill the existing instance of the program and start a new one.

The latter option is what happens for all frontends.

Note that programs that are started this way do NOT get told about the begin-of-run transition, as the transition is already in progress when they get started.

If Auto stop is set to y, the program will be stopped after the end of the run. It makes most sense to have Auto start and Auto stop set to the same value, and in most cases it makes most sense to have them set to n and just have the program be running all the time.

If Auto restart is set to y, and the program has not been running for longer than /Programs/xxx/Check interval (default 3 minutes), the program's "start command" will be executed to restart it. This is part of the alarm checks, so it only happens while /Alarms/Alarm system active is y. It doesn't need an alarm class to be set.

Context of 'auto' executions

As noted in the start command documentation, programs may be launched in different contexts/environments.

For Auto start, the program will be launched in the environment on whichever program initiated the transition. This could be mhttpd (the web server), a sequencer, the odbedit command-line tool, or custom code written by the user. If the program initiating the transition runs on a remote machine, the launched program will also run on the remote machine!

For Auto restart, the program will be launched as part of the alarm checks. For programs running on remote machines, these checks are performed by the mserver program that is running on the main machine. So auto-restart programs are always launched on the main machine, but may still have a different environment compared to clicking the "Start program" button on the Programs webpage.

Multiple instances of the same program

Code can choose whether to allow multiple instances of a program to be running at once. For example, we definitely want to allow multiple people to use the odbedit command-line tool, but we probably don't want to have multiple programs fighting to talk to the same piece of hardware at the same time.

Unique names

Programs page If the program allows multiple instances to run at once, they all appear with the same name on the Programs webpage, and do not cause extra entries to be created in /Programs.

However the different instances have different names in /System/Clients, and these unique names are used in the midas message log, in the alarm system, and when other code wants to communicate with a program.

The unique names are <base_name>, <base_name>1, <base_name>2 etc. For example, if 3 instances of odbedit are running at the same time they are given the names ODBEdit, ODBEdit1, and ODBEdit2. Subsequent instances take the first name available (i.e. if the first instance shuts down, there is no client called ODBEdit any more, and the next instance will be given that name rather than ODBEdit3).

Edge case if programs talk to each other

You can run into trouble if you have programs that talk to each other directly (using the RPC system). The initiator of the request specifies the client name that it wants to talk to.

Say we have the clients named requester and responder. The requester is coded to talk to a program called "responder" (and only a program called "responder"!).

The failure case is:

  • Start two instances of responder. These are now called responder and responder1.
    • This is fine, and the requester can talk to the first instance that was launched.
  • Stop the first instance. Now we only have a client called responder1, and no client called responder.
    • This is a problem! The requester program tries to talk to responder, but it doesn't exist!

You can fix this by starting another instance of the responder program. It will see that there is no client running with the name responder, and will use that name.

Frontends

Frontends coded in C++ are coded to shut down other frontends with the same name, so that only one frontend can talk to a piece of hardware are once. However, the frontend must first connect to midas to do these checks!

For example, if the first instance was called my_fe, the second instance will be called my_fe1. The second instance will shut down the first one, and you will be left with just my_fe1 running and no my_fe.

An extra wrinkle is that frontends can be started with the -i flag to specify a frontend index. In this case, the client name generally becomes <base_name><index><instance>. For example, if two instances of the my_fe program are started with -i 4 on the command line, they will be called my_fe04 and my_fe041! Although it looks like the second instance was started with -i 41, it is actually the second instance started with -i 4!

Starting programs at boot

Midas does not start any programs when the machine boots. Some experiments use a start script that launches mhttpd, mlogger and their frontends, and run it at boot from cron. See writing a start script and starting midas at boot for an example.