Release notes

• Web interface
  • ODB browser
    • Duplicate a key or directory
    • Preview a file before loading or importing it. Save, export or copy several selected directories at once
    • Many keyboard shortcuts, type-ahead to jump to a key, and keyboard editing of array elements
    • Search results in a floating dialog
    • Open records only lists records below the current directory
    • Change the mode bits of one or more keys, or of a directory and everything in it. Active hotlinks are shown as H in the Mode column
    • Loading or saving an ODB file (browser or odbedit) writes a line to the message log
  • Keyboard shortcuts for the built-in menu pages: Alt and a letter (e.g. Alt+S for Status, Alt+O for ODB). Holding Alt underlines the letters in the menu
  • Snooze... button on the Status and Alarms pages silences alarm sounds and speech in that browser for 1 min to 1 h
  • Programs page groups clients by program name and has a Stop all button
  • Event dump page: bank selection, and bytes shown as decimal with ASCII
  • Cursor and Tab key navigation in all dialog boxes
  • Built-in pages use the new modbswitch toggle
• Sequencer
  • Debugger highlights variables that changed
  • Syntax validation is always done before a script is started, saved or queued (no longer optional). MSL is checked by the same C++ parser that msequencer uses, including INCLUDE files and warnings for missing ODB keys. Python is checked by PySequencer (with pyflakes if installed)
  • New /Sequencer/Command/Goto line jumps a running script to a line (ODB keys). Jumps into or out of a LOOP, IF or SUBROUTINE are refused
  • MSL: subroutines can be called without CALL. Commands can have long arguments and more than 99 arguments
  • Reworked web page: state badges (running, paused, stopped, debug, waiting), redesigned side panel with draggable divider, more keyboard shortcuts
  • PySequencer: many fixes, and a new example pysequencer_mscb_read.py
• Alarms
  • Conditions can combine terms with && and ||. && binds tighter than ||, and parentheses are not supported
  • al_reset_alarm() runs inside mserver for remote clients (faster)
  • If the browser blocks the alarm sound, it is played after the next click on the page
• ODB
  • Key and data free lists are checked when a client connects, and more kinds of corruption are reported. Repair is done only if MIDAS_ODB_REPAIR_FREE_LIST=1 is set
  • New odbedit du command shows the memory used by each subtree
  • Program name stored in /System/Clients/<pid>/Program (details)
  • JSON saves stay valid if an ODB read fails, and handle links to single array elements
• History
  • Measure the time and value differences between two points on a plot
  • /History/Images/Demo is no longer created by default
  • mdev define_history_panel() can take a group name
• Frontends and drivers
  • New mdev driver for TDK power supplies
  • RevolutionPi example frontend moved to examples/revolutionpi, now with EPICS
  • MSetPoint: comma-separated EPICS commands (e.g. EIN,ON) and new documentation
  • msysmon-lmsensors is installed if it was built
• C++ API
  • New db_get_data_vec() and db_get_data_index_vec() read ODB data into a std::vector, also over RPC
  • odbxx: typed exceptions (odb_missing_key, odb_type_error, ...), file- and string-backed objects (odb::file(), odb::string()), contains(), is_directory(), named_items(), and full dirty-write support
  • odbxx: compare() two trees, and midas::odb::array() creates an array of a size chosen at runtime
  • odbxx string arrays grow to fit the longest string
• Python API
  • Robustness review of the midas package: about 100 fixes (memory release, string truncation, UTF-8, links on remote clients, numpy, file reader)
  • New reset_all_alarms(). clients_exist() takes exact_match
  • KeyError and TypeError raised for midas errors carry the midas status in e.code
  • odb_set() accepts "0x..." strings for float and double keys
  • Frontend data rate calculated exactly as in mfe
• Web API (JavaScript)
  • New widgets modbtile, modbswitch and modbgauge2
  • Array elements can be selected by name in data-odb-path, e.g. Variables/Input['Temp1']
  • mplot: function plots (data-function), measuring the differences between two points, Viridis palette, faster colour maps, CSV export of functions
  • mtabs remember the selected tab and can be bookmarked with a URL parameter
  • New dlgListBox(). dlgGeneral() dialogs can be resized
  • New JSON-RPC methods seq_validate, db_set_mode, ext_rename_file and ext_delete_file
• Build
  • install.sh compiles in parallel and adds the mscb Python package to PYTHONPATH
  • New cmake option -DMIDAS_BUILD_DOCS=1 builds this documentation (details)
• The shiftschedule shift-planning page and frontend is now part of the midas repository, in shiftschedule/ (set up with its own setup.bash)
• Internal
  • More legacy C code moved to C++ (std::string, std::vector, odbxx instead of db_create_record())
  • Clients with the same name handled more consistently: cm_shutdown() matches exact client names, cm_exist() matches client or program name
  • msequencer command handling refactored into a command registry
  • Old mongoose 4/6 web server libraries and the old C-generated equipment page removed

• Web interface
  • custom_example.html is organised in tabs and now includes the mplot examples
  • New experiments show the example status page (Example) in the menu
  • Pages refresh faster for half a second after a value is written to the ODB, so hardware changes show up quickly.
  • Dialogs can be resized from any edge
  • History plots
    • Binned data is used for time ranges of 8 hours or more (was 1 day)
    • Better resizing of floating panels
    • Warning when the time range is too long for automatic scrolling
  • New midas icon
  • mhttpd
    • New --expdir option, needed when mhttpd connects through the mserver
    • With -D, mhttpd becomes a daemon only after the web server has started, so errors such as a port already in use are shown.
• Sequencer
  • Web page
    • Scripts can be edited in the main tab while the sequencer is stopped. A lock icon shows when it is running.
    • Editor tabs can be inserted and reordered
    • Better syntax validation with clearer error messages, also for Python scripts (e.g. indentation)
    • New Auto validate option checks the syntax before a script is started, saved or set as next, and refuses invalid scripts
  • MSL: WAIT ODBValue takes an optional timeout, e.g. WAIT ODBValue, <path>, <op>, <value>, timeout, <seconds>[, <status variable>]. The status variable is 1 if the condition was met and 0 if the timeout expired. The web page shows the timeout countdown.
  • New /Sequencer/Command/Goto line jumps to a line of the running script
  • PySequencer
    • Timeouts for wait_odb(), wait_events() and wait_func(), and more options for wait_func()
    • Timeout progress of waits is shown on the sequencer page
    • keep_modules_when_reloading_script() keeps modules (e.g. Rust/PyO3 ones) from being reloaded when a script starts
• Alarms: periodic alarms trigger exactly one period after the previous trigger, even if they were reset later
• History: faster reads from SQL databases with many schema versions
• ODB: better recovery when the ODB is full, and less fragmentation when allocating keys
• Frontends and drivers
  • mdev skips equipment that has Common/Enabled set to n
  • The mdev example frontends (mdev_fe, mdev_revpi_fe) are built by default
  • tmfe frontends handle binary RPCs (brpc) and the new jrpc_cxx/brpc_cxx calls
  • MSetPoint: non-working EPICS channels are disabled, and the page starts the EPICS frontend if it isn't running
• C++ API
  • New db_enum() returns the handles and keys of all subkeys of a directory
  • New db_delete() deletes a key by path
  • odbxx
    • TID_CHAR support
    • String arrays get the size of their longest string. New string_size()
    • Offline use: XML files without handle attributes can be read, and files are parsed only once
• Python API
  • Frontends support manually-triggered equipment (EQ_MANUAL_TRIG), triggered with manual_trig_client_call()
  • New odb_chmod() sets the access mode of an ODB key
  • New check_deferred_transition() for deferred transitions in programs that don't use the frontend framework
• Web API (JavaScript)
  • mplot
    • Bar plots with string X data get a category axis, and several bar datasets are drawn side by side (plot types)
    • Box of statistics for scatter plots, histograms, bar plots and color maps. data-stats="0" hides it
    • New addPlot(), modifyPlot() and deletePlot()
    • New mouseWheelZoom setting. Set it to false to scroll the page instead of zooming.
    • A maximize button opens the plot in a resizable dialog
    • mplot.js also works without midas (example in mplot.html)
    • An invalid plot type gives an error
  • modbgauge: data-color can be a list of value/color pairs, to draw colored segments
  • Sliders accept data-min-value and data-max-value
• Build
  • CMake writes compile_commands.json (e.g. for clangd)
  • New thread sanitizer option (cmake -DUSE_THREAD_SANITIZER=ON, or YES_THREAD_SANITIZER=1 with the Makefile)
  • The example CMake files need CMake 3.10 or newer
• Internal
  • Refactored ODB code (key lookup, deletion and locking), with new ODB torture tests in make test
  • Alarm classes use odbxx
  • Fixes for warnings from the address, thread and undefined behaviour sanitizers

• Sequencer
  • PySequencer is run with python3 $MIDASSYS/python/midas/sequencer.py and uses /PySequencer in the ODB. Only one instance can run at a time (use -c <name> for more).
  • PySequencer: wait_func(), more wait_odb() options, disable_tracing() for hot loops, numpy variables and params in the Variables list
  • PySequencer: logging to the midas message log (--log-to-midas, set_py_logger())
  • PySequencer: modules imported by a script are reloaded each time it runs
  • PySequencer creates an example script in userfiles/sequencer when first run
  • MSL: $variables in ODBLOOKUP, and an error if a $variable is not defined
  • MSL: the script stops if ODBLOOKUP fails or ODBSET writes beyond the end of an array
  • MSL: INCLUDE files may have an explicit .msl extension
  • MSL: # comments at the end of lines
• Web interface
  • Sequencer page supports PySequencer (Python syntax highlighting and validation, .py files, offers to start the program). New PySequencer menu entry (/Experiment/Menu/PySequencer).
  • Sequencer page: validation is also in the drop-down menu
  • Sequencer page: improved next file queue (added to the next free slot, length limit, drag-and-drop fixes)
  • Status page shows Restart: PySequencer when needed
  • ODB editor
    • Select keys and array elements by dragging
    • Integers can be edited in decimal or hex
    • Drop-down to set all values of a boolean array
    • The ODB picker for links starts in the previously used directory
  • Equipment page: device names, group lines without grid display, and a warning if the frontend is not running
  • Custom pages
    • /Custom menu entries without a file extension run JavaScript
    • The Status menu entry also opens the custom status page
  • Works behind F5 proxies (modbvalue links no longer use #)
  • mhttpd serves .ROOT files
• Web API (JavaScript)
  • mplot: time axis, error bars, data-bgcolor, and data-tooltip for custom hover text
  • RPC errors show the server response
• History
  • History data is flushed every second
  • History plots reload with one second of overlap, so late data is not lost
  • History editor handles tag names containing colons (e.g. EXP:HTR107:CUR)
  • Error if a history variable name starts with a space
  • New hs_define_panel2() taking vectors of variables, labels, formulas and colors
• Alarms: the alarm message says how many times the condition was met when Trigger count required is set
• Logger
  • Read cache increased to 1 MB
  • The history system is no longer re-opened at every run start
• Frontends and drivers
  • mfe: new set_write_cache_size() (see Breaking changes)
  • mfe: set_max_event_size() can be called in frontend_init(). The event buffer is set to twice the maximum event size.
  • mfe: messages in the message log when a frontend starts or stops
  • mfe: rates are computed over 1 s instead of 3 s
  • mdev: configurable readout period. MSCB driver gains groups, units, formats, conversion functions and error reporting.
• ODB
  • /Programs/<name>/Start command is filled with the full command line, including arguments
• C++ API
  • odbxx: key lookups are case-insensitive (like db_find_key()), and links are followed throughout
  • odbxx: resize(size, default) fills new elements with a default value
  • manalyzer: midas::odb uses the ODB from the run file when running offline
• Python API
  • Logging handler that copies messages to the midas message log
  • MidasFile can be used in a with statement
  • MidasClient can raise an exception if a client with the same name exists
  • Equipment overwrite_common argument (like mfe's equipment_common_overwrite)
  • Faster odb_set() for directories with many entries
• RPC and networking
  • Falls back to IPv4 if the kernel has no IPv6 support
  • The watchdog timeout set by cm_set_watchdog_params() also applies to the mserver process
  • New BM_TIMEOUT status: flushing the event buffer cache over RPC gives up instead of looping forever
• Build
  • find_package(Midas) can provide manalyzer as a component
  • New make clang-tidy target. AlmaLinux 10 and macOS build fixes.
• Internal: many fixes from static analyzers (cppcheck, clang-tidy), including checks for failed malloc()/fopen()

• Alarms: new Alarm sound setting for alarm classes, also in the Python helper
• Web interface
  • Equipment page: pager, units for extra display items
  • modbvalue shows arrays as comma-separated lists
  • modbselect works with arrays (shows ... if values differ)
• Frontends and drivers: mdev drivers take the event ID from the equipment
• Build: examples use the new CMake targets

• Web interface
  • Options <key> ODB keys show <key> as a drop-down list
  • History plots: binned display, touch zoom and a reworked panel editor
  • Custom pages can read and write files in <experiment>/userfiles (filesrw.js)
  • Custom pages
    • /Custom/Default start page and link: menu entries
    • setValue() on custom elements
    • data-confirm and data-input on modbvalue
  • Message notifications and search, hex view in the event dump
  • New MSetPoint application for beamline setpoints
• Sequencer
  • Several sequencers can run at the same time (msequencer -c <name>)
  • Full control through /Sequencer/Command
  • New EXIT, ATEXIT and ODBLOOKUP commands
  • SCRIPT output is stored in SCRIPT_RESULT
  • PARAM supports defaults, option lists and booleans
  • WAIT accepts fractional seconds
  • Relative INCLUDE paths
  • Scripts live in <experiment>/userfiles/sequencer
• History
  • Binned history reads via JSON-RPC
  • mh2sql can write to PostgreSQL
  • New experiments use FILE history by default
• ODB
  • The ODB is flushed to disk periodically (/System/Flush)
  • odbxx: automatic local/remote access, odb::save()/odb::load() and other new methods
  • New MIDAS_EXPT_NAME environment variable
• Alarms: conditions accept index lists and ranges (e.g. [1,3,5-7])
• Logger: new JSON runlog
• Frontends and drivers
  • The multi class driver supports formulas and units
  • New CAEN v1740 driver, EPICS frontend improvements
  • mfe no longer uses 100% CPU with only periodic equipment
• RPC and networking: new BRPC (JRPC returning binary data), also in tmfe and Python
• Python API
  • Read history and create history plots
  • Higher event rates and numpy support (about 10x faster)
  • BRPC, frontend_exit() and disconnect callbacks
• Build
  • armv7_remoteonly and arm64_remoteonly cross-compile targets
  • Coverage build type
  • make install also installs drivers, headers and cmake config
  • Examples are built by cmake
• Internal: much legacy char* code converted to std::string

• Web interface
  • Time zone selection for history plots and CSV exports
  • Submenus for custom pages
  • New modbvalue formats (thousands separators, %x, %b, ...)
  • The ODB page can show open records and clients
  • New JSON-RPC calls db_save, db_load and bm_receive_event
  • Rewritten sequencer page with debug mode
  • mhttpd exits if it cannot listen on all enabled ports
• History
  • Factor/offset migrated into formulas
  • Tag names with colons and infinite values supported
  • Faster MySQL and FILE backends
  • RTSP camera recording via OpenCV
• ODB
  • odbxx: new methods such as connect_and_fix_structure(). Arrays auto-enlarge by default
  • db_load() and odbedit load accept .json files
• Sequencer
  • New ODBCREATE, ODBDELETE, ODBSAVE, ODBLOAD, BREAK and MSG commands
  • PARAM default values
  • Recursive INCLUDE and deeper nesting
• Frontends
  • New mfed.cxx helper library and mdevice classes
  • Slow control reworked for multithreaded readout
  • New VXI-11 and mscbhv4 drivers
• Event buffers and RPC
  • Vector and scatter-gather send/receive functions
  • Sensible write cache limits (elog 2401)
• Logger and run control
  • New /Logger/Auto restart delay
  • More reliable transitions from mlogger
• Python API
  • Daemon mode for frontends
  • odb_watch reports which item changed
  • Alarm functions
  • New mmessenger forwards messages to Discord, Slack or Mattermost
• Build
  • midasio and manalyzer are now submodules
  • Exported cmake targets (midas-targets.cmake)
  • Default build type is RelWithDebInfo
  • New sanitizer and NO_* cmake options
• mdump reads .lz4 and .bz2 files
• Internal: many sanitizer fixes, old elog and sequencer code removed

midas-2022-05-b and midas-2022-05-c add:

• Faster odbxx subkey reading
• History request limit raised to 1000 MB
• Fixes for an mserver freeze and crash
• mdump displays up to 9999 events

• New ODB variable /Experiment/Enable sound can be used to globally prevent mhttpd from playing sounds
• modbvalue elements on custom pages now support an onload() callback as well as onchange(). Most elements now also support a data-validate callback.
• Minor improvements to mdump program: support for 64-bit data types and ability to load larger events if needed
• Minor improvements to History plots and Buffers webpage

• Sequencer page is now JavaScript-based and more responsive
• UTF-8 clean ODB (complains if any TID_STRING is invalid UTF-8)
• mhttpd updated to mongoose 6.16 with much improved multithreading
• mhttpd updated to use MBEDTLS in preference to problematic OpenSSL
• MidasConfig.cmake contributed by Mathieu Guigue

• New JavaScript-based sequencer page (via the "NewSequencer" link in the mhttpd menu)

• Reduced memory and CPU usage of mhttpd web pages
• Latest version of mxml
• Support for MySQL 8+, which removed the my_global.h header

• Stability improvements

• Location of most recent end-of-run ODB dump can be set using new ODB key /Logger/ODB Last Dump File
• New mhttpd_exec_script() JavaScript function to execute a custom script
• Stability and usability improvements

• mlogger stops if an event buffer is corrupted
• A program can connect to an existing event buffer even if it asks for a different buffer size
• The message page remembers the selected facility when the page is reloaded
• Improved parameter checking in odbedit

• New ODB key /Experiment/ODB timeout sets how long programs wait to lock the ODB
• New ODB key /Experiment/Enable core dumps
• Maximum number of banks in an event increased from 1024 to 4096
• Faster loading of JSON ODB files
• New preserve_case option for the db_get_values JSON-RPC method
• Tracing and timing of HTTP and JSON-RPC requests in mhttpd
• The git revision is included in the midas version string
• midas can be optionally compiled as C++