Midas C++ API - midas::odb (odbxx)

midas::odb is a modern C++ interface to the ODB. A midas::odb object behaves like a "magic dictionary" that is kept in sync with the ODB: reading a value from the object re-reads it from the ODB, and assigning a value writes it to the ODB straight away. There are no handles, sizes or TID_xxx types to manage, and values convert to and from the usual C++ types, including std::string, std::vector and std::array.

#include "midas.h"
#include "odbxx.h"

cm_connect_experiment("", "", "my_program", nullptr);

midas::odb exp("/Experiment");
int timeout = exp["Transition timeout"];   // Read from the ODB
exp["Transition timeout"] = timeout + 100; // Written to the ODB immediately

midas::odb needs C++11 or later. When accessing the live ODB, you must call cm_connect_experiment() before creating any object that refers to an ODB path. Objects that read a saved ODB dump (see Offline use) work without a midas connection.

The program examples/odbxx/odbxx_test.cxx in the midas repository exercises almost every feature described on this page.

Using midas::odb

Paths and values

A string that starts with / is an ODB path. Any other string is a plain string value:

midas::odb settings("/Equipment/Trigger/Settings"); // Connected to the ODB
midas::odb label("detector");                       // A local string value

Connecting to a path that doesn't exist creates it as an empty directory. If a misspelled path should be an error instead, check it first with midas::odb::exists(). Subkeys are accessed with [], and you can index with a relative path such as o["Logger/Write data"]. As in the rest of the ODB, key names are case-insensitive.

You can read a single value without keeping the object around:

int run_number = midas::odb("/Runinfo/Run number");

Keep an object for the directory when you access several values from it, so that the directory is only looked up once.

Reading and writing values

A value is read by converting it to a C++ type, and written by assigning to it. Increment, decrement and compound assignment (+=, -=, *=, /=) read the current value, update it and write it back. Binary arithmetic (o["Gain"] + 1.0) and comparisons (o["Gain"] > 1.0) return ordinary C++ values and don't change the ODB.

midas::odb o("/Equipment/Trigger/Settings");

int threshold = o["Threshold"];
std::string name = o["Name"];

o["Threshold"] = 30;
o["Counter"]++;
o["Gain"] *= 1.1;

Assigning to a subkey that doesn't exist creates it, with a type matching the C++ value. To read a value that might not exist yet, call the object with a default value. A missing key is created with that value, and an existing key keeps its current value:

double gain = o["New gain"](1.0);

Arrays and strings

ODB arrays convert to and from std::vector, and can also be assigned from a std::array. Use [] with an integer to access a single element, and size() and resize() to query and change the number of elements:

std::vector<float> hv = o["Voltages"];
o["Voltages"][1] = 1500.0f;
o["Voltages"] = std::vector<float>(16, 1200.0f);
o["Voltages"].resize(32);

for (float v : o["Voltages"])
   std::cout << v << "\n";

Some behaviours to be aware of:

• Writing to an element beyond the end of an array enlarges the array. Call set_auto_enlarge_array(false) to throw midas::odb_out_of_range instead
• Arithmetic operators applied to a whole array apply to every element, so o["Gains"] *= 2.0 doubles all the gains
• Strings are returned as std::string, and string arrays as std::vector<std::string>. Assigning a shorter string may shrink the ODB string length; call set_preserve_string_size(true) to keep the existing length

Iterating over subkeys

Iterating over a directory gives one midas::odb object per subkey. Use named_items() with structured bindings if you also want the name, and contains() to check whether a subkey exists without creating it:

midas::odb exp("/Experiment");

for (midas::odb& subkey : exp)
   std::cout << subkey.get_name() << " = " << subkey << "\n";

for (auto& [name, subkey] : exp.named_items())
   std::cout << name << " = " << subkey << "\n";

for (auto it = exp.begin(); it != exp.end(); ++it) {
   midas::odb& subkey = *it;
   std::cout << subkey.get_full_path() << "\n";
}

if (exp.contains("Transition timeout"))
   std::cout << "Found it\n";

named_items() throws midas::odb_type_error if the object isn't a directory. items() returns the object itself, so for (midas::odb& subkey : exp.items()) is the same as the first loop. contains() accepts a direct subkey name or a relative path.

Key information and management

Objects can tell you about the ODB key they represent:

midas::odb o("/Equipment/Trigger/Settings");

std::string path   = o.get_full_path();   // "/Equipment/Trigger/Settings"
std::string parent = o.get_parent_path(); // "/Equipment/Trigger"
std::string name   = o.get_name();        // "Settings"
int         n      = o.size();            // Number of values (1 for a single value)
bool        dir    = o.is_directory();    // A directory
bool        array  = o.is_array();        // A value with more than one element
bool        scalar = o.is_scalar();       // A single value
std::string type   = o.type_name();       // e.g. "INT32", "STRING" or "KEY"
HNDLE       hKey   = o.get_hkey();        // For use with the low-level db_xxx functions

For an array, type_name() gives the type of its elements. An empty object (not yet assigned or connected) has the type name "NULL", and isn't a directory, array or single value.

Some static functions work on ODB paths without creating an object. Paths are always relative to the top of the ODB:

bool found = midas::odb::exists("/Equipment/Trigger/Settings");
midas::odb::create("/Equipment/Trigger/Settings/Gain", TID_FLOAT); // Create one key
midas::odb::delete_key("/Equipment/Trigger/Settings/Gain");

To delete the key an object represents, call o.delete_key(). Unlike accessing a missing subkey with [], exists() never creates anything.

Defining a directory structure

A program usually expects its settings in a particular layout. You can describe that layout, with default values, using an initializer list, then connect() it to the ODB:

midas::odb settings = {
   {"Threshold (mV)", -30},                      // TID_INT32
   {"Scale factor", 1.5f},                       // TID_FLOAT (note the f)
   {"Enabled", true},                            // TID_BOOL
   {"Label", "detector"},                        // TID_STRING
   {"Gains", {1.0, 1.1, 1.2}},                   // TID_DOUBLE array
   {"Masks", std::array<int, 32>{}},             // TID_INT32 array with 32 elements
   {"Comment", std::string(255, '\0')},          // TID_STRING of length 255
   {"Calibration", {{"Offset", 0.0}, {"Gain", 1.0}}} // Subdirectory
};

settings.connect("/Equipment/Trigger/Settings");

The ODB type is taken from the C++ type, so write 1.5f for a TID_FLOAT (1.5 would create a TID_DOUBLE). Use std::array to create an array of a fixed size, and a string of '\0' characters to reserve the length of a string.

There are three ways to connect the structure, which differ in how they treat keys that already exist:

In all cases, an existing key whose type differs from your default is deleted and re-created with your default type and value. connect_and_fix_structure() behaves like the old db_check_record() function. You can't connect a structure to the root of the ODB (/).

Comparing trees

Use compare() to check two midas::odb objects without writing either tree to the ODB or a file:

std::string compare(const odb& other,
                    const std::vector<std::string>& ignore_values = {}) const;

The result is an empty string if the trees match, or a diagnostic string containing the differences. Diagnostics contain relative key paths, zero-based array indices, and neutral reasons: missing, type differs, array size differs, or differs. They do not identify either object or include raw values, so null characters in string values cannot truncate a message passed to cm_msg().

midas::odb expected = {
   {"Threshold", 60.f},
   {"Enabled", true},
   {"Calibration", {{"Gain", 1.0}}}
};
midas::odb existing("/Equipment/Trigger/Settings");

std::string differences = expected.compare(existing);
if (!differences.empty())
   cm_msg(MERROR, "frontend_init", "Settings differ:%s", differences.c_str());

For example, a different threshold produces a diagnostic such as:

  Threshold[0]: differs

The comparison recursively checks missing keys on either side, MIDAS types, array sizes, and values. It reports every differing array element. Key order, root names, access modes, timestamps, and reserved string capacity do not affect equality. Floating-point values are compared exactly, with two NaNs treated as equal.

By default, every key and value is checked. The optional ignore_values list skips values at exact relative key paths, while still checking that those keys exist and have matching types and array sizes. For example, operational switches can retain their current state while their array lengths are checked:

std::string differences = expected.compare(existing, {"Output On", "Enabled"});

Application-specific context belongs in the caller's error message:

std::string differences = expected.compare(existing, {"Output On", "Enabled"});
if (!differences.empty())
   cm_msg(MERROR, "frontend_init", "Settings differ between frontend code and ODB:%s",
          differences.c_str());

A key present in only one object is reported as Threshold: missing; use compare_details() to identify which object lacks the key.

Use a path such as Subdir/Enabled for a nested value. Listing a directory does not exclude its descendants. A size mismatch is included in the returned diagnostics even for an excluded value; the caller can report it with cm_msg(MERROR, ...) and throw to stop initialization.

For decisions based on the kind of difference, use the structured result from compare_details():

std::vector<midas::odb::difference> differences = expected.compare_details(existing);
for (const auto& entry : differences) {
   if (entry.kind == midas::odb::difference_kind::missing_in_other)
      continue; // This key may be added later with expected.connect(path).
   std::cerr << entry.message() << "\n";
}

Each entry contains a relative path and a kind: missing_in_this, missing_in_other, type, array_size, or value. this is the object on which the method is called; other is its argument. entry.message() returns the same neutral diagnostic text used by compare(). A completely missing directory is reported once at its path rather than once per descendant. Both methods accept the same ignore_values argument and leave the objects unchanged.

For example, a frontend can allow missing_in_other when comparing its code defaults with existing ODB settings, while treating all other differences as errors. Only after validating all differences should it unlock the settings and call expected.connect(path) to add missing keys and retain existing values. Comparison itself never creates the missing keys.

If automatic reads are enabled on an input object, compare() refreshes a private copy from its backing ODB before comparing. If automatic reads are disabled, it compares the current local values. Neither input object is modified. The method also works with file- and string-backed objects. A mismatch is returned as text; ODB read failures can still throw the usual exceptions.

Arrays filled with a default value

Use midas::odb::array(value, size) to construct an array of a runtime size, with every element initialized to the same value:

const size_t channels = 18;
midas::odb settings = {
   {"Enabled", midas::odb::array(true, channels)},
   {"Output On", midas::odb::array(false, channels)},
   {"Voltage limit", midas::odb::array(60.f, channels)},
   {"Names", midas::odb::array(std::string(31, '\0'), channels)}
};

settings.connect("/Equipment/HV/Settings");

The MIDAS type is inferred from value, just as for scalar defaults: true produces TID_BOOL, 60.f produces TID_FLOAT, and 60.0 produces TID_DOUBLE. Strings are copied independently for each element. A string beginning with / is a literal value, not an ODB path.

The factory allocates and fills the array directly in local memory, leaving automatic reads and writes at their normal default of true. Once connected, the array therefore has the same automatic read/write behavior as other ODB values; no explicit flag changes are needed. To build or edit a local tree without writing changes, explicitly disable automatic reads and writes on that tree.

size must be between 1 and INT_MAX, inclusive; zero and oversized lengths throw midas::odb_out_of_range. A size of 1 produces a single-value key. A null-filled string such as std::string(31, '\0') reserves the initial string length for every element.

Automatic reads and writes

By default, every read of a value fetches it from the ODB, and every assignment writes it to the ODB. This means your values are always up to date, but each access costs an ODB operation. For a program connected to a remote experiment, each operation is a network round trip, so reading 1000 values one at a time can take several seconds.

There are two ways to reduce this cost:

• Read a tree in one go: pass true as the second constructor argument (midas::odb o("/Test/Settings", true)). This fetches the whole tree as a single XML transfer, instead of one read per value. It only speeds up creating the object; later reads and writes are still individual ODB operations
• Turn off automatic reads and writes: call set_auto_refresh_read(false) and/or set_auto_refresh_write(false). Reads then return the local copy, and assignments only change the local copy. Call read() to refresh the local copy from the ODB, and write() to send it to the ODB

midas::odb o("/Test/Settings", true);
o.set_auto_refresh_read(false);
o.set_auto_refresh_write(false);

for (int i = 0; i < 10; i++)
   o["Values"][i] = i; // Only changes the local copy

o.write(true); // Write only the values that have changed

While automatic writes are off, every change marks the value as "dirty". write() writes every value in the tree, while write(true) only writes the dirty values and clears their dirty flags. Arrays have one dirty flag for the whole array, so changing any element writes the whole array.

To read or write a single array element explicitly, use read(index) and write(index, str_size) (str_size sets the ODB string length for strings, or 0 for the default). Note that if you have selected an element with [] since the last write, write() only writes that element.

When automatic reads and writes are off, keeping the local copy in step with the ODB is your responsibility. Your reads won't see changes made by other programs until you call read() (use a watch if you need to know when that happens), and a later write() can overwrite newer ODB values with your older local values.

midas::odb::set_debug(true) prints every ODB operation that midas::odb performs, which is a useful way to check how many you're making.

Behaviour flags

Each object has a set of flags that control its behaviour. They are read with is_xxx() and changed with set_xxx(), and changing a flag on a directory also changes it for everything inside it.

Objects that read a saved ODB dump use different defaults; see Offline use.

Watching for changes

watch() registers a function that is called whenever a value in the object changes in the ODB, whoever changed it. It's passed the midas::odb object of the key that changed, which for a directory may be a subkey rather than the directory you watched:

midas::odb settings("/Equipment/Trigger/Settings");

settings.watch([](midas::odb& changed) {
   std::cout << changed.get_full_path() << " = " << changed << "\n";
});

while (cm_yield(100) != RPC_SHUTDOWN) {
   // Callbacks are called from within cm_yield()
}

Callbacks are only called from within cm_yield(), so your program must call it regularly. Frontends using mfe or TMFE do this for you.

The callback can be a lambda, a free function, or a member function bound to an object with std::bind:

class MyEquipment {
public:
   void settings_changed(midas::odb& changed) {
      // Apply the changed value to the hardware
   }

   void init() {
      midas::odb settings("/Equipment/Trigger/Settings");
      settings.watch(std::bind(&MyEquipment::settings_changed, this, std::placeholders::_1));
   }
};

For an array, get_last_index() inside the callback returns the index of the element that changed (0 is the first element), or a negative value if the whole array was written. Call it inside the callback, as it is reset afterwards:

settings.watch([](midas::odb& changed) {
   int index = changed.get_last_index();
   if (changed.is_array() && index >= 0) {
      int value = changed[index];
      std::cout << changed.get_full_path() << "[" << index << "] = " << value << "\n";
   }
});

watch() keeps its own copy of the tree, so the callback carries on after the original object is destroyed (as in the init() example above). To remove it, call unwatch() on any object for the same ODB key.

Offline use

The same code can read a saved ODB dump instead of the live ODB, which is useful in analysis programs. Call set_odb_source() before creating your objects:

midas::odb::set_odb_source(midas::odb::FILE, "run00123.json"); // JSON or XML file
// or: midas::odb::set_odb_source(midas::odb::STRING, dump);   // JSON or XML in a string

midas::odb settings("/Equipment/Trigger/Settings");
int threshold = settings["Threshold (mV)"];

midas::odb::set_odb_source(midas::odb::ONLINE); // Back to the live ODB

ODB: programming shows how to get the dump from a midas data file.

If you need the live ODB and a dump at the same time, use midas::odb::file() or midas::odb::string() instead. These return an independent tree for a JSON file or string, without changing the source used by other objects:

midas::odb live("/Equipment/Trigger/Settings");
midas::odb saved = midas::odb::file("run00123.json");

int live_threshold = live["Threshold (mV)"];
int saved_threshold = saved["Equipment"]["Trigger"]["Settings"]["Threshold (mV)"];

Offline objects never change the live ODB. Automatic reads, writes, key creation, array enlargement and hotlink triggering are all off, and assigning to a value only changes the in-memory copy. read() and write() only work with the live ODB (write() throws an exception for an offline object). To keep your changes, use:

• save(filename) to write the object to a JSON file. Saving / keeps the full hierarchy of the original dump, while saving a subtree writes that subtree as the top-level key
• save() to write an object from file() back to the file it was read from
• dump() to get the object as a JSON string, which string() can load again

A saved file can be loaded into the live ODB with midas::odb::load(). save() writes the object's directory itself as the top-level key, so load the file into the parent directory to restore it in the same place. See ODB: programming for saving and loading with the other APIs.

Errors

midas::odb reports errors by throwing exceptions. All of them can be caught as midas::odb_error, and there are more specific classes for the common types of error (listed under Exceptions):

try {
   settings.set_auto_create(false);
   int threshold = settings["Threshold"];
} catch (const midas::odb_missing_key& e) {
   std::cerr << "Missing setting: " << e.what() << "\n";
} catch (const midas::odb_error& e) {
   std::cerr << "Other ODB error: " << e.what() << "\n";
}

For compatibility with older code, midas::odb_out_of_range also derives from std::out_of_range, and all the other exceptions derive from mexception (and so can be caught as std::runtime_error).

Example

This program defines its settings with default values, prints them, and then prints any changes until it is stopped:

#include <iostream>
#include "midas.h"
#include "odbxx.h"

void settings_changed(midas::odb& changed) {
   std::cout << changed.get_full_path() << " changed to " << changed << "\n";
}

int main() {
   cm_connect_experiment("", "", "odbxx_example", nullptr);

   // Create any settings that don't exist yet, and keep the values of those that do
   midas::odb settings = {
      {"Threshold (mV)", -30},
      {"Width (ns)", 20},
      {"Coincidence", true},
      {"Gains", std::array<float, 16>{}},
      {"Channel labels", std::array<std::string, 16>{}}
   };
   settings.connect("/Equipment/Trigger/Settings");

   int threshold = settings["Threshold (mV)"];
   std::vector<float> gains = settings["Gains"];
   std::cout << "Threshold is " << threshold << " mV, gain[0] is " << gains[0] << "\n";

   settings.watch(settings_changed);

   // cm_yield() calls settings_changed() when a setting changes
   while (true) {
      int status = cm_yield(100);
      if (status == SS_ABORT || status == RPC_SHUTDOWN)
         break;
   }

   cm_disconnect_experiment();
   return 0;
}

Reference

Creating and connecting

odb()

odb(const odb &o)

template<typename T>
odb(T v)

odb(std::initializer_list<std::pair<const char *, midas::odb>> list)

template<typename T>
odb(std::initializer_list<T> list)

template<typename T, size_t SIZE>
odb(const std::array<T, SIZE> &arr)

template<size_t SIZE>
odb(const std::array<std::string, SIZE> &arr)

odb(const std::string &path, bool init_via_xml = false)

odb(const char *s)

odb(std::initializer_list<const char *> list)

void connect(const std::string &path, const std::string &name, bool write_defaults, bool delete_keys_not_in_defaults = false)

void connect(std::string str, bool write_defaults = false, bool delete_keys_not_in_defaults = false)

void connect_and_fix_structure(std::string path)

static bool exists(const std::string &name)

static int create(const char *name, int type = TID_KEY)

static int delete_key(const std::string &name)

void delete_key()

static void load(const std::string &filename, const std::string &odb_path)

Values and arrays

template<typename T, typename std::enable_if< std::is_same<T, uint8_t>::value || std::is_same<T, int8_t>::value || std::is_same<T, uint16_t>::value || std::is_same<T, int16_t>::value || std::is_same<T, uint32_t>::value || std::is_same<T, int32_t>::value || std::is_same<T, uint64_t>::value || std::is_same<T, int64_t>::value || std::is_same<T, bool>::value || std::is_same<T, float>::value || std::is_same<T, double>::value, T>::type * = nullptr>
operator T()

operator std::string()

template<typename T>
operator std::vector<T>()

operator std::vector<std::string>()

odb operator=(odb &&o) = delete

template<typename T>
const T &operator=(const T &v)

template<typename T>
const std::vector<T> &operator=(const std::vector<T> &v)

template<typename T, size_t SIZE>
const std::array<T, SIZE> &operator=(const std::array<T, SIZE> &arr)

template <typename T>
odb&operator()(T v)

u_odb &operator[](int index)

odb &operator[](std::string str)

odb &operator[](const char *str)

template<typename T>
T operator+(T i)

template<typename T>
T operator-(T i)

template<typename T>
T operator*(const T i)

template<typename T>
T operator/(const T i)

odb &operator++()

odb operator++(int)

odb &operator--()

odb operator--(int)

odb &operator+=(double d)

odb &operator-=(double d)

odb &operator*=(double d)

odb &operator/=(double d)

std::ostream &operator<<(std::ostream &output, odb &o)

int size()

void resize(int size)

void resize(int size, bool b)

template<typename T>
static odb array(T value, size_t size)

std::string s()

void set(std::string str)

void set(std::string s, int i)

void set_string_size(std::string s, int size)

void deep_copy(odb &d, const odb &s)

Subkeys

iterator begin() const

iterator end() const

odb&items()

named_item_range named_items()

bool contains(std::string str)

bool is_subkey(std::string str)

void fix_order(std::vector<std::string> target_subkey_order)

Comparing

std::string compare(const odb &other, const std::vector<std::string> &ignore_values = {}) const

std::vector<difference> compare_details(const odb &other, const std::vector<std::string> &ignore_values = {}) const

Reading, writing and saving

void read()

void read(int index)

void write(bool dirty = false, int str_size = 0)

void write(int str_size)

void write(int index, int str_size)

void save()

void save(const std::string &filename)

std::string dump()

void dump(std::string &s, int indent = 0)

std::string print()

void print(std::string &s, int indent = 0)

Watching

void watch(std::function<void(midas::odb &)> f)

void unwatch()

int get_last_index()

void set_last_index(int i)

Key information

get_mode() and set_mode() use MODE_READ (1, R), MODE_WRITE (2, W), MODE_DELETE (4, D) and MODE_EXCLUSIVE (8, X in the ODB browser). The browser's H indicates an active hotlink from the notification count; it is not a mode bit and is not set with set_mode(). See changing permissions.

std::string get_name()

std::string get_full_path()

std::string get_parent_path()

HNDLE get_hkey()

int get_tid()

std::string type_name() const

int get_num_values()

bool is_directory() const

bool is_array() const

bool is_scalar() const

unsigned int get_last_written()

int get_mode()

void set_mode(int mode)

Flags

bool is_auto_refresh_read() const

void set_auto_refresh_read(bool f)

bool is_auto_refresh_write() const

void set_auto_refresh_write(bool f)

bool is_auto_create() const

void set_auto_create(bool f)

bool is_auto_enlarge_array() const

void set_auto_enlarge_array(bool f)

bool is_write_protect() const

void set_write_protect(bool f)

bool is_trigger_hotlink() const

void set_trigger_hotlink(bool f)

bool is_preserve_string_size() const

void set_preserve_string_size(bool f)

bool is_dirty() const

void set_dirty(bool f)

ODB source and debugging

enum odb_source

static void set_odb_source(odb::odb_source s, std::string str)

static void set_odb_source(odb::odb_source s)

static odb::odb_source get_odb_source()

static bool is_connected_odb()

static odb file(const std::string &filename)

static odb string(const std::string &buffer)

static void set_debug(bool flag)

static bool get_debug()

Exceptions

virtual ~odb_error() = default

virtual const char *what() const noexcept = 0

class odb_runtime_error : public odb_error, public mexception

const char *what() const noexcept override

class odb_missing_key : public odb_runtime_error

class odb_type_error : public odb_runtime_error

class odb_connection_error : public odb_runtime_error

class odb_io_error : public odb_runtime_error

class odb_read_only : public odb_runtime_error

class odb_out_of_range : public odb_error, public std::out_of_range

const char *what() const noexcept override

class mexception : public std::runtime_error

mexception(const std::string &arg, bool stack_trace, const char *file, int line)

mexception(const mexception &ex) noexcept

std::ostream &operator<<(std::ostream &output, mexception &m)

const char *what() const

~mexception()

Comparisons

template<typename T>
bool operator==(const midas::odb &o, const T &d)

template<typename T>
bool operator==(const T &d, const midas::odb &o)

template<typename T>
bool operator!=(const midas::odb &o, const T &d)

template<typename T>
bool operator!=(const T &d, const midas::odb &o)

template<typename T>
bool operator<(const midas::odb &o, const T &d)

template<typename T>
bool operator<(const T &d, const midas::odb &o)

template<typename T>
bool operator<=(const midas::odb &o, const T &d)

template<typename T>
bool operator<=(const T &d, const midas::odb &o)

template<typename T>
bool operator>(const midas::odb &o, const T &d)

template<typename T>
bool operator>(const T &d, const midas::odb &o)

template<typename T>
bool operator>=(const midas::odb &o, const T &d)

template<typename T>
bool operator>=(const T &d, const midas::odb &o)