History System: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
==== Links ====
== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[/History ODB tree]]
Line 9: Line 9:
</div>
</div>


== Introduction ==
= Introduction =


The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging. This information is recorded with a time stamp and saved into a history file (in a  [[Mhformat|special format]] of the form of a database) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a history file (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.


<div id="Three ways history saved"></div>
= MidasHistoryInterface =
There are currently three ways the history data may be saved
 
; MIDAS history : history is saved into a MIDAS History file. By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#/Logger/History/0/ subtree |history channel 0]]''' is set up to write '''MIDAS''' history.  
New history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The mlogger can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h
; ODBC history  : history is saved using ODBC driver. By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#/Logger/History/1/ subtree |history channel 1]]'''  is set up to write '''ODBC''' history.
 
; SQLITE history : history data saved into an SQL database '''[[Keys in the ODB /Logger/History subtree#/Logger/History/0/ subtree |history channel 2]]'''  is set up to write '''SQLITE''' history.
* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h
 
= History drivers =
 
The MIDAS distribution contains these drivers:
 
; MIDAS history  
: the traditional MIDAS history, data is stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#/Logger/History/0/ subtree |history channel 0]]''' is set up to write '''MIDAS''' history.  
 
; ODBC history   
: stores history data in SQL database using the ODBC abstraction layer. MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC driver is easy to add.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#/Logger/History/1/ subtree |history channel 1]]'''  is set up to write '''ODBC''' history.
 
; SQLITE history  
: stores data in SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC history, performance is similar, no database server required.
: history data saved into an SQL database '''[[Keys in the ODB /Logger/History subtree#/Logger/History/0/ subtree |history channel 2]]'''  is set up to write '''SQLITE''' history.
 
 
More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_histrory(), create the ODB /Logger/History/x/channel).
 
= Location of History Files =


== Location of History Files ==
MIDAS history is enabled by default, writing history files into the MIDAS data directory  path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
MIDAS history is enabled by default, writing history files into the MIDAS data directory  path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).
If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).
Line 27: Line 48:
[[Keys in the ODB /Logger tree#History dir|History dir]].  
[[Keys in the ODB /Logger tree#History dir|History dir]].  


== Types of History Events ==
= Types of History Events =


There are two basic types of history events, which are defined in different ways:
There are two basic types of history events, which are defined in different ways:


*    "Frontend" History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
*    "[[#Frontend History Event|Frontend]]" History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
*    "Virtual" History event composed within the ODB under the specific tree /History/Links (see ODB History tree)
*    "Virtual" History event composed within the ODB under the specific tree /History/Links (see ODB History tree)


Line 38: Line 59:




=== Frontend History Event ===
= Frontend History Event =


Each  [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".
Each  [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".


==== To write a Frontend History Event ====
== To write a Frontend History Event ==


* Review [[#Location of History Files|location of History Files]]
* Review [[#Location of History Files|location of History Files]]
* Set the [[/Equipment ODB tree#Log history|Log history]] key of each equipment to be logged non-zero.
* Set the [[/Equipment ODB tree#Log history|Log history]] key of each equipment to be logged non-zero. This causes the event contents to be copied to the  ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system .
* Select  '''[[#Three ways history saved|channel]]''' to write the history files  
* Select the '''[[#History drivers|history driver]]''' and therefore '''history channel''' to use to write the history files.
* Set [[Keys in the ODB /Logger/History subtree#/Logger/History/0/Active|Active]]
* set [[Keys in the ODB /Logger/History subtree|Active]] ODB key in the selected [[#History drivers|History channel]] to "y"
* set [[Keys in the ODB /Logger/History subtree|Active]] ODB key in the selected [[#Three ways history saved|channel]] to "y"
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] to the selected [[#History drivers|History channel]]
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] to the appropriate [[#Three ways history saved|channel]]
* restart [[mhttpd]] and [[mlogger]], start the run.
* restart [[mhttpd]] and [[mlogger]], start the run.


In the  [[#Location of History Files|history data directory]] you should see history files with name "YYMMDD.hst" corresponding to each day of midas operation. Contents of .hsr files can be examined directly using [[mhdump]] or indirectly using [[mhist]] or the history display page of [[mhttpd]].
History files will be saved into the  [[#Location of History Files|history data directory]].  
 
Enabling the history system for an Equipment causes the event contents to be copied to the  ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system . The contents of the event will then be completely copied into the history files.
 
 
In the case of MIDAS history, the definition of the event will be used as tag names for every element of the event.




The main data path for the frontend History Event is as follows:
The main data path for the frontend History Event is as follows:


    The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
    In mfe.c, if /Equipment/<equipment-name>/Common/Log history > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
 
* This ODB write triggers an ODB hot-link into mlogger
The frequency of ODB writes is limited by ODB_UPDATE_TIME (1/sec in mfe.c rev 4298).
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
 
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.
    The odb write triggers an odb hot-link into mlogger
    The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key /Equipment/<Equipment-Name>/Common/Log history.
    history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.
 
The internal structure of MIDAS history files is documented in History format .
 




Line 82: Line 91:


;During the transition period:
;During the transition period:
:   Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' <span style="color:purple; font-style:italic">/History/Tags</span>. The history tags are read directly from the history files themselves. To run in this mode, set  ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.
:   Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' <span style="color:purple; font-style:italic">/History/Tags</span>. The history tags are read directly from the history files themselves. To run in this mode, set  ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.


Two downsides to running like this are
Two downsides to running like this are
Line 93: Line 102:
However, the format of the tags is now changed to simplify management and reduce ODB size.  
However, the format of the tags is now changed to simplify management and reduce ODB size.  


: If you have old-style tags
; For NEW implementations
; [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].
 


* using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
;If you have old-style tags
* using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).


** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.


: For new implementations
; the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].




Line 114: Line 124:
:                The size of this array must match the size of the /equipment/equipment-name/Variables/<bankname>[ ] array, or an error will be generated.
:                The size of this array must match the size of the /equipment/equipment-name/Variables/<bankname>[ ] array, or an error will be generated.
:            Otherwise the history name is composed of the bank name followed by the corresponding index of the element.
:            Otherwise the history name is composed of the bank name followed by the corresponding index of the element.
 
See example
        See example .


;In the case of a Fixed format event :
;In the case of a Fixed format event :
*No array : the names of the individual element under /Equipment/equipment-name/Variables/ are used for the history name composition.
*No array : the names of the individual element under /Equipment/equipment-name/Variables/ are used for the history name composition.
*With array: if the key /Equipment/equipment-name/Settings/Names[ ] exists, each element of the array is referenced using the corresponding name of the ../Settings/Names[ ] array.
*With array: if the key /Equipment/equipment-name/Settings/Names[ ] exists, each element of the array is referenced using the corresponding name of the ../Settings/Names[ ] array.

Revision as of 17:34, 12 June 2014

Links

Introduction

The history system is an add-on capability built into the MIDAS data logger mlogger to record useful information in parallel to the data logging. This information is recorded with a time stamp and saved into a history file (see History drivers) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the mhttpd MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time. The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any data logging channel enabled.

MidasHistoryInterface

New history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The mlogger can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

History drivers

The MIDAS distribution contains these drivers:

MIDAS history
the traditional MIDAS history, data is stored in the .hst binary files. (format is documented here mhformat, see also mhist and mhdump)
By default ODB subtree history channel 0 is set up to write MIDAS history.
ODBC history
stores history data in SQL database using the ODBC abstraction layer. MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC driver is easy to add.
By default ODB subtree history channel 1 is set up to write ODBC history.
SQLITE history
stores data in SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC history, performance is similar, no database server required.
history data saved into an SQL database history channel 2 is set up to write SQLITE history.


More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_histrory(), create the ODB /Logger/History/x/channel).

Location of History Files

MIDAS history is enabled by default, writing history files into the MIDAS data directory path given by the ODB key Data dir. If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

The location of the history data files can be changed by use of the ODB Key History dir.

Types of History Events

There are two basic types of history events, which are defined in different ways:

  • "Frontend" History event composed in the frontend. See Frontend History Event for details.
  • "Virtual" History event composed within the ODB under the specific tree /History/Links (see ODB History tree)

Both these history event definitions take effect when the data logger gets a start run transition. Any modifications made during the run are not applied until the start of the next run.


Frontend History Event

Each Equipment has the capability to generate "history data".

To write a Frontend History Event

History files will be saved into the history data directory.


The main data path for the frontend History Event is as follows:

  • The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
  • In mfe.c, if the key Log history > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
  • This ODB write triggers an ODB hot-link into mlogger
  • The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key Log history.
  • history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.


History Tag names

Tags are only used for MIDAS history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.


During the transition period
Implementations starting with Version 4435 of mlogger and mhttpd (i.e. "new" versions) will now work without /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key /History/DisableTags to "y". Existing /History/Tags subtree will be automatically deleted.

Two downsides to running like this are

  • it is slower and tags become non-persistent
  • if some frontends have not been running for a while, their variables may vanish from the history panel editor.


New-style History Tags

For the above two reasons, using /History/Tags is still recommended. However, the format of the tags is now changed to simplify management and reduce ODB size.

For NEW implementations
the key CreateOldTags will not be present. "New-style" history tags will be created automatically in the Tags subtree.


If you have old-style tags
mlogger will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).
    • using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
    • using "old" mhttpd with "new" mlogger is OK provided the ODB key /History/CreateOldTags is set to "y" before starting mlogger.


Old-style History Tags

Prior to versions 4435 of mlogger and mhttpd, old-style history tags were generated. The history variable name for each element of the Frontend History Event was composed as follows, where

  • "equipment-name" is the name of the equipment, and
  • "bankname" is the name of a bank generated by the "equipment-name" equipment :
In the case of a MIDAS format event
If the ODB key /Equipment/equipment-name/Settings/Names <bankname>[ ] IS present,
the history name is composed of the corresponding name found in the Names <bankname>[ ] array.
The size of this array must match the size of the /equipment/equipment-name/Variables/<bankname>[ ] array, or an error will be generated.
Otherwise the history name is composed of the bank name followed by the corresponding index of the element.

See example

In the case of a Fixed format event
  • No array : the names of the individual element under /Equipment/equipment-name/Variables/ are used for the history name composition.
  • With array: if the key /Equipment/equipment-name/Settings/Names[ ] exists, each element of the array is referenced using the corresponding name of the ../Settings/Names[ ] array.