Skip to content

Data logging: archiving (lazylogger)

The midas lazylogger allows you to run a script on every midas file that is written to disk. It is not designed for running time-consuming analysis packages, but rather for quick tasks such as archiving, renaming or copying the files.

There is built-in support for copying files to disk/tape, and copying using FTP/SFTP. You can also write a custom script to perform the copy, for example if you want to rename a file or copy to the grid etc.

Multiple channels

You may have multiple copies of the lazylogger running simultaneously, each running independently. Each instance handles a "channel" of data. The channel name is specified when you start the lazylogger program.

Starting the lazylogger

The first time you run the lazylogger, you should:

  1. Choose your channel name (e.g. MyBackup)
  2. Run the lazylogger and stop it (e.g. lazylogger -c MyBackup, then hit Ctrl+C)
  3. Edit the settings in the ODB (e.g. at /Lazy/MyBackup/Settings)
  4. Run the lazylogger in the background (e.g. lazylogger -c MyBackup -D)
Full usage of the lazylogger tool 
Usage: lazylogger [-D] [-h <Hostname>] [-e <Experiment>] [-c <channel name>]
Argument Meaning
[-h hostname ] see common parameters
[-e experiment_name ] see common parameters
[-D ] start program as a daemon.
[-c channel ] logging channel name. Specify the lazylogger to activate.
[-z ] zap statistics. Clear the statistics tree of this channel.
[-t ] permit lazylogger to TALK msg.
[-n ] No file purging on the source dir.
[-d ] Additional debug messages
[-C ] Convert the list of copied files from the format used by old midas versions, then exit.

See lazylogger for more details of each option.

Configuration

For all the settings, see /Lazy/xxx/Settings.

The settings of each lazylogger channel are in the ODB in /Lazy/<channel_name>/Settings (e.g. /Lazy/MyBackup/Settings).

There are many settings, some of which may not apply to your backup method.

Critical keys

The critical keys to set are:

  • Data dir and Filename format (so lazylogger knows which files to process)
    • The default Filename format is run%05d.mid. This doesn't match the .mid.lz4 files that mlogger writes by default, so you will usually need to change it.
  • Backup type and Path (so lazylogger knows what to do)

The other keys likely only need to be tweaked in complex situations.

Files in subdirectories are not copied

lazylogger only looks for files directly in Data dir, not in its subdirectories. If mlogger writes files into subdirectories (using Subdir format in /Logger/Channels/xxx/Settings), lazylogger won't find them. Leave Subdir format empty for the mlogger channels whose files lazylogger should copy.

Running scripts

To process a file

If you set a Backup type of Script, then your custom script will be called for each file that gets written. Specify the path to your script in the Path key. This script could implement custom logic for moving or renaming files, or do some quick processing of the file. The script can be written in any language.

Script arguments

The script will be called with one argument - the full path to the midas file to process.

It should return 0 if it processed the file successfully, and >0 if it failed. lazylogger will automatically retry files that failed. Anything written by the script to stdout will appear in the midas message log.

Note that only 1 instance of the script is ever run at once - if you are writing new files every 60s, but your script takes 90s to execute, you will get a backlog of data and should consider using a proper queue system to implement parallel processing, rather than using the lazylogger.

Example scripts

Remember to return a non-zero exit code if processing the file failed, so that lazylogger will retry later!

import sys

if __name__ == "__main__":
    path = sys.argv[1]

    some_error = False

    if some_error:
        print(f"Failed to process {path}!")
        exit(1)

    print(f"I am processing {path}!")

#!/bin/bash
path=$1

some_error=0

if [ $some_error -eq 1 ]; then
    echo "Failed to process $path"
    exit 1
fi

echo "I am processing $path"

Before processing a file

You can run a script before each file gets processed.

For Script, Disk, SFTP, the script gets called with two arguments - the path to the source file, and the number of files that have been processed so far. It does not matter what the script returns - failures are ignored.

For Tape, the script gets called with five arguments - the path to the source file, path to the destination file, the tape block number, the "list label", and the number of files that have been processed so far. It does not matter what the script returns - failures are ignored.

For FTP, the script does not appear to get called.

After processing a file

For Tape backups, you can run a script after each file gets processed. The script gets called with 5 arguments - the "list label", the number of files that have been processed so far, the destination (the destination directory and the filename, as <directory>/<filename>), the file size in MB, and the tape block number. It does not matter what the script returns - failures are ignored.

SFTP

If copying using SFTP, you must have compiled midas with CURL support enabled.

Lazylogger does not support using a password for the copy - you must ensure that your public SSH key has been added to the destination host, to allow passwordless access.

Status

Messages

lazylogger sends a message for every major operation it does. The messages are sent to the lazy facility, and are the primary means of finding out what/when/where/how the lazylogger has operated on a data file.

Example excerpt of lazy.log

In this particular case, the lazylogger was using a script to copy files, and the channel name was Dcache.

Wed May  6 17:26:00 2015 [Lazy_Dcache] Dcache[30] (cp:1.0s) iris_000032140000.mid.gz 6.146MB  file NEW
Wed May  6 17:26:02 2015 [Lazy_Dcache] Dcache[31] (cp:0.1s) iris_000032150000.mid.gz 22.894MB  file NEW
Wed May  6 17:26:04 2015 [Lazy_Dcache] Dcache[32] (cp:0.1s) iris_000032160000.mid.gz 116.110MB  file NEW
Wed May  6 17:26:07 2015 [Lazy_Dcache] Dcache[33] (cp:0.1s) iris_000032170000.mid.gz 110.228MB  file NEW
Wed May  6 17:26:09 2015 [Lazy_Dcache] Dcache[34] (cp:0.0s) iris_000032180000.mid.gz 101.333MB  file NEW

ODB statistics

Summary statistics are stored in the ODB in /Lazy/xxx/Statistics. They include the file being copied, the copy progress and rate, and the number of files copied. The Status page shows Copy progress (%), Number of Files and Backup status (%) for each channel. See /Lazy/xxx/Statistics for all the keys.

Log files

Two hidden log files are created for each channel, and reside in the main experiment directory. They are called .Lazy_<channel_name>.dirlist and .Lazy_<chanel_name>.donelist. They reflect the current state of the data copy, and allow lazylogger to cleanly resume operation after being stopped/started.

Stopping lazylogger

Once the lazylogger has started a job on a data file, trying to terminate the application will result in a log message informing the user of the actual percentage of the backup completed so far. This message will repeat itself until completion of the backup, and only then the lazylogger application will terminate.

Related pages

lazylogger /Lazy keys