TotalDepth.common.process (Monitoring a Process’s CPU and Memory)

A HOWTO is here Process Monitoring with TotalDepth.common.process

Logs process information, such as memory usage, to a log as JSON. Example with (‘memory_info’, ‘cpu_times’):

(Thread-7  ) ProcessLoggingThread JSON: {"memory_info": {"rss": 145448960, "vms": 4542902272, "pfaults": 37618, "pageins": 0}, "cpu_times": {"user": 0.28422032, "system": 0.099182912, "children_user": 0.0, "children_system": 0.0}}

There are several DoF here:

  • Logging interval in seconds. Or by poke()?
  • Logging level, DEBUG, INFO etc.
  • Logging verbosity, for example just memory? Or everything about the process (self._process.as_dict())

Also need to add a log parser to, well what?

TotalDepth.common.process.LOGGER_PREFIX = 'ProcessLoggingThread-JSON'

Unique string in the log line

TotalDepth.common.process.RE_LOG_LINE = re.compile('^.+?ProcessLoggingThread-JSON(-START|-STOP)?\\s*(.+)$')

Regex for the unique string in the log line

TotalDepth.common.process.DATETIME_NOW_FORMAT = '%Y-%m-%d %H:%M:%S.%f'

Regex for timestam, matches ‘2019-06-07 11:57:58.390921’

TotalDepth.common.process.KEY_TIMESTAMP = 'timestamp'

The JSON key that is the timestamp

TotalDepth.common.process.KEY_ELAPSED_TIME = 'elapsed_time'

The JSON key that is elapsed (wall clock) time in seconds. This is time.time() - self._process.create_time()

TotalDepth.common.process.KEY_LABEL = 'label'

The JSON key that is the label

TotalDepth.common.process.KEY_PROCESS_ID = 'pid'

The JSON key that is the process ID

TotalDepth.common.process.PSUTIL_PROCESS_AS_DICT_KEYS = ['cmdline', 'connections', 'cpu_percent', 'cpu_times', 'create_time', 'cwd', 'environ', 'exe', 'gids', 'memory_full_info', 'memory_info', 'memory_percent', 'name', 'nice', 'num_ctx_switches', 'num_fds', 'num_threads', 'open_files', 'pid', 'ppid', 'status', 'terminal', 'threads', 'uids', 'username']

psutil.Process().as_dict() has the following keys:

TotalDepth.common.process.GNUPLOT_PLT = '\nset grid\nset title "Memory and CPU Usage." font ",14"\nset xlabel "Elapsed Time (s)"\n# set mxtics 5\n# set xrange [0:3000]\n# set xtics\n# set format x ""\n\n#set logscale y\nset ylabel "Memory Usage (Mb)"\n# set yrange [0:500]\n# set ytics 20\n# set mytics 2\n# set ytics 8,35,3\n\n#set logscale y2\nset y2label "CPU Usage (%), Page Faults (10,000/s)"\n# set y2range [0:200]\nset y2tics\n\nset pointsize 1\nset datafile separator whitespace#"\t"\nset datafile missing "NaN"\n\nset terminal svg size 1000,700 # choose the file format\nset output "{name}.svg" # choose the output device\n\n# set key off\n\n{labels}\n\n#set key title "Window Length"\n# lw 2 pointsize 2\n\nplot "{name}.dat" using 1:($2 / 1024**2) axes x1y1 title "RSS (Mb), left axis" with lines lt 1 lw 2, \\\n "{name}.dat" using 1:($3 / 10000) axes x1y2 title "Page Faults (10,000/s), right axis" with lines lt 3 lw 1, \\\n "{name}.dat" using 1:5 axes x1y2 title "Mean CPU (%), right axis" with lines lt 2 lw 1, \\\n "{name}.dat" using 1:6 axes x1y2 title "Instantaneous CPU (%), right axis" with lines lt 7 lw 1\n\nreset\n'

Usage: GNUPLOT_PLT.format(name=dat_file_name)

TotalDepth.common.process.parse_timestamp(s: str) → datetime.datetime

Read a string such as ‘2019-06-07 11:57:58.390921’ and return a datetime.

TotalDepth.common.process.extract_json(istream: TextIO) → List[Dict[str, Any]]

Reads a log file and returns the JSON as a list of dicts. Non-matching lines are ignored.

TotalDepth.common.process.extract_labels_from_json(json_data: List[Dict[str, Any]]) → List[Dict[str, Any]]

Returns a list of dicts of JSON data where ‘label’ is a key’.

TotalDepth.common.process.extract_json_as_table(json_data: List[Dict[str, Any]]) → Tuple[Dict[int, List[List[str]]], Dict[int, float], Dict[int, float], Dict[int, float], Dict[int, float]]

Create a table from JSON suitable for a Gnuplot .dat file.

Returns:

  • { process_id : [rows of data, …], …}
  • { process_id : t min, …}
  • { process_id : t max, …}
  • { process_id : RSS min, …}
  • { process_id : RSS max, …}

A row of data is:

time, RSS, PageFaults, User, Mean CPU, Insantanous CPU, Timestamp, PID, Label
TotalDepth.common.process.invoke_gnuplot(log_path: str, gnuplot_dir: str) → int

Reads a log file, extracts the data, writes it out to gnuplot_dir and invokes gnuplot on it.

TotalDepth.common.process.add_message_to_queue(msg: str) → None

Adds a message onto the queue.

class TotalDepth.common.process.ProcessLoggingThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None)

Thread that regularly logs out process parameters.

__init__(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None)

Constructor. args[0], or interval=… must be the reporting interval in seconds, default 1.0. args[1], or log_level=… must be the log level to report with, default logging.INFO.

run() → None

thread.run(). Write to log then sleep.

join(*args, **kwargs)

thread.join(). Write to log last time.

TotalDepth.common.process.log_process(*args, **kwargs)

Context manager to log process data at regular intervals.

TotalDepth.common.process.add_process_logger_to_argument_parser(parser: argparse.ArgumentParser) → None

Add a --log-process option to the argument parser.

TotalDepth.common.process.main() → int

Main CLI entry point. For testing.