TotalDepth.common.process (Monitoring a Process’s CPU and Memory)¶
A HOWTO is here Process Monitoring with TotalDepth.common.process
Logs process information, such as memory usage, to a log as JSON. Example with (‘memory_info’, ‘cpu_times’):
(Thread-7 ) ProcessLoggingThread JSON: {"memory_info": {"rss": 145448960, "vms": 4542902272, "pfaults": 37618, "pageins": 0}, "cpu_times": {"user": 0.28422032, "system": 0.099182912, "children_user": 0.0, "children_system": 0.0}}
There are several DoF here:
- Logging interval in seconds. Or by poke()?
- Logging level, DEBUG, INFO etc.
- Logging verbosity, for example just memory? Or everything about the process (self._process.as_dict())
Also need to add a log parser to, well what?
-
TotalDepth.common.process.
LOGGER_PREFIX
= 'ProcessLoggingThread-JSON'¶ Unique string in the log line
-
TotalDepth.common.process.
RE_LOG_LINE
= re.compile('^.+?ProcessLoggingThread-JSON(-START|-STOP)?\\s*(.+)$')¶ Regex for the unique string in the log line
-
TotalDepth.common.process.
DATETIME_NOW_FORMAT
= '%Y-%m-%d %H:%M:%S.%f'¶ Regex for timestam, matches ‘2019-06-07 11:57:58.390921’
-
TotalDepth.common.process.
KEY_TIMESTAMP
= 'timestamp'¶ The JSON key that is the timestamp
-
TotalDepth.common.process.
KEY_ELAPSED_TIME
= 'elapsed_time'¶ The JSON key that is elapsed (wall clock) time in seconds. This is
time.time() - self._process.create_time()
-
TotalDepth.common.process.
KEY_LABEL
= 'label'¶ The JSON key that is the label
-
TotalDepth.common.process.
KEY_PROCESS_ID
= 'pid'¶ The JSON key that is the process ID
-
TotalDepth.common.process.
PSUTIL_PROCESS_AS_DICT_KEYS
= ['cmdline', 'connections', 'cpu_percent', 'cpu_times', 'create_time', 'cwd', 'environ', 'exe', 'gids', 'memory_full_info', 'memory_info', 'memory_percent', 'name', 'nice', 'num_ctx_switches', 'num_fds', 'num_threads', 'open_files', 'pid', 'ppid', 'status', 'terminal', 'threads', 'uids', 'username']¶ psutil.Process().as_dict() has the following keys:
-
TotalDepth.common.process.
GNUPLOT_PLT
= '\nset grid\nset title "Memory and CPU Usage." font ",14"\nset xlabel "Elapsed Time (s)"\n# set mxtics 5\n# set xrange [0:3000]\n# set xtics\n# set format x ""\n\n#set logscale y\nset ylabel "Memory Usage (Mb)"\n# set yrange [0:500]\n# set ytics 20\n# set mytics 2\n# set ytics 8,35,3\n\n#set logscale y2\nset y2label "CPU Usage (%), Page Faults (10,000/s)"\n# set y2range [0:200]\nset y2tics\n\nset pointsize 1\nset datafile separator whitespace#"\t"\nset datafile missing "NaN"\n\nset terminal svg size 1000,700 # choose the file format\nset output "{name}.svg" # choose the output device\n\n# set key off\n\n{labels}\n\n#set key title "Window Length"\n# lw 2 pointsize 2\n\nplot "{name}.dat" using 1:($2 / 1024**2) axes x1y1 title "RSS (Mb), left axis" with lines lt 1 lw 2, \\\n "{name}.dat" using 1:($3 / 10000) axes x1y2 title "Page Faults (10,000/s), right axis" with lines lt 3 lw 1, \\\n "{name}.dat" using 1:5 axes x1y2 title "Mean CPU (%), right axis" with lines lt 2 lw 1, \\\n "{name}.dat" using 1:6 axes x1y2 title "Instantaneous CPU (%), right axis" with lines lt 7 lw 1\n\nreset\n'¶ Usage: GNUPLOT_PLT.format(name=dat_file_name)
-
TotalDepth.common.process.
parse_timestamp
(s: str) → datetime.datetime¶ Read a string such as ‘2019-06-07 11:57:58.390921’ and return a datetime.
-
TotalDepth.common.process.
extract_json
(istream: TextIO) → List[Dict[str, Any]]¶ Reads a log file and returns the JSON as a list of dicts. Non-matching lines are ignored.
-
TotalDepth.common.process.
extract_labels_from_json
(json_data: List[Dict[str, Any]]) → List[Dict[str, Any]]¶ Returns a list of dicts of JSON data where ‘label’ is a key’.
-
TotalDepth.common.process.
extract_json_as_table
(json_data: List[Dict[str, Any]]) → Tuple[Dict[int, List[List[str]]], Dict[int, float], Dict[int, float], Dict[int, float], Dict[int, float]]¶ Create a table from JSON suitable for a Gnuplot
.dat
file.Returns:
- { process_id : [rows of data, …], …}
- { process_id : t min, …}
- { process_id : t max, …}
- { process_id : RSS min, …}
- { process_id : RSS max, …}
A row of data is:
time, RSS, PageFaults, User, Mean CPU, Insantanous CPU, Timestamp, PID, Label
-
TotalDepth.common.process.
invoke_gnuplot
(log_path: str, gnuplot_dir: str) → int¶ Reads a log file, extracts the data, writes it out to gnuplot_dir and invokes gnuplot on it.
-
TotalDepth.common.process.
add_message_to_queue
(msg: str) → None¶ Adds a message onto the queue.
-
class
TotalDepth.common.process.
ProcessLoggingThread
(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None)¶ Thread that regularly logs out process parameters.
-
__init__
(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None)¶ Constructor. args[0], or interval=… must be the reporting interval in seconds, default 1.0. args[1], or log_level=… must be the log level to report with, default logging.INFO.
-
run
() → None¶ thread.run(). Write to log then sleep.
-
join
(*args, **kwargs)¶ thread.join(). Write to log last time.
-
-
TotalDepth.common.process.
log_process
(*args, **kwargs)¶ Context manager to log process data at regular intervals.
-
TotalDepth.common.process.
add_process_logger_to_argument_parser
(parser: argparse.ArgumentParser) → None¶ Add a
--log-process
option to the argument parser.
-
TotalDepth.common.process.
main
() → int¶ Main CLI entry point. For testing.