Low Level API

Beware, this package is under heavy development and the low level API is subject to change without warning!

Dimemas Automation

Helper routines to run Dimemas from Python, including the special case of ideal trace generation.

pypop.dimemas.dimemas_analyse(tracefile, configfile, outpath=None, substrings=None)

Run a Dimemas simulation given a configuration and tracefile

The configuration file may be modifed with key->value substitutions prior to running dimemas using the substrings parameter. This allows the use of a predefined skeleton configuration file rather than requiring programmatic production of the complete config.

Parameters:

• tracefile (str) – Path to Extrae tracefile (*.prv)
• configfile (str) – Path to Dimemas configfile
• outpath (str or None) – Optional path to output idealised trace. (If not specified, will be created in a temporary folder.)
• substrings (dict) – Dict of keys in the config file to be replaced with corresponding values.

Returns:

simulated – Path to simulated tracefile in prv format.

Return type:

str

pypop.dimemas.dimemas_idealise(tracefile, outpath=None)

Idealise a tracefile using Dimemas

Parameters:

• tracefile (str) – Path to Extrae tracefile (*.prv)
• outpath (str) – Optional path to output idealised trace. (If not specified will be created in a temporary folder.)

Returns:

idealised – Path to idealised tracefile in prv format.

Return type:

str

Extrae Trace Utilities

Routines for analysis and management of Extrae traces, including Paramedir automation.

pypop.extrae.chop_prv_to_roi(prv_file, outfile=None)

Cut down a prv trace to just the region of interest

This will cut a trace to just include the region of interest, which is assumed to be bracketed by calls to Extrae_Restart() and Extrae_Shutdown() which are located by their corresponding events (40000012) in the trace.

Parameters:

• prv_file (str) – Trace file to be cut
• outfile (str or None) – Optional output file for chopped trace. (If not specified will be created in a temporary folder.)

Returns:

chopped – Path to chopped tracefile in prv format.

Return type:

str

pypop.extrae.load_paraver_histdata(hist_file)

Read a Paraver histogram file and return pandas dataframe containing the data.

Parameters:hist_file (str) – Path to the histogram data file

pypop.extrae.paramedir_analyze(tracefile, config, variables=None, index_by_thread=False, statistic_names=None)

Analyze a tracefile with paramedir

Parameters:

• tracefile (str) – Path to *.prv tracefile from Extrae
• config (str) – Path to Paraver/Paramedir *.cfg
• variables (dict or None) – Optional dict of key-value pairs for replacement in config file prior to running paramedir.
• index_by_thread (bool) – If True return data organised by a multilevel index of MPI ranks and threads. Note that this discards Paramedir calculated statistical info.
• statistic_names (list of str or None) – Optional list of string names for the statistics returned by the config file. If not provided names will be taken from paramedir output.

Returns:

result – Result data loaded from the resulting csv.

Return type:

pandas.DataFrame

pypop.extrae.reindex_by_thread(stats_dframe, thread_prefix='THREAD')

Convert stats Dataframe index in-place to a rank,thread MultiIndex

Parameters:

• stats_dframe (pd.DataFrame) – Dataframe to reindex. Typically this will have been produced using paramedir_analyze().
• thread_prefix (str) – Prefix before thread number pattern in current index. Should almost always be “THREAD”. Paraver/Paramedir default is “THREAD a.r.t” with r the rank number and t the thread number.

pypop.extrae.remove_trace(tracefile, failure_is_error=False)

Remove prv or dim file and its associated .row and .pcf files

The function should be passed a filename ending in .dim or .prv and will attempt to delete this and the pcf and row file. The filenames of these will be constructed by direct substitution of the file extension.

Parameters:tracefile (str) – Path to tracefile, should end with .dim or prv

pypop.extrae.run_paramedir(tracefile, config, outfile=None, variables=None)

Run paramedir on a tracefile

Parameters:

• tracefile (str) – Path to *.prv tracefile from Extrae
• config (str) – Path to Paraver/Paramedir *.cfg
• outfile (str or None) – Path to output file. If None or “” a randomly named temporary file will be used.
• variables (dict of str:str) – Dict of variables to replace in the config file. For a key-value pair “key”:val any occurrence of @key@ in the file will be replaced with “val”

Returns:

outfile – Path to the output file.

Return type:

str

pypop.extrae.sort_traces_by_commsize(tracelist)

Take a list of traces and return them sorted in ascending order of their MPI commsize

Parameters:tracelist (iterable of str) – List of traces in arbitrary order Returns:

• commsizes (list of int) – Sorted list of comm sizes
• traces (list of str) – Sorted list of tracefiles

PRV Trace Loader

Support for directly loading PRV trace data as Pandas dataframes

class pypop.prv.ApplicationLayout(commsize, rank_threads)

Create new instance of ApplicationLayout(commsize, rank_threads)

commsize

Alias for field number 0

count(value) → integer -- return number of occurrences of value

index(value[, start[, stop]]) → integer -- return first index of value.

Raises ValueError if the value is not present.

rank_threads

Alias for field number 1

class pypop.prv.PRV(prv_path)

colnames = {'comm': None, 'event': ['task', 'thread', 'cpu', 'time', 'event', 'value'], 'state': ['task', 'thread', 'cpu', 'time', 'endtime', 'state']}

coltypes = {'comm': None, 'event': {'task': <class 'numpy.int32'>, 'thread': <class 'numpy.int32'>, 'cpu': <class 'numpy.int32'>, 'begin': <class 'numpy.int64'>, 'event': <class 'numpy.int32'>, 'value': <class 'numpy.int64'>}, 'state': {'task': <class 'numpy.int32'>, 'thread': <class 'numpy.int32'>, 'cpu': <class 'numpy.int32'>, 'begin': <class 'numpy.int64'>, 'end': <class 'numpy.int64'>, 'state': <class 'numpy.int32'>}}

profile_openmp_regions(no_progress=False)

record_types = {1: 'state', 2: 'event', 3: 'comm'}

save(filename)

class pypop.prv.Trace(info, data)

Create new instance of Trace(info, data)

count(value) → integer -- return number of occurrences of value

data

Alias for field number 1

index(value[, start[, stop]]) → integer -- return first index of value.

Raises ValueError if the value is not present.

info

Alias for field number 0

class pypop.prv.TraceMetadata(captured, ns_elapsed, nodes, procs_per_node, num_appl, application_layout)

Create new instance of TraceMetadata(captured, ns_elapsed, nodes, procs_per_node, num_appl, application_layout)

application_layout

Alias for field number 5

captured

Alias for field number 0

count(value) → integer -- return number of occurrences of value

index(value[, start[, stop]]) → integer -- return first index of value.

Raises ValueError if the value is not present.

nodes

Alias for field number 2

ns_elapsed

Alias for field number 1

num_appl

Alias for field number 4

procs_per_node

Alias for field number 3

pypop.prv.get_prv_header_info(prv_file)

Get basic run information from the prv file header

Parameters:

• prv_file (str) – Path to prv trace file.
• Returns –
• -------- –
• metadata (NamedTuple) – Named tuple containing the header information.

pypop.prv.zipopen(path, modespec)

Miscellaneous utility routines

General utility and helper routines not specific to a particular tool.

pypop.utils.chunked_md5sum(filename, blocksize=8388608)

Chunked md5 checksum using hashlib

Parameters:

• filename (str) – File to calculate checksum.
• blocksize (int) – Blocksize in bytes to iterate with, (default 8MB).

Returns:

hexdigest – String containing hexadecimal representation of the checksum.

Return type:

str

pypop.utils.return_first_arg(arg, *args, **kwargs)

Return first argument and disregard remaining, intended for silently replacing tqdm when unavailable

Parameters:

• arg – Argument to be returned
• *args (Will be ignored) –
• **kwargs (Will be ignored) –

Returns:

First argument passed to function

Return type:

arg