hpct-importer(1)

NAME
hpct-importer - HiPerConTracer Trace Data Importer

SYNOPSIS
hpct-importer [-C importer_configuration | - -importer-config importer_configuration] [-D database_configuration | - -database-config database_configuration] [-L level | - -loglevel level] [-Z on|off | - -logcolor on|off] [-O file | - -logfile file] [-q | - -quiet] [-v | - -verbose] [-X mode | - -import-mode mode] [-Y depth | - -import-max-depth depth] [-I path | - -import-file-path path] [-B path | - -bad-file-path path] [-G path | - -good-file-path path] [-F filter_regexp | - -import-file-path-filter filter_regexp] [-Q | - -quit-when-idle] [- -ping-workers number] [- -ping-files number] [- -traceroute-workers number] [- -traceroute-files number]
hpct-importer [-h | - -help]
hpct-importer [- -version]

DESCRIPTION
hpct-importer imports HiPerConTracer results into an SQL or NoSQL database. Currently, the following database backends are supported: MySQL/MariaDB (SQL), PostgreSQL (SQL), MongoDB (NoSQL).
Results files imported successfully can be moved into a "good" directory, deleted or kept as-in (for debugging only!). Results files which failed to import a re moved into a "bad" directory.
The input of multiple input files can be combined into a single database transaction, to improve performance on short files. In case of an import error, each file is imported sequentially to identify faulty ones.

ARGUMENTS
The following argument may be provided:

-L level | - -loglevel level
    Sets the minimum logging level to the provided value: 0=trace, 1=debug, 2=info, 3=warning, 4=error, 5=fatal.

-Z on|off | - -logcolor on|off
    Enables (on) or disables (off) the usage of ANSI color escape sequences to colorise the log output according to its level.

-O file | - -logfile file
    Sets a file name for appending the log output to. If a file is set, no log output is made to the console. Otherwise, all log output will be written to the console.

-q | - -quiet
    Sets the minimum logging level to 3 (warning).

-v | - -verbose
    Sets the minimum logging level to 0 (trace).

-C importer_configuration | - -importer-config importer_configuration
    Sets the name of the importer configuration file. This option must be provided!
    See /usr/share/hipercontracer/hipercontracer-importer.conf for an example.

-D database_configuration | - -database-config database_configuration
    Sets the name of the database configuration file. This option must be provided!
    See /usr/share/hipercontracer/hipercontracer-database.conf for an example.

-X mode | - -import-mode mode
    Specifies the import mode, overriding the mode read from the database configuration file: DeleteImportedFiles (delete successfully imported files) MoveImportedFiles (move successfully imported files into "good" directory), KeepImportedFiles (keep successfully imported files where they are). Do not use KeepImportedFiles execpt for testing. Otherwise, the next re-run of hpct-importer tries to import them again! They may create duplicates when the underlying database system (e.g. NoSQL without unique indices) does not ensure uniqueness!

-Y depth | - -import-max-depth depth
    Specifies the import directory traversal depth, overriding the mode read from the database configuration file.

-I path | - -import-file-path path
    Specifies the import file path, overriding the mode read from the database configuration file. This path is used to look for files to import.

-B path | - -bad-file-path path
    Specifies the "good" file path, overriding the mode read from the database configuration file. This path is used to move successfully imported files to, when the import mode is set to KeepImportedFiles.

-G path | - -good-file-path path
    Specifies the "bad" file path, overriding the mode read from the database configuration file. This path is used to move files to which failed to import.

-F filter_regexp | - -import-file-path-filter filter_regexp
    Specifies the import path filter, overriding the mode read from the database configuration file. This filter can be used to only import a subset of the import files for testing. The filter expression is an std::regex regular expression.

-Q | - -quit-when-idle
    Quit importer when all workers are idle. This will perform only one import round.

- -ping-workers number
    Sets the number of Ping import workers. Default: 1. An import file is distributed to a worker by hashing the source address given in the file name.

- -ping-files number
    Sets the number of Ping import files per database transaction. Default: 1.

- -traceroute-workers number
    Sets the number of Traceroute import workers. Default: 1. An import file is distributed to a worker by hashing the source address given in the file name.

- -traceroute-files number
    Sets the number of Traceroute import files per database transaction. Default: 1.

-h | - -help
    Prints command help.

- -version
    Prints program version.

EXAMPLES

hpct-importer -C hipercontracer-importer.conf -D hipercontracer-database.conf

hpct-importer --importer-config hipercontracer-importer.conf --database-config hipercontracer-database.conf --verbose

hpct-importer -C hipercontracer-importer.conf -D hipercontracer-database.conf -F 1234/.*

hpct-importer --version

hpct-importer --h

AUTHORS
Thomas Dreibholz
https://www.nntb.no/~dreibh/hipercontracer
mailto://dreibh@simula.no