yugabyte_pycommon package¶

Submodules¶

yugabyte_pycommon.collection_util module¶

yugabyte_pycommon.collection_util.group_by_to_dict(arr, key_fn)¶

Given a list-like collection and a function that computes a key, returns a map from keys to all values with that key.

>>> group_by_to_dict([100, 201, 300, 301, 400], lambda x: x % 2)
{0: [100, 300, 400], 1: [201, 301]}
>>> group_by_to_dict([100, 201, 300, 301, 400, 401, 402], lambda x: x % 3)
{0: [201, 300, 402], 1: [100, 301, 400], 2: [401]}

yugabyte_pycommon.collection_util.group_by_to_list(arr, key_fn)¶

Group the given list-like collection by the key computed using the given function. The collection does not have to be sorted in advance.

Returns:	a list of (key, list_of_values) tuples where keys are sorted

>>> group_by_to_list([100, 201, 300, 301, 400], lambda x: x % 2)
[(0, [100, 300, 400]), (1, [201, 301])]
>>> group_by_to_list([100, 201, 300, 301, 400, 401, 402], lambda x: x % 3)
[(0, [201, 300, 402]), (1, [100, 301, 400]), (2, [401])]

yugabyte_pycommon.collection_util.make_list(obj)¶

Convert the given object to a list. Strings get converted to a list of one string, not to a list of their characters. Sets are sorted.

>>> make_list('asdf')
['asdf']
>>> make_list(['a', 'b', 'c'])
['a', 'b', 'c']
>>> make_list(set(['z', 'a', 'b']))
['a', 'b', 'z']
>>> make_list(set(['z', 'a', 10, 20]))
[10, 20, 'a', 'z']
>>> make_list(set([10, 20, None, 'a', 'z']))
[10, 20, None, 'a', 'z']

yugabyte_pycommon.collection_util.make_set(obj)¶

Makes a set of a given object. Returns the same object if it is already a set. Otherwise follows the same logic as make_list().

Parameters:	obj – a collection object
Returns:	a set created from the given object

yugabyte_pycommon.external_calls module¶

Utilities for running external commands.

exception yugabyte_pycommon.external_calls.ExternalProgramError(message, result)¶: Bases: exceptions.Exception

class yugabyte_pycommon.external_calls.ProgramResult(cmd_line, cmd_line_str, returncode, stdout, stdout_path, stderr, stderr_path, program_path, invocation_details_str, max_lines_to_show, output_captured)¶

This represents the result of executing an external program.

failure()¶

Returns:	whether the external program exited with a failure

get_stderr()¶

get_stdout()¶

get_stdout_and_stderr_together()¶

Returns:	a string with user-friendly versions of stdout and stderr of the external program, concatenated together.

get_user_friendly_stderr_msg()¶

Returns:	a user-friendly version of the external program’s standard error

get_user_friendly_stdout_msg()¶

Returns:	a user-friendly version of the external program’s standard output

print_output_and_raise_error_if_failed()¶

print_output_to_stdout()¶: Print both stdout and stderr of the external program to the stdout.

raise_error_if_failed()¶: This is useful for delayed handling of external program errors. Raises an error if the external program failed. Otherwise does nothing.

success()¶

Returns:	whether the external program exited with a success

class yugabyte_pycommon.external_calls.WorkDirContext(work_dir)¶

Allows setting a working directory context for running external programs. The directory will be changed to the given directory on entering the block, and will be restored to the old directory on exit.

with WorkDirContext('/tmp'):
    run_program('ls')

yugabyte_pycommon.external_calls.check_run_program(*args, **kwargs)¶: Similar to subprocess.check_call but using our run_program facility.

yugabyte_pycommon.external_calls.program_fails_no_log(args, **kwargs)¶

Run the given program, and returns if it failed. Does not log anything in case of success or failure.

Parameters:	args – command line arguments or a single string to run as a shell command kwargs – additional keyword arguments for subprocess.Popen
Returns:	`True` if the program succeeded

yugabyte_pycommon.external_calls.program_succeeds_empty_output(args, **kwargs)¶

Runs a program that is not expected to produce any output.

Parameters:	args – command line arguments or a single string to run as a shell command kwargs – additional keyword arguments for subprocess.Popen
Raises:	ExternalProgramError – if the program succeeds but produces extra output
Returns:	`True` if the program succeeds and does not produce any output

yugabyte_pycommon.external_calls.program_succeeds_no_log(args, **kwargs)¶

Run the given program, and returns True if it succeeded. Does not log anything in case of success or failure.

Parameters:	args – command line arguments or a single string to run as a shell command kwargs – additional keyword arguments for subprocess.Popen
Returns:	`True` if the program failed

yugabyte_pycommon.external_calls.run_program(args, error_ok=False, report_errors=None, capture_output=True, max_lines_to_show=1000, cwd=None, shell=None, stdout_path=None, stderr_path=None, stdout_stderr_prefix=None, **kwargs)¶

Run the given program identified by its argument list, and return a ProgramResult object.

Parameters:

args – This could be a single string, or a tuple/list of elements where each element is either a string or an integer. If a single string is given as args, and the shell parameter is not specified, it is automatically set to true.
report_errors – whether errors during execution (as identified by exit code) should be reported in the log.
capture_output – whether standard output and standard error of the program need to be captured in variables inside of the resulting ProgramResult object.
error_ok – if this is true, we won’t raise an exception in case the external program fails.
stdout_path – instead of trying to capture all standard output in memory, save it to this file. Both stdout_file_path and stderr_file_path have to be specified or unspecified at the same time. Also shell has to be true in this mode as we are using shell redirections to implement this.
stderr_path – similar to stdout_file_path but for standard error.
stdout_stderr_prefix – allows setting both stdout_path and stderr_path quickly. Those variables are set to the value of this parameter with .out and .err appended.

yugabyte_pycommon.fs_util module¶

Utilities for manipulating files and the file system.

yugabyte_pycommon.fs_util.get_tmp_file_path(*args, **kwargs)¶: Generates a temporary file name. Arguments are exactly like those of tempfile.NamedTemporaryFile. The file is immediately closed and scheduled to be deleted at exit, unless the delete_at_exit parameter value says otherwise.

yugabyte_pycommon.fs_util.mkdir_p(d)¶: Similar to the “mkdir -p …” shell command. Creates the given directory and all enclosing directories. No-op if the directory already exists.

yugabyte_pycommon.fs_util.read_file(file_path)¶: Reads the contents of the given file. :param file_path: the file path to read :return: the contents of the file

yugabyte_pycommon.logging_util module¶

yugabyte_pycommon.logging_util.get_default_log_level()¶

yugabyte_pycommon.logging_util.init_logging(log_level=None)¶

yugabyte_pycommon.logging_util.is_verbose_mode()¶

yugabyte_pycommon.text_manipulation module¶

yugabyte_pycommon.text_manipulation.cmd_line_args_to_str(args)¶

Converts a list of command-line arguments, including an executable program in the beginning, to a single command-line string suitable for pasing into Bash.

Parameters:	args – an array with a program path and command-line arguments
Returns:	a string suitable for pasing into Bash

yugabyte_pycommon.text_manipulation.decode_utf8(bytes)¶

yugabyte_pycommon.text_manipulation.get_bool_env_var(env_var_name)¶

>>> os.environ['YB_TEST_VAR'] = '  1 '
>>> get_bool_env_var('YB_TEST_VAR')
True
>>> os.environ['YB_TEST_VAR'] = '  0 '
>>> get_bool_env_var('YB_TEST_VAR')
False
>>> os.environ['YB_TEST_VAR'] = '  TrUe'
>>> get_bool_env_var('YB_TEST_VAR')
True
>>> os.environ['YB_TEST_VAR'] = 'fAlSe '
>>> get_bool_env_var('YB_TEST_VAR')
False
>>> os.environ['YB_TEST_VAR'] = '  YeS '
>>> get_bool_env_var('YB_TEST_VAR')
True
>>> os.environ['YB_TEST_VAR'] = 'No'
>>> get_bool_env_var('YB_TEST_VAR')
False
>>> os.environ['YB_TEST_VAR'] = ''
>>> get_bool_env_var('YB_TEST_VAR')
False

yugabyte_pycommon.text_manipulation.quote_for_bash(s)¶

yugabyte_pycommon.text_manipulation.safe_path_join(*args)¶

Like os.path.join, but allows arguments to be None. If all arguments are None, returns None. A special case: if the first argument is None, always return None. That allows to set a number of constants as relative paths under a certain path which may itself be None.

>>> safe_path_join()
>>> safe_path_join(None)
>>> safe_path_join(None, None)
>>> safe_path_join('/a', None, 'b')
'/a/b'
>>> safe_path_join(None, '/a', None, 'b')  # special case: first arg is None

yugabyte_pycommon.text_manipulation.trim_long_text(text, max_lines)¶: Trim a potentially long multi-line message at the given number of lines. :param text: the input text :param max_lines: maximum number of lines :return: the trimmed message