slitflow.info module

class Info(Data, info_path=None)

Bases: object

Data information class.

Parameters:

Data (Data) – Parent Data class.

column

Column information dictionaries. The dictionary should contain depth, name, type, unit and description.

Type:

list of dict

param

Parameter dictionaries. The dictionary should contain name, value, unit and description.

Type:

list of dict

meta

Metadata containing information about the data path, datetime, and required data.

Type:

dict

path

Absolute path to save this Info object in JSON format with an extension of .sf, including column, param, and meta dictionaries.

Type:

str

index

Index table to describe the data hierarchy.

Type:

pandas.DataFrame

file_nos

List of split file numbers.

Type:

list of int

load_split_depth

Split depth number for loading data.

Type:

int

data_split_depth

Split depth number to split the data property.

Type:

int

set_path(info_path)

Set path string to this object.

Parameters:

info_path (str) – Absolute path to the information file.

load(info_path=None)

Load information file.

Parameters:

info_path (str, optional) – Absolute path to the information file. Defaults to None.

load_index()

Load index table from the index file.

Info object should have the path attribute. See save_index() docstring for the index file format.

save_index(load_index=True)

Update index information file.

The file size is reduced by excluding duplicate higher-level hierarchical numbers as follows:

img_no frm_no
     1      1      1,1
     1      2  ->  ,2
     1      3      ,3

Caution

This method updates rather than overwrites existing index files. This process is necessary to save the split file, but if there is an unrelated index file with the same name, it must be deleted.

set_index_file_no()

Add file number column to the index table according to split depth.

set_file_nos(file_nos)

file_nos()

file_index()

Return index table of current file number.

Returns:

Index table of current split file

Return type:

pandas.DataFrame

save(info_path=None)

Save data information as a JSON file.

Parameters:

info_path (str, optional) – Path to the info file. Defaults to None.

split(split_depth=None)

Add a _split column to the index table.

Caution

This split depth is used to save newly generated data. Do not use set_split_depth() for this usage. If you use set_split_depth(), an error will occur when loading data that has already been saved.

Parameters:

split_depth (int, optional) – Split depth for the save data. Defaults to None.

get_dict()

Return a dictionary of all information for saving.

Returns:

Dictionary of column, param and meta dictionaries

Return type:

dict

add_column(depth, name, type, unit, description)

Add column information of data hierarchy.

Frequently used in slitflow.data.Data.set_info().

Parameters:

• depth (int) – Column depth. If None then set as the last depth + 1.

• name (str) – Column name used for pandas.DataFrame columns.

• type (str) – Value type.

• unit (str) – Unit of column value.

• description (str) – Explanation of column value.

insert_depth(insert_depth)

Add 1 to the depth after the specified depth.

Parameters:

insert_depth (int) – First index depth at which you want to shift depth.

copy_req_columns(req_no=0, names=None)

Import column info from required data info.

Frequently used in slitflow.data.Data.set_info().

Parameters:

• req_no (int, optional) – Index of required data list. Defaults to 0.

• names (list of str, optional) – Column names to copy from req. All columns are copied if None. Defaults to None.

delete_column(names=None, keeps=None)

Delete column information.

Frequently used in slitflow.data.Data.set_info().

Parameters:

• names (list of str, optional) – Column names to delete. All columns are deleted if None. Defaults to None.

• keeps (list of str, optional) – Column names not to delete.

get_column_name(type='all')

Get column names from the column property.

Parameters:

type (str) –

Column type to return

• all : all column names.

• index : column names whose depth > 0.

• col : column names that do not belong to the index.

Returns:

List of column names

Return type:

list of str

change_column_item(name, item, new_value)

Change a column item.

Used when you want to change a item in the column information. e.g. int32 to float32 in “type”.

Parameters:

• name (str) – Column name.

• item (str) – Column item to change. The item should be “depth”, “name”, “type”, “unit” or “description”.

• new_value (str) – New value.

get_column_type()

Get column type dict for DataFrame dtype.

Returns:

Dictionary of column types

Return type:

dict

get_column_dict(name)

Return dictionary of selected column information.

Parameters:

name (str) – Column name to select.

Returns:

“name”, “type”, “unit” and “description” of the column

Return type:

dict

get_column_depth(name=None)

Get column depth list.

Parameters:

name (str) – Column name to get depth number.

Returns:

List of depth number

Return type:

list

reset_depth(name, depth=None)

Change the depth of selected column.

Parameters:

• name (str) – Column name to change depth.

• depth (int, optional) – Target depth. If not specified, then the deepest depth + 1 is set. Defaults to None.

sort_index()

Rewrite depth so that there are no missing index numbers.

sort_column()

Sort column dictionaries according to depth.

add_param(name, value, unit, description)

Add parameter dictionary to the param property.

Frequently used in slitflow.data.Data.set_info().

Parameters:

• name (str) – Parameter name.

• value (any) – Parameter value.

• unit (str) – Unit of parameter value.

• description (str) – Explanation of parameter.

addel_param(new_params, name, name_temp, unit, description, init_val=None)

Add a parameter dictionary to param property if name is exist in temporal parameter dictionary else if init_val is not None. Else delete selected parameter from param property.

Parameters:

• new_params (dict) – Dictionary of new parameters.

• name (str) – Parameter name for save.

• name_temp (str) – Parameter name in new_params.

• unit (str) – Unit of parameter value.

• description (str) – Explanation of parameter.

• init_val (any) – Initial value of parameter.

copy_req_params(req_no=0, names=None)

Reuse parameters from the information of required data.

Frequently used in slitflow.data.Data.set_info().

Parameters:

• req_no (int, optional) – Index of required data list. Defaults to 0.

• names (list of str, optional) – Parameter names to copy from req. All parameters are copied if None. Defaults to None.

delete_param(name)

Delete selected parameter from params.

Parameters:

name (str) – Parameter name to delete.

delete_private_param()

Delete parameters containing _ prefix.

The parameter prefixed with “_” is used when the same large parameter is needed for all processes. This method is used in slitflow.data.Data.post_run() when you have registered with slitflow.data.Data.set_info() but do not want to save it in the info file.

add_user_param(param)

Add user-defined parameters from param[“user_param”].

Parameters:

param (list of list) – Parameter dictionary containing the “user_param” item. param[“user_param”] should be list of list contain [name, value, unit, description].

get_param_names()

Return parameter names from the param property.

Returns:

List of parameter names

Return type:

list of str

get_param_value(name)

Return parameter value from the param property.

Parameters:

name (str) – Parameter name to get the value.

Returns:

Value of selected parameter

Return type:

any

get_param_dict(name=None)

Return dictionary of selected parameter information.

Parameters:

name (str, optional) – Parameter name to select.

Returns:

“name”, “type”, “unit” and “description” of the parameter

Return type:

dict

set_split_depth(depth)

Set split depth into parameter dictionary.

Caution

This method is used in slitflow.data.Data.set_info() to set how to split the result data. If you want split load data, please use split().

Parameters:

depth (int) – File split depth number.

split_depth()

Return split depth of the result data.

set_group_depth(depth)

Add index_cols into param.

This method adds the “index_cols” parameter to split data using pandas.DataFrame.groupby().

Parameters:

depth (int) – Data grouping depth number.

group_depth()

Return group depth number.

Returns:

Group depth number

Return type:

int

group_cols()

Return index column names for groupby of DataFrame.

Returns:

List of index column names

Return type:

list of str

copy_req(req_no=0, type='all', names=None)

Import column and parameter info from required data.

Parameters:

• req_no (int, optional) – List number of required data.

• type (str, optional) – “all”, “column”, “index” or “param”.

• names (list of str, optional) – List of copy names.

set_meta()

Set meta data.

Analysis records, including path, timestamp, and required data information, are created in meta property to save them into the info file.

to_json()

Returns a string representation of info file to export.

Caution

Private parameters will be removed.

Returns:

a string representation of info file

Return type:

str

get_depth_id()

Return depth id used in data file names.

This id is used in split file names.

Returns:

List of depth id string. The id format is “D[depth 1 value]D[depth 2 value]…”.

Return type:

list of str

rename_class_name(new_name)

Rename class name in meta data and save info.

Parameters:

new_name (str) – New class name.

fullname(o)

Returns full name of object.

Parameters:

o (object) – Object.

Returns:

Full name of object

Return type:

str