namespace for all model-related io functions. More...

Classes
struct	SaveOption

struct	WeightFileEntry
	Collect the data about a weight file. More...

class	PartialModelIO
	This class is used as an implementation detail to capture the common code of PartialModelSaver and PartialModelLoader. More...

class	PartialModelSaver
	Manage saving a model consisting of multiple partial models. More...

class	PartialModelLoader
	This class allows loading only a subset of the weights of a large model. More...

Enumerations
enum class	WeightFormat { DENSE_TXT = 0 , SPARSE_TXT = 1 , DENSE_NPY = 2 , NULL_FORMAT = 3 }
	Describes the format in which the weight data has been saved. More...

Functions
WeightFormat	parse_weights_format (std::string_view name)
	Gets the eighs. More...

const char *	to_string (WeightFormat format)

void	save_model (const path &target_file, const std::shared_ptr< const Model > &model, SaveOption options)
	Saves a complete model to a file. More...

std::shared_ptr< Model >	load_model (path source)

void	save_dense_weights_txt (std::ostream &target, const Model &model)
	Saves the dense weights in a plain-text format. More...

void	load_dense_weights_txt (std::istream &source, Model &target)
	Loads weights saved by io::model::save_dense_weights_txt. More...

void	save_dense_weights_npy (std::streambuf &target, const Model &model)
	Saves the dense weights in a npy file. More...

void	load_dense_weights_npy (std::streambuf &target, Model &model)
	Loads dense weights from a npy file. More...

void	save_as_sparse_weights_txt (std::ostream &target, const Model &model, double threshold)
	Saves the weights in sparse plain-text format, culling small weights. More...

void	load_sparse_weights_txt (std::istream &source, Model &target)
	Loads sparse weights from plain-text format. More...

Detailed Description

namespace for all model-related io functions.

Enumeration Type Documentation

◆ WeightFormat

enum dismec::io::model::WeightFormat

strong

Describes the format in which the weight data has been saved.

Enumerator
DENSE_TXT	Dense Text Format
SPARSE_TXT	Sparse Text Format
DENSE_NPY	Dense Numpy Format
NULL_FORMAT	This format exists for testing purposes only, and indicates that the weights will not be saved.

Definition at line 99 of file model-io.h.

Function Documentation

◆ load_dense_weights_npy()

void dismec::io::model::load_dense_weights_npy	(	std::streambuf &	target,
		Model &	model
	)

Loads dense weights from a npy file.

The npy file needs to contain a row-major array, with the number of rows corresponding to the number of labels. The data type needs to be an exact match to the datatype used in the model, i.e. real_t.

Parameters

target	Input stream buffer from which the data is read.
target	Model whose weights to fill in. It is assumed that `target` is already of the correct size.

Definition at line 82 of file weights.cpp.

References dismec::io::binary_load(), dismec::model::Model::contained_labels(), anonymous_namespace{weights.cpp}::load_weights(), dismec::model::Model::num_features(), dismec::io::parse_npy_header(), and THROW_ERROR.

Referenced by anonymous_namespace{model-io.cpp}::read_weights_dispatch(), and TEST_CASE().

◆ load_dense_weights_txt()

void dismec::io::model::load_dense_weights_txt	(	std::istream &	source,
		Model &	target
	)

Loads weights saved by io::model::save_dense_weights_txt.

Parameters

source	The stream from which to load the weights.
target	Model whose weights to fill in. It is assumed that `target` is already of the correct size.

Exceptions

If	there is an error when reading the weights, or if the number of labels in `target` mismatches the number of lines in `source`.

Definition at line 61 of file weights.cpp.

References anonymous_namespace{weights.cpp}::load_weights(), and dismec::io::read_vector_from_text().

Referenced by anonymous_namespace{model-io.cpp}::read_weights_dispatch(), and TEST_CASE().

◆ load_model()

std::shared_ptr< Model > dismec::io::model::load_model ( path source )

Definition at line 334 of file model-io.cpp.

References dismec::io::model::PartialModelLoader::load_model().

Referenced by PYBIND11_MODULE().

◆ load_sparse_weights_txt()

void dismec::io::model::load_sparse_weights_txt	(	std::istream &	source,
		Model &	target
	)

Loads sparse weights from plain-text format.

Parameters

source	The stream from which to load the weights.
target	Model whose weights to fill in. It is assumed that `target` is already of the correct size.

Exceptions

If	there is an error when reading the weights, or if the number of labels in `target` mismatches the number of lines in `source`.

Definition at line 140 of file weights.cpp.

References dismec::model::Model::labels_begin(), dismec::model::Model::labels_end(), dismec::model::Model::num_features(), dismec::model::Model::num_labels(), dismec::io::parse_sparse_vector_from_text(), dismec::model::Model::set_weights_for_label(), and THROW_ERROR.

Referenced by anonymous_namespace{model-io.cpp}::read_weights_dispatch(), and TEST_CASE().

◆ parse_weights_format()

WeightFormat dismec::io::model::parse_weights_format ( std::string_view name )

Gets the eighs.

Definition at line 84 of file model-io.cpp.

References anonymous_namespace{model-io.cpp}::weight_format_names.

Referenced by PYBIND11_MODULE(), and dismec::io::model::PartialModelIO::read_metadata_file().

◆ save_as_sparse_weights_txt()

void dismec::io::model::save_as_sparse_weights_txt	(	std::ostream &	target,
		const Model &	model,
		double	threshold
	)

Saves the weights in sparse plain-text format, culling small weights.

Parameters

target	Stream to which the weights are written.
model	Reference to the model whose weights will be saved. Note that if the model already has sparse weights, additional culling based in threshold will be performed over the nonzero weights. Each line in the resulting file corresponds to the weight vector of one label.
threshold	Threshold below which weights will be set to zero and omitted from the file.

Exceptions

If	`threshold < 0`. `target` remains unmodified in that case.

Definition at line 111 of file weights.cpp.

References dismec::model::Model::contained_labels(), dismec::model::Model::num_features(), and anonymous_namespace{weights.cpp}::save_weights().

Referenced by anonymous_namespace{model-io.cpp}::save_weights_dispatch(), and TEST_CASE().

◆ save_dense_weights_npy()

void dismec::io::model::save_dense_weights_npy	(	std::streambuf &	target,
		const Model &	model
	)

Saves the dense weights in a npy file.

The weights are saved in as a two dimensional array, with rows corresponding to labels and columns corresponding to features. The data is written in row-major format to allow loading a subset of the labels by reading contiguous parts of the file. Since the output is binary, we operate directly on a stream-buffer here.

Parameters

target	Output stream buffer where the data is written.
model	Reference to the model whose weights will be saved. If the model has sparse weights, the weight vectors will be converted to a dense format.

Definition at line 73 of file weights.cpp.

References dismec::io::binary_dump(), dismec::model::Model::contained_labels(), dismec::io::make_npy_description(), dismec::model::Model::num_features(), anonymous_namespace{weights.cpp}::save_weights(), and dismec::io::write_npy_header().

Referenced by anonymous_namespace{model-io.cpp}::save_weights_dispatch(), and TEST_CASE().

◆ save_dense_weights_txt()

void dismec::io::model::save_dense_weights_txt	(	std::ostream &	target,
		const Model &	model
	)

Saves the dense weights in a plain-text format.

Each row corresponds to one label, and each column to a feature.

Parameters

target	Output stream where the data is written.
model	Reference to the model whose weights will be saved. If the model has sparse weights, the weight vectors will be converted to a dense format.

Attention: Note that this is transposed from the way liblinear saves weights, but it makes it easier to read the weights for only a subset of the labels.

Definition at line 51 of file weights.cpp.

References anonymous_namespace{weights.cpp}::save_weights(), and dismec::io::write_vector_as_text().

Referenced by anonymous_namespace{model-io.cpp}::save_weights_dispatch(), and TEST_CASE().

◆ save_model()

void dismec::io::model::save_model	(	const path &	target_file,
		const std::shared_ptr< const Model > &	model,
		SaveOption	options
	)

Saves a complete model to a file.

This function saves a complete model using the specified options, to target_file. This function cannot be used to save a partial model!

Parameters

target_file	The path to the file where the metadata will be saved. Will also be used as prefix for the names of the weight files. If the files already exist, they will be overwritten.
model	The model to be saved.
options	Additional options to influence how the save file is generated.

Definition at line 308 of file model-io.cpp.

References dismec::io::model::SaveOption::SplitFiles.

◆ to_string()

const char * dismec::io::model::to_string ( WeightFormat format )

Definition at line 89 of file model-io.cpp.

References anonymous_namespace{model-io.cpp}::weight_format_names.

Referenced by main(), dismec::io::detail::print_char(), PYBIND11_MODULE(), and dismec::io::model::PartialModelSaver::update_meta_file().

Classes

Enumerations

Functions

Detailed Description

Enumeration Type Documentation

◆ WeightFormat

Function Documentation

◆ load_dense_weights_npy()

◆ load_dense_weights_txt()

◆ load_model()

◆ load_sparse_weights_txt()

◆ parse_weights_format()

◆ save_as_sparse_weights_txt()

◆ save_dense_weights_npy()

◆ save_dense_weights_txt()

◆ save_model()

◆ to_string()