The Benchmark class is designed to evaluate the performance of an Engine. More...

#include <Benchmark.h>

Collaboration diagram for fl::Benchmark:

Public Types
enum	TimeUnit { NanoSeconds, MicroSeconds, MilliSeconds, Seconds, Minutes, Hours }
	Unit of time to utilize in the results. More...

enum	TableShape { Horizontal, Vertical }
	Shape of the table of results. More...

enum	TableContents { Header = 1, Body = 2, HeaderAndBody = (Header \| Body) }
	Contents of the table of results. More...

enum	ErrorType { NonFinite, Accuracy, All }
	Type of error between expected and obtained values. More...

typedef std::pair< std::string, std::string >	Result
	Result is a type definition for a pair of strings. More...

Public Member Functions
	Benchmark (const std::string &name="", Engine *engine=fl::null, scalar tolerance=fuzzylite::macheps())

virtual	~Benchmark ()

void	setName (const std::string &name)
	Sets the name of the benchmark. More...

std::string	getName () const
	Gets the name of the benchmark. More...

void	setEngine (Engine *engine)
	Sets the engine to benchmark. More...

Engine *	getEngine () const
	Gets the engine to benchmark. More...

void	setExpected (const std::vector< std::vector< scalar > > &expected)
	Sets the set of expected values from the engine, where the inner vector contains the input values and output values. More...

const std::vector< std::vector< scalar > > &	getExpected () const
	Gets the set of expected values from the engine, where the inner vector contains the input values and output values. More...

void	setObtained (const std::vector< std::vector< scalar > > &obtained)
	Sets the set of obtained values from the engine, where the inner vector contains the input values and output values. More...

const std::vector< std::vector< scalar > > &	getObtained () const
	Gets the set of obtained values from the engine, where the inner vector contains the input values and output values. More...

void	setTimes (const std::vector< scalar > times)
	Sets the vector of nanoseconds taken to produce the set of obtained values from the set of expected input values. More...

const std::vector< scalar > &	getTimes () const
	Gets the vector of nanoseconds taken to produce the set of obtained values from the set of expected input values. More...

void	setTolerance (scalar tolerance)
	Sets the tolerance above which the difference between an expected and obtained value from the engine is considered an error. More...

scalar	getTolerance () const
	Gets the tolerance above which the difference between an expected and obtained value from the engine is considered an error. More...

virtual void	prepare (int values, FldExporter::ScopeOfValues scope)
	Produces and loads into memory the set of expected values from the engine. More...

virtual void	prepare (std::istream &reader, long numberOfLines=-1)
	Reads and loads into memory the set of expected values from the engine. More...

virtual scalar	runOnce ()
	Runs the benchmark on the engine only once. More...

virtual std::vector< scalar >	run (int times)
	Runs the benchmark on the engine multiple times. More...

virtual void	reset ()
	Resets the benchmark to be ready to run again. More...

virtual bool	canComputeErrors () const
	Indicates whether errors can be computed based on the expected and obtained values from the benchmark. More...

virtual scalar	meanSquaredError () const
	Computes the mean squared error over all output variables considering only those cases where there is an accuracy error as defined in Benchmark::accuracyErrors(). More...

virtual scalar	meanSquaredError (const OutputVariable *outputVariable) const
	Computes the mean squared error of the given output variable considering only those cases where there is an accuracy error as defined in Benchmark::accuracyErrors(). More...

virtual int	allErrors () const
	Computes the number of errors over all the output variables caused by non-finite differences or accuracy differences. More...

virtual int	allErrors (const OutputVariable *outputVariable) const
	Computes the number of errors of the given output variable caused by non-finite differences or accuracy differences. More...

virtual int	nonFiniteErrors () const
	Computes the number of errors over all the output variables caused by non-finite differences (ie, infinity and NaN). More...

virtual int	nonFiniteErrors (const OutputVariable *outputVariable) const
	Computes the number of errors of the given output variable caused by non-finite differences (ie, infinity and NaN). More...

virtual int	accuracyErrors () const
	Computes the number of errors over all the output variables caused by a significant difference in accuracy. More...

virtual int	accuracyErrors (const OutputVariable *outputVariable) const
	Computes the number of errors over the given output variable caused by a significant difference in accuracy. More...

virtual int	numberOfErrors (ErrorType errorType) const
	Computes the number of errors of the given type over all the output variables. More...

virtual int	numberOfErrors (ErrorType errorType, const OutputVariable *outputVariable) const
	Computes the number of errors of the given type over the given output variable. More...

virtual std::vector< std::string >	header (int runs, bool includeErrors=true)
	Returns the header of a horizontal table of results. More...

virtual std::vector< Result >	results (TimeUnit timeUnit=NanoSeconds, bool includeTimes=true) const
	Computes and returns the results from the benchmark aggregating the statistics of all the output variables. More...

virtual std::vector< Result >	results (const OutputVariable *outputVariable, TimeUnit timeUnit=NanoSeconds, bool includeTimes=true) const
	Computes and returns the results from the benchmark for the given output variable. More...

virtual std::string	format (std::vector< Result > results, TableShape shape, TableContents contents, const std::string &delimiter="\) const
	Formats the results. More...

Static Public Member Functions
static std::string	stringOf (TimeUnit unit)
	Returns the name of the time unit. More...

static scalar	factorOf (TimeUnit unit)
	Returns the factor of the given unit from NanoSeconds. More...

static scalar	convert (scalar time, TimeUnit from, TimeUnit to)
	Converts the time to different scales. More...

Detailed Description

The Benchmark class is designed to evaluate the performance of an Engine.

Author: Juan Rada-Vilela, Ph.D.

See also: Engine

Since: 6.0

Definition at line 39 of file Benchmark.h.

Member Typedef Documentation

◆ Result

typedef std::pair<std::string, std::string> fl::Benchmark::Result

Result is a type definition for a pair of strings.

Definition at line 364 of file Benchmark.h.

Member Enumeration Documentation

◆ ErrorType

enum fl::Benchmark::ErrorType

Type of error between expected and obtained values.

Enumerator
NonFinite
Accuracy
All

Definition at line 74 of file Benchmark.h.

◆ TableContents

enum fl::Benchmark::TableContents

Contents of the table of results.

Enumerator
Header
Body
HeaderAndBody

Definition at line 67 of file Benchmark.h.

◆ TableShape

enum fl::Benchmark::TableShape

Shape of the table of results.

Enumerator
Horizontal
Vertical

Definition at line 60 of file Benchmark.h.

◆ TimeUnit

enum fl::Benchmark::TimeUnit

Unit of time to utilize in the results.

Enumerator
NanoSeconds
MicroSeconds
MilliSeconds
Seconds
Minutes
Hours

Definition at line 53 of file Benchmark.h.

Constructor & Destructor Documentation

◆ Benchmark()

fl::Benchmark::Benchmark	(	const std::string &	name = `""`,
		Engine *	engine = `fl::null`,
		scalar	tolerance = `fuzzylite::macheps()`
	)

explicit

◆ ~Benchmark()

virtual fl::Benchmark::~Benchmark ( )

virtual

Member Function Documentation

◆ accuracyErrors() [1/2]

virtual int fl::Benchmark::accuracyErrors ( ) const

virtual

Computes the number of errors over all the output variables caused by a significant difference in accuracy.

An error is counted when the absolute difference between the expected and obtained values is not smaller than the tolerance.

$\text{E} = \sum_y \sum_i \epsilon_i^y, \text{where } \epsilon_i^y = \begin{cases} 0 & \text{if} |e_i^y - o^y_i| < \theta\\ 1 & \text{otherwise} \end{cases} $, $y$ is the set of output variables, $e$ is the set of expected output values, $o$ is the set of obtained output values, and $\theta$ is the tolerance

Returns: the number of errors over all the output variables caused by a significant difference in accuracy

◆ accuracyErrors() [2/2]

virtual int fl::Benchmark::accuracyErrors ( const OutputVariable * outputVariable ) const

virtual

Computes the number of errors over the given output variable caused by a significant difference in accuracy.

An error is counted when the absolute difference between the expected and obtained values is not smaller than the tolerance.

$\text{E} = \sum_i \epsilon_i, \text{where } \epsilon_i = \begin{cases} 0 & \text{if} |e_i - o_i| < \theta\\ 1 & \text{otherwise} \end{cases} $, $e$ is the set of expected output values, $o$ is the set of obtained output values, and $\theta$ is the tolerance

Parameters

outputVariable is the output variable to account the errors for

Returns: the number of errors of the given output variable caused by a significant difference in accuracy

◆ allErrors() [1/2]

virtual int fl::Benchmark::allErrors ( ) const

virtual

Computes the number of errors over all the output variables caused by non-finite differences or accuracy differences.

An error is counted when the difference between the expected and obtained values is not finite, or the absolute difference between the expected and obtained values is not smaller than the tolerance.

Returns: the number of errors over all the output variables caused by non-finite differences or accuracy differences

◆ allErrors() [2/2]

virtual int fl::Benchmark::allErrors ( const OutputVariable * outputVariable ) const

virtual

Computes the number of errors of the given output variable caused by non-finite differences or accuracy differences.

An error is counted when the difference between the expected and obtained values is not finite, or the absolute difference between the expected and obtained values is not smaller than the tolerance.

Parameters

outputVariable is the output variable to account the errors for

Returns: the number of errors of the given output variable caused by non-finite differences or accuracy differences

◆ canComputeErrors()

virtual bool fl::Benchmark::canComputeErrors ( ) const

virtual

Indicates whether errors can be computed based on the expected and obtained values from the benchmark.

If the benchmark was prepared from a file reader and the file included columns of expected output values and the benchmark has been run at least once, then the benchmark can automatically compute the errors and will automatically include them in the results.

Returns: whether errors can be computed based on the expected and obtained values from the benchmark

◆ convert()

static scalar fl::Benchmark::convert	(	scalar	time,
		TimeUnit	from,
		TimeUnit	to
	)

static

Converts the time to different scales.

Parameters

time	is the time to convert
from	is the units of the time to convert from
to	is the units of the time to convert to

Returns: the time in the units specified

◆ factorOf()

static scalar fl::Benchmark::factorOf ( TimeUnit unit )

static

Returns the factor of the given unit from NanoSeconds.

Parameters

unit	is the time unit

Returns: the factor of the given unit from NanoSeconds

◆ format()

virtual std::string fl::Benchmark::format	(	std::vector< Result >	results,
		TableShape	shape,
		TableContents	contents
	)

virtual

Formats the results.

Parameters

results	is the vector of results
shape	is the shape to present the table of results
contents	indicates the information to include in the table of results
delimiter	is the delimiter of the table of results

Returns: the formatted results from the benchmark

◆ getEngine()

Engine* fl::Benchmark::getEngine ( ) const

Gets the engine to benchmark.

Returns: the engine to benchmark

◆ getExpected()

const std::vector<std::vector<scalar> >& fl::Benchmark::getExpected ( ) const

Gets the set of expected values from the engine, where the inner vector contains the input values and output values.

Returns: the set of expected values from the engine

◆ getName()

std::string fl::Benchmark::getName ( ) const

Gets the name of the benchmark.

Returns: name is the name of the benchmark

◆ getObtained()

const std::vector<std::vector<scalar> >& fl::Benchmark::getObtained ( ) const

Gets the set of obtained values from the engine, where the inner vector contains the input values and output values.

Returns: the set of obtained values from the engine

◆ getTimes()

const std::vector<scalar>& fl::Benchmark::getTimes ( ) const

Gets the vector of nanoseconds taken to produce the set of obtained values from the set of expected input values.

Returns: the vector of nanoseconds taken to produce the set of obtained values from the set of expected input values

◆ getTolerance()

scalar fl::Benchmark::getTolerance ( ) const

Gets the tolerance above which the difference between an expected and obtained value from the engine is considered an error.

Returns: the tolerance above which the difference between an expected and obtained value from the engine is considered an error

◆ header()

virtual std::vector<std::string> fl::Benchmark::header	(	int	runs,
		bool	includeErrors = `true`
	)

virtual

Returns the header of a horizontal table of results.

Parameters

runs	is the number of times the benchmark will be run, hence producing the relevant number of columns for each run
includeErrors	indicates whether to include columns for computing the errors

Returns: the header of a horizontal table of results

◆ meanSquaredError() [1/2]

virtual scalar fl::Benchmark::meanSquaredError ( ) const

virtual

Computes the mean squared error over all output variables considering only those cases where there is an accuracy error as defined in Benchmark::accuracyErrors().

Returns: the mean squared error over all the output variables.

◆ meanSquaredError() [2/2]

virtual scalar fl::Benchmark::meanSquaredError ( const OutputVariable * outputVariable ) const

virtual

Computes the mean squared error of the given output variable considering only those cases where there is an accuracy error as defined in Benchmark::accuracyErrors().

Parameters

outputVariable is the output variable to compute the errors for

Returns: the mean squared error over the given output variable.

◆ nonFiniteErrors() [1/2]

virtual int fl::Benchmark::nonFiniteErrors ( ) const

virtual

Computes the number of errors over all the output variables caused by non-finite differences (ie, infinity and NaN).

An error is counted when the difference between the expected and obtained values is not finite.

Returns: the number of errors over all the output variables caused by non-finite differences

◆ nonFiniteErrors() [2/2]

virtual int fl::Benchmark::nonFiniteErrors ( const OutputVariable * outputVariable ) const

virtual

Computes the number of errors of the given output variable caused by non-finite differences (ie, infinity and NaN).

An error is counted when the difference between the expected and obtained values is not finite.

Parameters

outputVariable is the output variable to account the errors for

Returns: the number of errors of the given output variable caused by non-finite differences

◆ numberOfErrors() [1/2]

virtual int fl::Benchmark::numberOfErrors ( ErrorType errorType ) const

virtual

Computes the number of errors of the given type over all the output variables.

Parameters

errorType is the type of error to account for

Returns: the number of errors over all the output variables

◆ numberOfErrors() [2/2]

virtual int fl::Benchmark::numberOfErrors	(	ErrorType	errorType,
		const OutputVariable *	outputVariable
	)		const

virtual

Computes the number of errors of the given type over the given output variable.

Parameters

errorType	is the type of error to account for
outputVariable	is the output variable to account the errors for

Returns: the number of errors over the given output variable

◆ prepare() [1/2]

virtual void fl::Benchmark::prepare	(	int	values,
		FldExporter::ScopeOfValues	scope
	)

virtual

Produces and loads into memory the set of expected values from the engine.

Parameters

values	is the number of values to evaluate the engine upon
scope	is the scope of the values to generate

Exceptions

Exception if the engine is not set

◆ prepare() [2/2]

virtual void fl::Benchmark::prepare	(	std::istream &	reader,
		long	numberOfLines = `-1`
	)

virtual

Reads and loads into memory the set of expected values from the engine.

Parameters

reader	is the reader of a set of lines containing space-separated values
numberOfLines	is the maximum number of lines to read from the reader, and a value $f =(, -1]$f@ reads the entire file.

◆ reset()

virtual void fl::Benchmark::reset ( )

virtual

Resets the benchmark to be ready to run again.

◆ results() [1/2]

virtual std::vector<Result> fl::Benchmark::results	(	TimeUnit	timeUnit = `NanoSeconds`,
		bool	includeTimes = `true`
	)		const

virtual

Computes and returns the results from the benchmark aggregating the statistics of all the output variables.

Parameters

timeUnit	is the unit of time of the results
includeTimes	indicates whether to include the times of each run

Returns: the results from the benchmark

◆ results() [2/2]

virtual std::vector<Result> fl::Benchmark::results	(	const OutputVariable *	outputVariable,
		TimeUnit	timeUnit = `NanoSeconds`,
		bool	includeTimes = `true`
	)		const

virtual

Computes and returns the results from the benchmark for the given output variable.

Parameters

outputVariable	is the output variable to compute the statistics for
timeUnit	is the unit of time of the results
includeTimes	indicates whether to include the times of each run

Returns: the results from the benchmark

◆ run()

virtual std::vector<scalar> fl::Benchmark::run ( int times )

virtual

Runs the benchmark on the engine multiple times.

Parameters

times is the number of times to run the benchmark on the engine

Returns: vector of the time in nanoseconds required by each run, which is also appended to the times stored in Benchmark::getTimes()

◆ runOnce()

virtual scalar fl::Benchmark::runOnce ( )

virtual

Runs the benchmark on the engine only once.

Returns: the time in nanoseconds required by the run, which is also appended to the times stored in Benchmark::getTimes()

◆ setEngine()

void fl::Benchmark::setEngine ( Engine * engine )

Sets the engine to benchmark.

Parameters

engine is the engine to benchmark

◆ setExpected()

void fl::Benchmark::setExpected ( const std::vector< std::vector< scalar > > & expected )

Sets the set of expected values from the engine, where the inner vector contains the input values and output values.

Parameters

expected is the set of expected values from the engine

◆ setName()

void fl::Benchmark::setName ( const std::string & name )

Sets the name of the benchmark.

Parameters

name	is the name of the benchmark

◆ setObtained()

void fl::Benchmark::setObtained ( const std::vector< std::vector< scalar > > & obtained )

Sets the set of obtained values from the engine, where the inner vector contains the input values and output values.

Parameters

obtained is the set of obtained values from the engine

◆ setTimes()

void fl::Benchmark::setTimes ( const std::vector< scalar > times )

Sets the vector of nanoseconds taken to produce the set of obtained values from the set of expected input values.

Parameters

times is the vector of nanoseconds taken to produce the set of obtained values from the set of expected input values

◆ setTolerance()

void fl::Benchmark::setTolerance ( scalar tolerance )

Sets the tolerance above which the difference between an expected and obtained value from the engine is considered an error.

Parameters

tolerance is the tolerance above which the difference between an expected and obtained value from the engine is considered an error

◆ stringOf()

static std::string fl::Benchmark::stringOf ( TimeUnit unit )

static

Returns the name of the time unit.

Parameters

unit	is the time unit

Returns: the name of the time unit

The documentation for this class was generated from the following file:

fuzzylite/fl/Benchmark.h

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Typedef Documentation

◆ Result

Member Enumeration Documentation

◆ ErrorType

◆ TableContents

◆ TableShape

◆ TimeUnit

Constructor & Destructor Documentation

◆ Benchmark()

◆ ~Benchmark()

Member Function Documentation

◆ accuracyErrors() [1/2]

◆ accuracyErrors() [2/2]

◆ allErrors() [1/2]

◆ allErrors() [2/2]

◆ canComputeErrors()

◆ convert()

◆ factorOf()

◆ format()

◆ getEngine()

◆ getExpected()

◆ getName()

◆ getObtained()

◆ getTimes()

◆ getTolerance()

◆ header()

◆ meanSquaredError() [1/2]

◆ meanSquaredError() [2/2]

◆ nonFiniteErrors() [1/2]

◆ nonFiniteErrors() [2/2]

◆ numberOfErrors() [1/2]

◆ numberOfErrors() [2/2]

◆ prepare() [1/2]

◆ prepare() [2/2]

◆ reset()

◆ results() [1/2]

◆ results() [2/2]

◆ run()

◆ runOnce()

◆ setEngine()

◆ setExpected()

◆ setName()

◆ setObtained()

◆ setTimes()

◆ setTolerance()

◆ stringOf()