elicit.utils module

elicit.utils.save_as_pkl(obj: Any, save_dir: str) → None

Helper functions to save a file as pickle.

Parameters:

objany

variable that needs to be saved.

save_dirstr

path indicating the file location.

Returns:

None.

Examples

>>> save_as_pkl(obj, "results/file.pkl")

elicit.utils.identity(x: float) → Any

identity function. Returns the input

Parameters:

xany

arbitrary input object.

Returns:

xany

returns the input object without any transformation.

class elicit.utils.DoubleBound(lower: float, upper: float)

Bases: object

A variable constrained to be in the open interval (lower, upper) is transformed to an unconstrained variable Y via a scaled and translated log-odds transform.

Basis for the here used constraints, is the constraint transforms implementation in Stan.

Parameters:

lowerfloat

lower bound of variable x.

upperfloat

upper bound of variable x.

__init__(lower: float, upper: float)

A variable constrained to be in the open interval (lower, upper) is transformed to an unconstrained variable Y via a scaled and translated log-odds transform.

Basis for the here used constraints, is the constraint transforms implementation in Stan.

Parameters:

lowerfloat

lower bound of variable x.

upperfloat

upper bound of variable x.

logit(u: float) → float

Helper function that implements the logit transformation for \(u \in (0,1)\):

\[logit(u) = \log\left(\frac{u}{1-u}\right)\]

Parameters:

ufloat

variable in open unit interval.

Returns:

vfloat

log-odds of u.

inv_logit(v: float) → float

Helper function that implements the inverse-logit transformation (i.e., the logistic sigmoid for \(v \in (-\infty,+\infty)\):

\[logit^{-1}(v) = \frac{1}{1+\exp(-v)}\]

Parameters:

vfloat

unconstrained variable

Returns:

ufloat

logistic sigmoid of the unconstrained variable

forward(x: float) → float

scaled and translated logit transform of variable x with lower and upper bound into an unconstrained variable y.

\[Y = logit\left(\frac{X - lower}{upper - lower}\right)\]

Parameters:

xfloat

variable with lower and upper bound.

Returns:

yfloat

unconstrained variable.

inverse(y: float) → float

inverse of the log-odds transform applied to the unconstrained variable y in order to transform it into a constrained variable x with lower and upper bound.

\[X = lower + (upper - lower) \cdot logit^{-1}(Y)\]

Parameters:

yfloat

unconstrained variable

Returns:

xfloat

constrained variable with lower and upper bound

class elicit.utils.LowerBound(lower: float)

Bases: object

A variable with a lower bound is transformed to an unconstrained variable Y via an inverse-softplus transform.

Basis for the here used constraints, is the constraint transforms implementation in Stan.

Parameters:

lowerfloat

lower bound of variable X.

__init__(lower: float)

A variable with a lower bound is transformed to an unconstrained variable Y via an inverse-softplus transform.

Basis for the here used constraints, is the constraint transforms implementation in Stan.

Parameters:

lowerfloat

lower bound of variable X.

forward(x: float) → float

inverse-softplus transform of variable x with lower bound into an unconstrained variable y.

\[Y = softplus^{-1}(X - lower)\]

Parameters:

xfloat

variable with a lower bound.

Returns:

yfloat

unconstrained variable.

inverse(y: float) → float

softplus transform of unconstrained variable y into a constrained variable x with lower bound.

\[X = softplus(Y) + lower\]

Parameters:

yfloat

unconstrained variable.

Returns:

xfloat

variable with a lower bound.

class elicit.utils.UpperBound(upper: float)

Bases: object

A variable with an upper bound is transformed into an unconstrained variable Y via an inverse-softplus transform.

Basis for the here used constraints, is the constraint transforms implementation in Stan.

Parameters:

upperfloat

upper bound of variable X.

__init__(upper: float)

A variable with an upper bound is transformed into an unconstrained variable Y via an inverse-softplus transform.

Basis for the here used constraints, is the constraint transforms implementation in Stan.

Parameters:

upperfloat

upper bound of variable X.

forward(x: float) → float

inverse-softplus transform of variable x with upper bound into an unconstrained variable y.

\[Y = softplus^{-1}(upper - X)\]

Parameters:

xfloat

variable with an upper bound.

Returns:

yfloat

unconstrained variable.

inverse(y: float) → float

softplus transform of unconstrained variable y into a constrained variable x with upper bound.

\[X = upper - softplus(Y)\]

Parameters:

yfloat

unconstrained variable.

Returns:

xfloat

variable with an upper bound.

elicit.utils.one_forward_simulation(prior_model: Priors, model: Dict[str, Any], targets: List[Target], seed: int) → Tuple[dict, Tensor, dict, dict]

One forward simulation from prior samples to elicited statistics.

Parameters:

prior_modelinstance of Priors class objects

initialized prior distributions which can be used for sampling.

modeldict

specification of generative model using elicit.elicit.model().

targetslist

list of target quantities specified with elicit.elicit.target().

seed: int

random seed.

Returns:

elicited_statisticsdict

dictionary containing the elicited statistics that can be used to compute the loss components

prior_samplestf.Tensor

samples from prior distributions

model_simulationsdict

samples from the generative model (likelihood) given the prior samples for the model parameters

target_quantitiesdict

target quantities as a function of the model simulations.

elicit.utils.get_expert_data(trainer: Trainer, model: Dict[str, Any], targets: List[Target], expert: ExpertDict, parameters: List[Parameter], network: NFDict | None, seed: int) → Tuple[Dict[str, Tensor], Tensor | None]

Wrapper for loading the training data which can be expert data or data simulations using a pre-defined ground truth.

Parameters:

trainerdict

specification of training settings and meta-information for workflow using trainer()

modeldict

specification of generative model using elicit.elicit.model().

targetslist

list of target quantities specified with elicit.elicit.target().

expertdict

provide input data from expert or simulate data from oracle with either the data or simulator method of the elicit.elicit.Expert module.

parameterslist

list of model parameters specified with elicit.elicit.parameter().

networkdict, optional

specification of neural network using a method implemented in elicit.networks. Only required for deep_prior method. For parametric_prior use None.

seedint

internal seed for reproducible results

Returns:

expert_datadict

dictionary containing the training data. Must have same form as the model-simulated elicited statistics. Correct specification of keys can be checked using elicit.utils.get_expert_datformat()

expert_priortf.Tensor, shape: [B,num_samples,num_params] or None

samples from ground truth. Exists only if expert data are simulated from an oracle. Otherwise this output is None

elicit.utils.save(eliobj, name: str | None = None, file: str | None = None, overwrite: bool = False) → None

saves the eliobj as pickle.

Parameters:

eliobj: callable

instance of the elicit.elicit.Elicit() class.

name: str, optional

name of the saved .pkl file. File is saved as .results/{method}/{name}_{seed}.pkl

file: str, optional

path to file, including file name, e.g. file=”res” (saved as res.pkl) or file=”method1/res” (saved as method1/res.pkl)

overwrite: bool

whether to overwrite existing file. Default value is False.

elicit.utils.load(file: str) → Callable

loads a saved eliobj from specified path.

Parameters:

filestr

path where eliobj object is saved.

Returns:

eliobjel.elicit.Elicit obj

loaded eliobj object.

elicit.utils.parallel(runs: int = 4, cores: int | None = None, seeds: list | None = None) → Parallel

Specification for parallelizing training by running multiple training instances with different seeds simultaneously.

Parameters:

runs: int

Number of replication. The default is 4.

coresint, optional

Number of cores that should be used. The default is None which implies cores=runs.

seedslist, optional

A list of seeds. If None seeds are drawn from a Uniform(0,999999) distribution. The seed information corresponding to each chain is stored in eliobj.results.

Returns:

parallel_dictdict

dictionary containing the parallelization settings.

elicit.utils.save_history(loss: bool = True, loss_component: bool = True, time: bool = True, hyperparameter: bool = True, hyperparameter_gradient: bool = True) → SaveHist

Controls whether sub-results of the history object should be included or excluded. Results are saved across epochs. By default all sub-results are included.

Parameters:

lossbool

total loss per epoch. The default is True.

loss_componentbool

loss per loss-component per epoch. The default is True.

timebool

time in sec per epoch. The default is True.

hyperparameterbool

‘parametric_prior’ method: Trainable hyperparameters of parametric prior distributions. ‘deep_prior’ method: Mean and standard deviation of each marginal from the joint prior. The default is True.

hyperparameter_gradientbool

Gradients of the hyperparameter. Only for ‘parametric_prior’ method. The default is True.

Returns:

save_hist_dictdict

dictionary with inclusion/exclusion settings for each sub-result in history object.

Warning

if loss_component or loss are excluded, elicit.plots.loss() can’t be used as it requires this information.

if hyperparameter is excluded, elicit.plots.hyperparameter() can’t be used as it requires this information. Only parametric_prior method.

if hyperparameter is excluded, elicit.plots.marginals() can’t be used as it requires this information. Only deep_prior method.

elicit.utils.save_results(target_quantities: bool = True, elicited_statistics: bool = True, prior_samples: bool = True, model_samples: bool = True, expert_elicited_statistics: bool = True, expert_prior_samples: bool = True, init_loss_list: bool = True, init_prior: bool = True, init_matrix: bool = True, loss_tensor_expert: bool = True, loss_tensor_model: bool = True) → SaveResults

Controls whether sub-results of the result object should be included or excluded in the final result file. Results are based on the computation of the last epoch. By default all sub-results are included.

Parameters:

target_quantitiesbool

simulation-based target quantities. The default is True.

elicited_statisticsbool

simulation-based elicited statistics. The default is True.

prior_samplesbool

samples from simulation-based prior distributions. The default is True.

model_samplesbool

output variables from the simulation-based generative model. The default is True.

expert_elicited_statisticsbool

expert-elicited statistics. The default is True.

expert_prior_samplesbool

if oracle is used: samples from the true prior distribution, otherwise it is None. The default is True.

init_loss_listbool

initialization phase: Losses related to the samples drawn from the initialization distribution. Only included for method ‘parametric_prior’. The default is True.

init_priorbool

initialized elicit model object including the trainable variables. Only included for method ‘parametric_prior’. The default is True.

init_matrixbool

initialization phase: samples drawn from the initialization distribution for each hyperparameter. Only included for method ‘parametric_prior’. The default is True.

loss_tensor_expertbool

expert term in loss component for computing the discrepancy. The default is True.

loss_tensor_modelbool

simulation-based term in loss component for computing the discrepancy. The default is True.

Returns:

save_res_dictdict

dictionary with inclusion/exclusion settings for each sub-result in results object.

Warning

if elicited_statistics is excluded elicit.plots.loss() can’t be used as it requires this information.

if init_matrix is excluded elicit.plots.initialization() can’t be used as it requires this information.

if prior_samples is excluded elicit.plots.prior_joint() can’t be used as it requires this information.

if target_quantities is excluded elicit.plots.priorpredictive() can’t be used as it requires this information.

if expert_elicited_statistics or elicited_statistics is excluded elicit.plots.elicits() can’t be used as it requires this information.

elicit.utils.clean_savings(history: dict, results: dict, save_history: SaveHist, save_results: SaveResults) → Tuple[dict, dict]

Parameters:

historydict

results that are saved across epochs including among others loss, loss_component, time, and hyperparameter. See save_history() for complete list.

resultsdict

results that are saved for the last epoch only including prior_samples, elicited_statistics, target_quantities, etc. See save_results() for complete list.

save_historydict, elicit.utils.save_history()

Exclude or include sub-results in the final result file. In the history object are all results that are saved across epochs. For usage information see How-To: Save and load the eliobj

save_resultsdict, elicit.utils.save_results()

Exclude or include sub-results in the final result file. In the results object are all results that are saved for the last epoch only. For usage information see How-To: Save and load the eliobj

Returns:

results, historyTuple[dict, dict]

final results taking in consideration exclusion criteria as specified in save_history() and save_results().

elicit.utils.get_expert_datformat(targets: List[Target]) → Dict[str, list]

helper function for the user to inspect which data format for the expert data is expected by the method.

Parameters:

targetslist

list of target quantities specified with elicit.elicit.target().

Returns:

elicit_dictdict[str, list]

expected format of expert data.

elicit.utils.softmax_gumbel_trick(likelihood: Callable, upper_thres: float, temp: float = 1.6, **kwargs)

The softmax-gumbel trick computes a continuous approximation of ypred from a discrete likelihood and thus allows for the computation of gradients for discrete random variables.

Currently this approach is only implemented for models without upper boundary (e.g., Poisson model).

Corresponding literature:

• Maddison, C. J., Mnih, A. & Teh, Y. W. The concrete distribution: A continuous relaxation of discrete random variables in International Conference on Learning Representations (2017). https://doi.org/10.48550/arXiv.1611.00712

• Jang, E., Gu, S. & Poole, B. Categorical reparameterization with gumbel-softmax in International Conference on Learning Representations (2017). https://openreview.net/forum?id=rkE3y85ee.

• Joo, W., Kim, D., Shin, S. & Moon, I.-C. Generalized gumbel-softmax gradient estimator for generic discrete random variables. Preprint at https://doi.org/10.48550/arXiv.2003.01847 (2020).

Parameters:

likelihoodCallable, tfp.distributions object, shape = [B, num_samples, num_obs, 1]

likelihood function used in the generative model. Must be a tfp.distributions object.

upper_thresfloat

upper threshold at which the distribution of the outcome variable is truncated. For double-bounded distribution (e.g. Binomial) this is simply the “total count” information. Lower-bounded distribution (e.g. Poisson) must be truncated to create an artificial double-boundedness.

tempfloat, temp > 0

temperature hyperparameter of softmax function. A temperature going towards zero yields approximates a categorical distribution, while a temperature >> 0 approximates a continuous distribution. The default value is 1.6.

kwargsany

additional keyword arguments including the seed information. Note: the **kwargs argument is required in this function (!) as it extracts internally the seed information.

Returns:

ypredtf.Tensor

continuously approximated ypred from the discrete likelihood.