probnmn.evaluators.program_prior_evaluator

class probnmn.evaluators.program_prior_evaluator.ProgramPriorEvaluator(config: probnmn.config.Config, models: Dict[str, Type[torch.nn.modules.module.Module]], gpu_ids: List[int] = [0], cpu_workers: int = 0)

Bases: probnmn.evaluators._evaluator._Evaluator

Performs evaluation for program_prior phase, using batches of evaluation examples from ProgramPriorDataset.

Parameters

config: Config

A Config object with all the relevant configuration parameters.

models: Dict[str, Type[nn.Module]]

All the models which interact with each other for evaluation. This should come from ProgramPriorTrainer.

gpu_ids: List[int], optional (default = [0])

List of GPU IDs to use or evaluation, [-1] - use CPU.

cpu_workers: int, optional (default = 0)

Number of CPU workers to use for fetching batch examples in dataloader.

Examples

To evaluate a pre-trained checkpoint:

>>> config = Config("config.yaml")  # PHASE must be "program_prior"
>>> trainer = ProgramPriorTrainer(config, serialization_dir="/tmp")
>>> trainer.load_checkpoint("/path/to/program_prior_checkpoint.pth")
>>> evaluator = ProgramPriorEvaluator(config, trainer.models)
>>> eval_metrics = evaluator.evaluate(num_batches=50)

evaluate(self, num_batches:Union[int, NoneType]=None)

Perform evaluation using first num_batches of dataloader and return all evaluation metrics from the models. After evaluation, also print some qualitative examples.

Parameters

num_batches: int, optional (default=None)

Number of batches to use from dataloader. If None, use all batches.

Returns

Dict[str, Any]

Final evaluation metrics for all the models. For program_prior phase, this dict will have keys: {"perplexity"}.

_do_iteration(self, batch:Dict[str, Any]) → Dict[str, Any]

Perform one iteration, given a batch. Take a forward pass to accumulate metrics in ProgramPrior.

Parameters

batch: Dict[str, Any]

A batch of evaluation examples sampled from dataloader.

Returns

Dict[str, Any]

An output dictionary containing predictions of next-time step, and loss (by teacher forcing). Nested dict structure:

{
    "program_prior": {"predictions", "loss"}
}