Running Models

deep_qa.run.compute_accuracy(predictions: <built-in function array>, labels: <built-in function array>)

Computes a simple categorical accuracy metric, useful if you used score_dataset to get predictions.

deep_qa.run.evaluate_model(param_path: str, dataset_files: typing.List[str] = None, model_class=None)

Loads a model and evaluates it on some test set.

Parameters:

param_path: str, required

A json file specifying a DeepQaModel.

dataset_files: List[str], optional, (default=None)

A list of dataset files to evaluate on. If this is None, we’ll evaluate from the test_files parameter in the input files. If that’s also None, we’ll crash.

model_class: DeepQaModel, optional (default=None)

This option is useful if you have implemented a new model class which is not one of the ones implemented in this library.

Returns:

Numpy arrays of model predictions in the format of model.outputs.

deep_qa.run.load_model(param_path: str, model_class=None)

Loads and returns a model.

Parameters:

param_path: str, required

A json file specifying a DeepQaModel.

model_class: DeepQaModel, optional (default=None)

This option is useful if you have implemented a new model class which is not one of the ones implemented in this library.

Returns:

A DeepQaModel instance.

deep_qa.run.prepare_environment(params: typing.Union[deep_qa.common.params.Params, dict])

Sets random seeds for reproducible experiments. This may not work as expected if you use this from within a python project in which you have already imported Keras. If you use the scripts/run_model.py entry point to training models with this library, your experiments should be reproducible. If you are using this from your own project, you will want to call this function before importing Keras.

Parameters:

params: Params object or dict, required.

A Params object or dict holding the json parameters.

deep_qa.run.run_model(param_dict: typing.Dict[str, <built-in function any>], model_class=None)

This function is the normal entry point to DeepQA. Use this to run a DeepQA model in your project. Note that if you care about exactly reproducible experiments, you should avoid importing Keras before you import and use this function, as Keras relies on random seeds which can be set in this function via a JSON specification file.

Note that this function performs training and will also evaluate the trained model on development and test sets if provided in the parameter json.

Parameters:

param_dict: Dict[str, any], required.

A parameter file specifying a DeepQaModel.

model_class: DeepQaModel, optional (default=None).

This option is useful if you have implemented a new model class which is not one of the ones implemented in this library.

deep_qa.run.run_model_from_file(param_path: str)

A wrapper around the run_model function which loads json from a file.

Parameters:

param_path: str, required.

A json paramter file specifying a DeepQA model.

deep_qa.run.score_dataset(param_path: str, dataset_files: typing.List[str], model_class=None)

Loads a model from a saved parameter path and scores a dataset with it, returning the predictions.

Parameters:

param_path: str, required

A json file specifying a DeepQaModel.

dataset_files: List[str]

A list of dataset files to score, the same as you would have specified as train_files or test_files in your parameter file.

model_class: DeepQaModel, optional (default=None)

This option is useful if you have implemented a new model class which is not one of the ones implemented in this library.

Returns:

predictions: numpy.array

Numpy array of model predictions in the format of model.outputs (typically one array, but could be List[numpy.array] if your model has multiple outputs).

labels: numpy.array

The labels on the dataset, as read by the model. We return this so you can compute whatever metrics you want, if the data was labeled.

deep_qa.run.score_dataset_with_ensemble(param_paths: typing.List[str], dataset_files: typing.List[str], model_class=None) → typing.Tuple[<built-in function array>, <built-in function array>]

Loads all of the models specified in param_paths, uses each of them to score the dataset specified by dataset_files, and averages their scores, return an array of ensembled model predictions.

Parameters:

param_paths: List[str]

A list of parameter files that were used to train models. You must have already trained the corresponding model, as we’ll load it and use it in an ensemble here.

dataset_files: List[str]

A list of dataset files to score, the same as you would have specified as test_files in any one of the model parameter files.

model_class: ``DeepQaModel``, optional (default=None)

This option is useful if you have implemented a new model class which is not one of the ones implemented in this library.

Returns:

predictions: numpy.array

Numpy array of model predictions in the format of model.outputs (typically one array, but could be List[numpy.array] if your model has multiple outputs).

labels: numpy.array

The labels on the dataset, as read by the first model. We return this so you can compute whatever metrics you want, if the data was labeled. Note that if your models all represent that data differently, this will only give the first one. Hopefully the representation of the labels is consistent across the models, though; if not, the whole idea of ensembling them this way is moot, anyway.