interface EvaluatorType {
    criteria: LLMStringEvaluator<EvalOutputType, BaseLanguageModelInterface>;
    embedding_distance: StringEvaluator;
    labeled_criteria: LLMStringEvaluator<EvalOutputType, BaseLanguageModelInterface>;
    labeled_pairwise_string: LLMPairwiseStringEvaluator;
    pairwise_embedding_distance: PairwiseStringEvaluator;
    pairwise_string: LLMPairwiseStringEvaluator;
    trajectory: AgentTrajectoryEvaluator;
}

Properties

criteria: LLMStringEvaluator<EvalOutputType, BaseLanguageModelInterface>

The criteria evaluator, which evaluates a model based on a custom set of criteria without any reference labels.

embedding_distance: StringEvaluator

Compare a prediction to a reference label using embedding distance.

labeled_criteria: LLMStringEvaluator<EvalOutputType, BaseLanguageModelInterface>

The labeled criteria evaluator, which evaluates a model based on a custom set of criteria, with a reference label.

labeled_pairwise_string: LLMPairwiseStringEvaluator

The labeled pairwise string evaluator, which predicts the preferred prediction from between two models based on a ground truth reference label.

pairwise_embedding_distance: PairwiseStringEvaluator

Compare two predictions using embedding distance.

pairwise_string: LLMPairwiseStringEvaluator

The pairwise string evaluator, which predicts the preferred prediction from between two models.

trajectory: AgentTrajectoryEvaluator

The agent trajectory evaluator, which grades the agent's intermediate steps.