EvaluationService

EvaluationService(self,
                  subset,
                  storage=None,
                  davis_root=None,
                  robot_parameters=None,
                  max_t=None,
                  max_i=None,
                  metric_to_optimize='J_AND_F',
                  time_threshold=None)
Class responsible of the evaluation.

This class is responsible of giving the samples to run the evaluation, to give the asked scribbles and to evaluate the interaction with the robot once the masks are submitted.

Arguments

  • subset: String. Subset to evaluate. Possible values are train, val, 'trainval' and test-dev.
  • davis_root: String or Path. Path to the DAVIS dataset root directory, where the scribbles and the masks are stored.
  • robot_parameters: Dictionary. Dictionary of parameters to initialize the scribbles robot.
  • max_t: Integer. Number of seconds maximum to evaluate a single sample. This value will overwrite the specified from the user at DavisInteractiveSession class.
  • max_i: Integer. Maximum number of interactions to evaluate per sample. This value will overwrite the specified from the user at DavisInteractiveSession class.
  • metric_to_optimize: Enum. Metric targeting to optimize. Possible values: J, F or J_AND_F.
  • time_threshold: Integer. Time in seconds to use it as threshold to compute the jaccard and compare the evaluation of different methods.

get_samples

EvaluationService.get_samples()
Get the list of samples.

Returns

List of Tuples: List of pairs where the first element is the sequence name and the second is the scribble index to evaluate.

get_scribble

EvaluationService.get_scribble(sequence, scribble_idx)
Get a scribble.

Arguments

  • sequence: String. Sequence name of the scribble.
  • scribble_idx: Integer. Index of the scribble to get.

Raises

  • ValueError: when the sequence is invalid or the scribble index is out of range.

post_predicted_masks

EvaluationService.post_predicted_masks(
  sequence,
  scribble_idx,
  pred_masks,
  timing,
  interaction,
  user_key,
  session_key,
  next_scribble_frame_candidates=None)
Post the predicted masks and return new scribble.

When the predicted masks are given, the metrics are computed and stored.

Arguments

  • sequence: String. Sequence name of the predicted masks.
  • scribble_idx: Integer. Scribble index of the sample evaluating.
  • pred_masks: Numpy Array. Predicted masks for the given sequence.
  • timing: Float. Timing in seconds of this interaction.
  • interaction: Integer. Interaction number.
  • user_key: String. User identifier.
  • session_key: String. Session identifier.
  • next_scribble_frame_candidates: List of Integers. Optional value specifying the possible frames from which generate the next scribble. If values given, the next scribble will be performed in the frame where the evaluation metric scores the least on the list of given frames. Invalid frames indexes are ignored.

Returns

Dictionary: Scribble returned by the scribble robot

Raises

  • RuntimeError: When a previous interaction is missing, or the interaction has already been submitted.
  • ValueError: When interaction is higher than the maximum number of interactions in the evaluation.

get_report

EvaluationService.get_report(**kwargs)
Get report for a session.

Arguments

  • user_key: String. User identifier.
  • session_key: String. Session identifier.

Returns

Pandas DataFrame: Report.

summarize_report

EvaluationService.summarize_report(df)
Given a report it will reconstruct the missing entries and compute a summarization of it.

Arguments

  • df: Pandas DataFrame. The report to summarize.

Returns

Dictionary: with different scores computed and the curve values