sleap.instance

Data structures for all labeled data contained with a SLEAP project.

The relationships between objects in this module:

• A LabeledFrame can contain zero or more Instance`s (and `PredictedInstance objects).

• Instance objects (and PredictedInstance objects) have PointArray (or PredictedPointArray).

• Instance (PredictedInstance) can be associated with a Track

• A PointArray (or PredictedPointArray) contains zero or more Point objects (or PredictedPoint objectss), ideally as many as there are in the associated Skeleton although these can get out of sync if the skeleton is manipulated.

class sleap.instance.Instance(skeleton: Skeleton, track: Track = None, from_predicted: Optional[PredictedInstance] = None, points: PointArray = None, nodes: List = None, frame: Optional[LabeledFrame] = None)

This class represents a labeled instance.

Parameters:

• skeleton – The skeleton that this instance is associated with.

• points – A dictionary where keys are skeleton node names and values are Point objects. Alternatively, a point array whose length and order matches skeleton.nodes.

• track – An optional multi-frame object track associated with this instance. This allows individual animals/objects to be tracked across frames.

• from_predicted – The predicted instance (if any) that this was copied from.

• frame – A back reference to the LabeledFrame that this Instance belongs to. This field is set when instances are added to LabeledFrame objects.

property bounding_box: ndarray

Return bounding box containing all points in [y1, x1, y2, x2] format.

property centroid: ndarray

Return instance centroid as an array of (x, y) coordinates

Notes

This computes the centroid as the median of the visible points.

fill_missing(max_x: Optional[float] = None, max_y: Optional[float] = None)

Add points for skeleton nodes that are missing in the instance.

This is useful when modifying the skeleton so the nodes appears in the GUI.

Parameters:

• max_x – If specified, make sure points are not added outside of valid range.

• max_y – If specified, make sure points are not added outside of valid range.

property frame_idx: Optional[int]

Return the index of the labeled frame this instance is associated with.

classmethod from_numpy(points: ndarray, skeleton: Skeleton, track: Optional[Track] = None) → Instance

Create an instance from a numpy array.

Parameters:

• points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

• skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the instance.

• track – Optional sleap.Track object to associate with the instance.

Returns:

A new Instance object.

Notes

This is an alias for Instance.from_pointsarray().

classmethod from_pointsarray(points: ndarray, skeleton: Skeleton, track: Optional[Track] = None) → Instance

Create an instance from an array of points.

Parameters:

• points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

• skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the instance.

• track – Optional sleap.Track object to associate with the instance.

Returns:

A new Instance object.

get_points_array(copy: bool = True, invisible_as_nan: bool = False, full: bool = False) → Union[ndarray, recarray]

Return the instance’s points in array form.

Parameters:

• copy – If True, the return a copy of the points array as an ndarray. If False, return a view of the underlying recarray.

• invisible_as_nan – Should invisible points be marked as NaN. If copy is False, then invisible_as_nan is ignored since we don’t want to set invisible points to NaNs in original data.

• full – If True, return all data for points. Otherwise, return just the x and y coordinates.

Returns:

Either a recarray (if copy is False) or an ndarray (if copy True).

The order of the rows corresponds to the ordering of the skeleton nodes. Any skeleton node not defined will have NaNs present.

Columns in recarray are accessed by name, e.g., [“x”], [“y”].

Columns in ndarray are accessed by number. The order matches the order in Point.dtype or PredictedPoint.dtype.

matches(other: Instance) → bool

Whether two instances match by value.

Checks the types, points, track, and frame index.

Parameters:

other – The other Instance.

Returns:

True if match, False otherwise.

property midpoint: ndarray

Return the center of the bounding box of the instance points.

property n_visible_points: int

Return the number of visible points in this instance.

property nodes: Tuple[Node, ...]

Return nodes that have been labelled for this instance.

property nodes_points: List[Tuple[Node, Point]]

Return a list of (node, point) tuples for all labeled points.

numpy() → ndarray

Return the instance node coordinates as a numpy array.

Alias for points_array.

Returns:

Array of shape (n_nodes, 2) of dtype float32 containing the coordinates of the instance’s nodes. Missing/not visible nodes will be replaced with NaN.

property points: Tuple[Point, ...]

Return a tuple of labelled points, in the order they were labelled.

property points_array: ndarray

Return array of x and y coordinates for visible points.

Row in array corresponds to order of points in skeleton. Invisible points will be denoted by NaNs.

Returns:

A numpy array of of shape (n_nodes, 2) point coordinates.

transform_points(transformation_matrix)

Apply affine transformation matrix to points in the instance.

Parameters:

transformation_matrix – Affine transformation matrix as a numpy array of shape (3, 3).

property video: Optional[Video]

Return the video of the labeled frame this instance is associated with.

class sleap.instance.InstancesList(*args, labeled_frame: Optional[LabeledFrame] = None)

A list of Instance`s associated with a `LabeledFrame.

This class should only be used for the LabeledFrame.instances attribute.

append(instance: Union[Instance, PredictedInstance])

Append an Instance or PredictedInstance to the list, setting the frame.

Parameters:

item – The Instance or PredictedInstance to append to the list.

clear() → None

Remove all instances from list, setting instance.frame to None.

copy() → list

Return a shallow copy of the list of instances as a list.

Note: This will not return an InstancesList object, but a normal list.

extend(instances: List[Union[PredictedInstance, Instance]])

Extend the list with a list of `Instance`s or `PredictedInstance`s.

Parameters:

instances – A list of Instance or PredictedInstance objects to add to the list.

Returns:

None

insert(index: int, instance: Union[Instance, PredictedInstance]) → None

Insert object before index.

property labeled_frame: LabeledFrame

Return the LabeledFrame associated with this list of instances.

pop(index: int) → Union[Instance, PredictedInstance]

Remove and return instance at index, setting instance.frame to None.

remove(instance: Union[Instance, PredictedInstance]) → None

Remove instance from list, setting instance.frame to None.

class sleap.instance.LabeledFrame(video: Video, frame_idx, instances: InstancesList = _Nothing.NOTHING)

Holds labeled data for a single frame of a video.

Parameters:

• video – The Video associated with this frame.

• frame_idx – The index of frame in video.

• instances – List of instances associated with the frame.

classmethod complex_frame_merge(base_frame: LabeledFrame, new_frame: LabeledFrame) → Tuple[List[Instance], List[Instance], List[Instance]]

Merge two frames, return conflicts if any.

A conflict occurs when * each frame has Instances which don’t perfectly match those

in the other frame, or

• each frame has PredictedInstances which don’t perfectly match those in the other frame.

Parameters:

• base_frame – The LabeledFrame into which we want to merge.

• new_frame – The LabeledFrame from which we want to merge.

Returns:

• list of instances that were merged

• list of conflicting instances from base

• list of conflicting instances from new

Return type:

tuple of three items

classmethod complex_merge_between(base_labels: Labels, new_frames: List[LabeledFrame]) → Tuple[Dict[Video, Dict[int, List[Instance]]], List[Instance], List[Instance]]

Merge data from new frames into a Labels object.

Everything that can be merged cleanly is merged, any conflicts are returned.

Parameters:

• base_labels – The Labels into which we are merging.

• new_frames – The list of LabeledFrame objects from which we are merging.

Returns:

• Dictionary, keys are Video, values are

  dictionary in which keys are frame index (int) and value is list of :class:`Instance`s

• list of conflicting Instance objects from base

• list of conflicting Instance objects from new frames

Return type:

tuple of three items

find(track: Optional[Union[Track, int]] = -1, user: bool = False) → List[Instance]

Retrieve instances (if any) matching specifications.

Parameters:

• track – The Track to match. Note that None will only match instances where :attribute:`Instance.track` is None. If track is -1, then we’ll match any track.

• user – Whether to only match user (non-predicted) instances.

Returns:

List of instances.

property has_predicted_instances: bool

Return whether the frame contains any predicted instances.

property has_tracked_instances: bool

Return whether the frame contains any predicted instances with tracks.

property has_user_instances: bool

Return whether the frame contains any user instances.

property image: ndarray

Return the image for this frame of shape (height, width, channels).

index(value: Instance) → int

Return index of given Instance.

insert(index: int, value: Instance)

Add instance to frame.

Parameters:

• index – The index in list of frame instances where we should insert the new instance.

• value – The instance to associate with frame.

Returns:

None.

property instances: List[Instance]

Return list of all instances associated with this frame.

property instances_to_show: List[Instance]

Return a list of instances to show in GUI for this frame.

This list will not include any predicted instances for which there’s a corresponding regular instance.

Returns:

List of instances to show in GUI.

static merge_frames(labeled_frames: List[LabeledFrame], video: Video, remove_redundant=True) → List[LabeledFrame]

Return merged LabeledFrames for same video and frame index.

Parameters:

• labeled_frames – List of LabeledFrame objects to merge.

• video – The Video for which to merge. This is specified so we don’t have to check all frames when we already know which video has new labeled frames.

• remove_redundant – Whether to drop instances in the merged frames where there’s a perfect match.

Returns:

The merged list of :class:`LabeledFrame`s.

property n_predicted_instances: int

Return the number of predicted instances in the frame.

property n_tracked_instances: int

Return the number of predicted instances with tracks in the frame.

property n_user_instances: int

Return the number of user instances in the frame.

numpy() → ndarray

Return the instances as an array of shape (instances, nodes, 2).

plot(image: bool = True, scale: float = 1.0)

Plot the frame with all instances.

Parameters:

• image – If False, only the instances will be plotted without loading the original image.

• scale – Relative scaling for the figure.

Notes

See sleap.nn.viz.plot_img and sleap.nn.viz.plot_instances for more plotting options.

plot_predicted(image: bool = True, scale: float = 1.0)

Plot the frame with all predicted instances.

Parameters:

• image – If False, only the instances will be plotted without loading the original image.

• scale – Relative scaling for the figure.

Notes

See sleap.nn.viz.plot_img and sleap.nn.viz.plot_instances for more plotting options.

property predicted_instances: List[PredictedInstance]

Return list of predicted instances associated with frame.

remove_empty_instances()

Remove instances with no visible nodes from the labeled frame.

remove_untracked()

Removes any instances without a track assignment.

property tracked_instances: List[PredictedInstance]

Return list of predicted instances with tracks associated with frame.

property training_instances: List[Instance]

Return list of user instances with points for training.

property unused_predictions: List[Instance]

Return a list of “unused” PredictedInstance objects in frame.

This is all the PredictedInstance objects which do not have a corresponding Instance in the same track in frame.

property user_instances: List[Instance]

Return list of user instances associated with this frame.

class sleap.instance.Point(x: float = nan, y: float = nan, visible: bool = True, complete: bool = False)

A labelled point and any metadata associated with it.

Parameters:

• x – The horizontal pixel location of point within image frame.

• y – The vertical pixel location of point within image frame.

• visible – Whether point is visible in the labelled image or not.

• complete – Has the point been verified by the user labeler.

dtype = dtype([('x', '<f8'), ('y', '<f8'), ('visible', '?'), ('complete', '?')])

isnan() → bool

Are either of the coordinates a NaN value.

Returns:

True if x or y is NaN, False otherwise.

numpy() → ndarray

Return the point as a numpy array.

class sleap.instance.PointArray(shape, buf=None, offset=0, strides=None, formats=None, names=None, titles=None, byteorder=None, aligned=False, order='C')

PointArray is a sub-class of numpy recarray which stores Point objects as records.

classmethod from_array(a: PointArray) → PointArray

Converts a PointArray (or child) to a new instance.

This will convert an object to the same type as itself, so a PredictedPointArray will result in the same.

Uses the default attribute values for new array.

Parameters:

a – The array to convert.

Returns:

A PointArray or PredictedPointArray with the same points as a.

classmethod make_default(size: int) → PointArray

Construct a point array where points are all set to default.

The constructed PointArray will have specified size and each value in the array is assigned the default values for a Point`.

Parameters:

size – The number of points to allocate.

Returns:

A point array with all elements set to Point()

class sleap.instance.PredictedInstance(skeleton: Skeleton, track: Track = None, from_predicted: Optional[PredictedInstance] = None, points: PointArray = None, nodes: List = None, frame: Optional[LabeledFrame] = None, score=0.0, tracking_score=0.0)

A predicted instance is an output of the inference procedure.

Parameters:

• score – The instance-level grouping prediction score.

• tracking_score – The instance-level track matching score.

classmethod from_arrays(points: ndarray, point_confidences: ndarray, instance_score: float, skeleton: Skeleton, track: Optional[Track] = None, tracking_score: float = 0.0) → PredictedInstance

Create a predicted instance from data arrays.

Parameters:

• points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

• point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

• instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

• skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

• track – Optional sleap.Track to associate with the instance.

• tracking_score – Optional float representing the track matching score.

Returns:

A new PredictedInstance.

classmethod from_instance(instance: Instance, score: float, tracking_score: float = 0.0) → PredictedInstance

Create a PredictedInstance from an Instance.

The fields are copied in a shallow manner with the exception of points. For each point in the instance a PredictedPoint is created with score set to default value.

Parameters:

• instance – The Instance object to shallow copy data from.

• score – The score for this instance.

• tracking_score – The tracking score for this instance.

Returns:

A PredictedInstance for the given Instance.

classmethod from_numpy(points: ndarray, point_confidences: ndarray, instance_score: float, skeleton: Skeleton, track: Optional[Track] = None, tracking_score: float = 0.0) → PredictedInstance

Create a predicted instance from data arrays.

Parameters:

• points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

• point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

• instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

• skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

• track – Optional sleap.Track to associate with the instance.

• tracking_score – Optional float representing the track matching score.

Returns:

A new PredictedInstance.

classmethod from_pointsarray(points: ndarray, point_confidences: ndarray, instance_score: float, skeleton: Skeleton, track: Optional[Track] = None, tracking_score: float = 0.0) → PredictedInstance

Create a predicted instance from data arrays.

Parameters:

• points – A numpy array of shape (n_nodes, 2) and dtype float32 that contains the points in (x, y) coordinates of each node. Missing nodes should be represented as NaN.

• point_confidences – A numpy array of shape (n_nodes,) and dtype float32 that contains the confidence/score of the points.

• instance_score – Scalar float representing the overall instance score, e.g., the PAF grouping score.

• skeleton – A sleap.Skeleton instance with n_nodes nodes to associate with the predicted instance.

• track – Optional sleap.Track to associate with the instance.

• tracking_score – Optional float representing the track matching score.

Returns:

A new PredictedInstance.

property points_and_scores_array: ndarray

Return the instance points and scores as an array.

This will be a (n_nodes, 3) array of (x, y, score) for each predicted point.

Rows in the array correspond to the order of points in skeleton. Invisible points will be represented as NaNs.

property scores: ndarray

Return point scores for each predicted node.

class sleap.instance.PredictedPoint(x: float = nan, y: float = nan, visible: bool = True, complete: bool = False, score: float = 0.0)

A predicted point is an output of the inference procedure.

It has all the properties of a labeled point, plus a score.

Parameters:

• x – The horizontal pixel location of point within image frame.

• y – The vertical pixel location of point within image frame.

• visible – Whether point is visible in the labelled image or not.

• complete – Has the point been verified by the user labeler.

• score – The point-level prediction score.

dtype = dtype([('x', '<f8'), ('y', '<f8'), ('visible', '?'), ('complete', '?'), ('score', '<f8')])

classmethod from_point(point: Point, score: float = 0.0) → PredictedPoint

Create a PredictedPoint from a Point

Parameters:

• point – The point to copy all data from.

• score – The score for this predicted point.

Returns:

A scored point based on the point passed in.

class sleap.instance.PredictedPointArray(shape, buf=None, offset=0, strides=None, formats=None, names=None, titles=None, byteorder=None, aligned=False, order='C')

PredictedPointArray is analogous to PointArray except for predicted points.

classmethod to_array(a: PredictedPointArray) → PointArray

Convert a PredictedPointArray to a normal PointArray.

Parameters:

a – The array to convert.

Returns:

The converted array.

class sleap.instance.Track(spawned_on=0, name='')

A track object is associated with a set of animal/object instances across multiple frames of video. This allows tracking of unique entities in the video over time and space.

Parameters:

• spawned_on – The video frame that this track was spawned on.

• name – A name given to this track for identifying purposes.

matches(other: Track)

Check if two tracks match by value.

Parameters:

other – The other track to check

Returns:

True if they match, False otherwise.

sleap.instance.make_instance_cattr() → Converter

Create a cattr converter for Lists of Instances/PredictedInstances.

This is required because cattrs doesn’t automatically detect the class when the attributes of one class are a subset of another.

Returns:

A cattr converter with hooks registered for structuring and unstructuring Instance and PredictedInstance objects.