talmolab / sleap Goto Github PK

View Code? Open in Web Editor NEW

425.0 23.0 97.0 113.93 MB

A deep learning framework for multi-animal pose tracking.

Home Page: https://sleap.ai

License: Other

Batchfile 0.02% Shell 0.06% Python 99.92%

pose-estimation deep-learning behavior-analysis sleap leap animal-pose-estimation animal-tracking

sleap's People

Contributors

Stargazers

Watchers

Forkers

filos93 jens-k wolfffff wishgale victor8733 xuerenjie124 xyz1214599961 preeti98 healthonrails nachonacho2 evermountaintech jensblack kant rlinus hectorcarrion oeway zshubin yooooona luoyizhi516 runninghsus hiphonl jackiezhai jampajeen jawaechan giannottigiuseppe tyelab smidm muwutufu shreyashkad themalaland xmikezheng20 yunoinsky tianyuzelin tvajtay tatiats padillacoreanolab jgrewe getzze jmdelahanty catalystneuro sai-priyanka jackzhousz shantanuray hsingchien djbutler linwang9926 eberrigan airhors jaeronga kennytc ioodo toshea111 annie444 artur551 fabianplum lochhh ozgurcemozgen06 kfahn22 joshghimire calebweinreb ajayjanapareddi sahilsingh2402 chronopoulos zaher503 andystopia neutth gitttt-1234 shrivaths16 kevinz0217 anhlbt lambdaloop ssrinath22 jagirhussan pingdior yarikoptic vtsai881 dearborn-open-ai oliverdeveloperprojects shehel juanjosecas vaibhavtrip29 emanuelfromflorence wux024 jvshen benshgh gqcpm justinvshen maia-g arthurmayo hajin-park charithmadhuranga davorvr mweinbergumass

sleap's Issues

Loading saved skeleton in new project and re-saving doesn't update nodes list

Reproduce:

New project
Create skeleton with a few node names in common with saved skeleton
Save
Load saved skeleton
Save
Close and re-open project -> only nodes that existed in step 2 will be present

labels json doesn't add new nodes to:
"skeletons" -> "nodes" list.

IDs of previously existing nodes are updated, but new nodes are not added to this list.

Top-level "nodes" key is updated with new nodes though.

Refactor flow tracker

Implementation needs to more closely match the described algorithm in Xiao et al. (2018):

Namely:

Greedy assignment instead of Hungarian matching
Generalize to similarity function and implement OKS/mPCK/etc
Add bounding box NMS for shifted instances
Break up into parallelizable sub-modules (e.g., loading, optical flow, matching)
Algorithmic correctness tests

Additional enhancements:

Allow for max tracks to be set and perform best assignments when possible

Loading saved skeleton doesn't trigger unsaved modifications check

Reproduce:

Start new project
Load Skeleton from another saved project
File -> Save is grayed out and Ctrl + S doesn't trigger saving.

Workarounds:

Create any instance with the new skeleton to enable saving.
Use File -> Save as... to force overwriting of existing project file.

Adding/deleting node or edge does trigger unsaved modifications.

Auto-resolve paths to "/" version

Replace "" or "\" with "/" always when doing file path resolutions. Store all paths with forward slashes since they will work on all OSes.

Expose single instance mode in GUI for active learning

Checkbox in Expert Controls window below Use trained paf model that says something like Single instance mode?

Progress bar or at least messagebox during suggestion generation

GUI indicator of what the current video is

Easiest: video index within Labels.videos being displayed in the status bar

Backbones

Finish canonical DenseNet implementation
Fix Stacked UNet and generalize the stacked motif
MobileNetV1 (and V2 if possible)
HRNet [1] [2]
The rest of the ResNet, ResNeXT and VGG models from keras_applications

Visual model outputs doesn't release gpu memory

There's no way to remove a model you've loaded and release the gpu memory. Currently it's a bad idea to run active learning after you've used "visualize model outputs", but the gui doesn't tell you this.

Export training package to HDF5

A single HDF5 file analogous to the *.json.zip training package.

Must include images and all other metadata to exactly unstructure it equivalently to the json.zip method.

Auto-detect default grayscale channel choice in add video menu

Probably should just auto-detect from whether first and last channel are all the same pixel values?

Stop using deprecated tensorflow.distributions

https://github.com/murthylab/sleap/blob/ff6119056118bbf7d42c0e6d0d5714a3eb6dcef9/sleap/nn/peakfinding_tf.py#L104

import tensorflow_probability as tfp
tf.distributions.Normal -> tfp.distributions.Normal

or:

def normal_probs(x, loc, scale):
    log_unnormalized = -0.5 * tf.math.squared_difference(
        x / scale, loc / scale)
    log_normalization = 0.5 * np.log(2. * np.pi) + tf.math.log(scale)
    return tf.exp(log_unnormalized - log_normalization)

should work and avoids the need for a new dependency, but will need to add tests and/or make sure that ndarrays <-> tensors <-> casting is all done properly

Running inference with models for different skeletons fails when initializing

Workflow 1 (works as expected):

Create skeleton that matches a saved model
Run inference on suggested frames using Predict -> Expert Controls -> select saved models
Add new nodes/edges to the skeleton
Initializing instances from predicted instances uses locations of old nodes and positions the new ones randomly as expected.

Workflow 2 (does not work):

Create skeleton that matches a saved model
Add new nodes/edges to the skeleton
Run inference on suggested frames using Predict -> Expert Controls -> select saved models
Initializing instances from predicted instances positions ALL nodes randomly instead of using predicted locations for subset of nodes that the model was trained on.

The only difference here is the order of steps 2 and 3. Is node identity checking based on ID instead of string name?

(I know equality checking probably is, but is matching from a saved model also done this way?)

Do the IDs of previously existing nodes change when the skeleton is modified? If so, how does workflow 1 work after new nodes are added?

Loss monitor fails to display progress during training

Possibly due to outlier estimation?

Worked out examples for core use cases

Single instance (can borrow from LEAP 1.0)
Multi-instance without centroids
Multi-instance with centroids

GUI-configurable keyboard shortcuts

Would be a nice feature to be able to customize sleap/config/shortcuts.yaml from the GUI.

Can also double as a table to be displayed from the Help -> Keyboard Reference menu item.

Persisting Instance <-> PredictedInstance association through HDF5 serialization

Instance.from_predicted field not stored in HDF5?

Consequence is that predicted instances are still displayed when saving -> reloading:

file access collision when saving h5 during inference

error:

   File "C:\Users\Sama\Anaconda\envs\sleap\lib\multiprocessing\pool.py", line 119, in worker
     result = (True, func(*args, **kwds))
   File "c:\code\sleap\sleap\nn\inference.py", line 355, in predict
     Labels.save_hdf5(labels, filename=self.output_path)
   File "c:\code\sleap\sleap\io\dataset.py", line 1076, in save_hdf5
     os.unlink(filename) PermissionError: [WinError 5] Access is denied: 'C:/Users/Sama/Desktop/abdus-saboor/grant\\models\\190923_104919.inferen ce.h5' """  The above exception was the direct cause of the following exception:  Traceback (most recent call last):
   File "c:\code\sleap\sleap\gui\active.py", line 344, in run
     with_tracking = with_tracking)
   File "c:\code\sleap\sleap\gui\active.py", line 632, in run_active_learning_pipeline
     run_active_inference(labels, trained_jobs, save_dir, frames_to_predict, with_tracking)
   File "c:\code\sleap\sleap\gui\active.py", line 777, in run_active_inference
     result.get()
   File "C:\Users\Sama\Anaconda\envs\sleap\lib\multiprocessing\pool.py", line 644, in get
     raise self._value PermissionError: [Errno 13] Access is denied: 'C:/Users/Sama/Desktop/abdus-saboor/grant\\models\\190923_104919.inference .h5'

Import labels with merging

Interface:

Open base Labels project
File -> Import labels...
Select another Labels file
(Optional?) Dialog asking how to resolve merge conficts

Functionality:

Merging of Video, Track, Instance, PredictedInstance based on .matches() (value-based) identity, adding to appropriate LabeledFrames as needed or importing new ones
Implement .matches()-like merging for Suggestions
For merge conflicts, possible options for user to select:
- Accept all from base Labels
- Accept all from new Labels
- (For PredictedInstances/Tracks:) Accept higher scoring predictions
Merging with mismatching skeletons:
- Initially, just reject since they won't pass matches() tests
- Lower priority but should be addressed before closing issue: handle use case of importing Instances/PredictedInstances with partially overlapping skeletons (e.g., 6 node vs 32 node flies), merge skeleton graphs, update existing instances if skeleton changed.

Test data:
/tigress/MMURTHY/talmo/wt_gold_labeling/091319.sleap_wt_gold.30pt_init.n=19.talmo.h5
+
/tigress/MMURTHY/talmo/wt_gold_labeling/091719.sleap_wt_gold.30pt_init.n=19.junyu.h5
These have the same suggestions, videos and predicted instances, but different user-labeled instances with some duplicates

GUI to review/edit negative samples

There's no way to get a list of all your negative samples or to remove any once they've been added.

UNet + PAFs

Using this profile results in two exceptions:

{
    "model": {
        "output_type": 1,
        "backbone": {
            "down_blocks": 3,
            "up_blocks": 3,
            "convs_per_depth": 0,
            "num_filters": 16,
            "kernel_size": 0,
            "upsampling_layers": true,
            "interp": "bilinear"
        },
        "skeletons": null,
        "backbone_name": "UNet"
    },
    "trainer": {
        "val_size": 0.1,
        "optimizer": "adam",
        "learning_rate": "5e-05",
        "amsgrad": true,
        "batch_size": 2,
        "num_epochs": 150,
        "steps_per_epoch": 200,
        "shuffle_initially": true,
        "shuffle_every_epoch": true,
        "augment_rotation": 180,
        "augment_scale_min": 1.0,
        "augment_scale_max": 1.0,
        "save_every_epoch": false,
        "save_best_val": true,
        "reduce_lr_min_delta": "1e-06",
        "reduce_lr_factor": 0.5,
        "reduce_lr_patience": 8,
        "reduce_lr_cooldown": 3,
        "reduce_lr_min_lr": "1e-10",
        "early_stopping_min_delta": "1e-08",
        "early_stopping_patience": 30.0,
        "scale": 1.0,
        "sigma": 5.0,
        "instance_crop": true,
        "bounding_box_size": 0,
        "min_crop_size": 0,
        "negative_samples": 0
    },
    "labels_filename": null,
    "run_name": "",
    "save_dir": null,
    "best_model_filename": null,
    "newest_model_filename": null,
    "final_model_filename": null
}

This first prints this exception after selecting the file from the GUI:

Traceback (most recent call last):
  File "d:\sleap\sleap\gui\active.py", line 105, in <lambda>
    self.form_widget.valueChanged.connect(lambda: self.update_gui())
  File "d:\sleap\sleap\gui\active.py", line 218, in update_gui
    paf_job, _ = self._get_current_job(ModelOutputType.PART_AFFINITY_FIELD)
  File "d:\sleap\sleap\gui\active.py", line 246, in _get_current_job
    job_filename, job = self.job_options[model_type][idx]
IndexError: list index out of range

Which doesn't prevent the profile from being selected or subsequently used for training. After going through, this is what happens:

Use tf.cast instead.                                                                           
INFO:sleap.nn.training:Closing the reporter controller/context.                                
INFO:sleap.nn.training:Closing the training controller socket/context.                         
multiprocessing.pool.RemoteTraceback:                                                          
"""                                                                                            
Traceback (most recent call last):                                                             
  File "C:\Anaconda3\envs\sleap\lib\multiprocessing\pool.py", line 119, in worker              
    result = (True, func(*args, **kwds))                                                       
  File "d:\sleap\sleap\nn\training.py", line 356, in train                                     
    workers=multiprocessing_workers,                                                           
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrap
per                                                                                            
    return func(*args, **kwargs)                                                               
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1418, in fit_
generator                                                                                      
    initial_epoch=initial_epoch)                                                               
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\engine\training_generator.py", line 40,
 in fit_generator                                                                              
    model._make_train_function()                                                               
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 509, in _make
_train_function                                                                                
    loss=self.total_loss)                                                                      
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrap
per                                                                                            
    return func(*args, **kwargs)                                                               
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\optimizers.py", line 501, in get_update
s                                                                                              
    self.updates.append(K.update(vhat, vhat_t))                                                
  File "C:\Anaconda3\envs\sleap\lib\site-packages\keras\backend\tensorflow_backend.py", line 97
3, in update                                                                                   
    return tf.assign(x, new_x)                                                                 
  File "C:\Anaconda3\envs\sleap\lib\site-packages\tensorflow\python\ops\state_ops.py", line 224
, in assign                                                                                    
    return ref.assign(value, name=name)                                                        
AttributeError: 'Tensor' object has no attribute 'assign'                                      
"""                                                                                            
                                                                                               
The above exception was the direct cause of the following exception:                           
                                                                                               
Traceback (most recent call last):                                                             
  File "d:\sleap\sleap\gui\active.py", line 337, in run                                        
    with_tracking = with_tracking)                                                             
  File "d:\sleap\sleap\gui\active.py", line 617, in run_active_learning_pipeline               
    trained_jobs = run_active_training(labels, training_jobs, save_dir)                        
  File "d:\sleap\sleap\gui\active.py", line 692, in run_active_training                        
    result.get()                                                                               
  File "C:\Anaconda3\envs\sleap\lib\multiprocessing\pool.py", line 644, in get                 
    raise self._value                                                                          
AttributeError: 'Tensor' object has no attribute 'assign'

Oddly, this training profile for PAFs using UNet does work and trains successfully:

{
    "model": {
        "output_type": 1,
        "backbone": {
            "down_blocks": 3,
            "up_blocks": 3,
            "convs_per_depth": 0,
            "num_filters": 16,
            "kernel_size": 5,
            "upsampling_layers": true,
            "interp": "bilinear"
        },
        "skeletons": null,
        "backbone_name": "UNet"
    },
    "trainer": {
        "val_size": 0.15,
        "optimizer": "adam",
        "learning_rate": "5e-05",
        "amsgrad": true,
        "batch_size": 2,
        "num_epochs": 150,
        "steps_per_epoch": 200,
        "shuffle_initially": true,
        "shuffle_every_epoch": true,
        "augment_rotation": 180,
        "augment_scale_min": 1.0,
        "augment_scale_max": 1.0,
        "save_every_epoch": false,
        "save_best_val": true,
        "reduce_lr_min_delta": "1e-06",
        "reduce_lr_factor": 0.5,
        "reduce_lr_patience": 8,
        "reduce_lr_cooldown": 3,
        "reduce_lr_min_lr": "1e-10",
        "early_stopping_min_delta": "1e-08",
        "early_stopping_patience": 30.0,
        "scale": 1.0,
        "sigma": 5.0,
        "instance_crop": true,
        "bounding_box_size": 0,
        "min_crop_size": 0,
        "negative_samples": 0
    },
    "labels_filename": null,
    "run_name": "",
    "save_dir": null,
    "best_model_filename": null,
    "newest_model_filename": null,
    "final_model_filename": null
}

So I guess it's an issue with the kernel_size parameter. I think this defaults to 0 when using the LEAP CNN since it's not an attribute of the model. Maybe some parameter validation that's model-specific would be useful?

Unclear why this results in a GUI error though. And how the model manages to get constructed in the first place?

Filename of video in videos table should be selectable/copyable (read-only?)

Use case: copying filename of video and/or going to the folder manually

Predicting on suggestions using models from another project

Steps to reproduce:

Open labels project at: M:\talmo\wt_gold_labeling\pilot_test_0910.json
If necessary, just create a new project with these videos:

M:/junyu/data/pair/wt/190612_110405_wt_16276625_rig2.1/000000.mp4
M:/junyu/data/pair/wt/190612_110405_wt_18159111_rig2.2/000000.mp4

And this skeleton:

M:\talmo\wt_gold_labeling\skeleton_30pts.json

The mount is: M: <-> \\tigress-cifs.princeton.edu\fileset-mmurthy <-> /tigress/MMURTHY

Predict -> Expert Controls and select the following models:

M:\Brandon\sLEAP\models\190911_100301.confmaps.UNet.n=848.json
M:\Brandon\sLEAP\models\190911_105618.pafs.LeapCNN.n=848.json
M:\Brandon\sLEAP\models\190812_144937.centroids.UNet.n=399.json

Generate some suggestions for Predict On (or just use random frames).

Inference should start running, but first bug:

Crash when serializing to the temp(?) inference.h5 file if the models/ subfolder does not exist. Save to temp folder using tempdir or make sure the models/ subfolder exists.

If creating the subfolder manually, then:

No crash after inference, console shows instances are detected but nothing appears in GUI on any of the suggestion frames
Saving the project results in exception:

Traceback (most recent call last):
  File "d:\sleap\sleap\gui\app.py", line 1314, in saveProject
    compress = compress)
  File "d:\sleap\sleap\io\dataset.py", line 803, in save_json
    d = labels.to_dict()
  File "d:\sleap\sleap\io\dataset.py", line 720, in to_dict
    dicts['labels'] = label_cattr.unstructure(self.labeled_frames)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 139, in unstructure
    return self._unstructure_func.dispatch(obj.__class__)(obj)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in _unstructure_seq
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in <genexpr>
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 204, in unstructure_attrs_asdict
    rv[name] = dispatch(v.__class__)(v)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in _unstructure_seq
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in <genexpr>
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "d:\sleap\sleap\instance.py", line 696, in unstructure_instance
    for field in attr.fields(x.__class__)
  File "d:\sleap\sleap\instance.py", line 697, in <dictcomp>
    if field.name not in ['_points', 'frame']}
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 139, in unstructure
    return self._unstructure_func.dispatch(obj.__class__)(obj)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in _unstructure_seq
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in <genexpr>
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "d:\sleap\sleap\io\dataset.py", line 700, in <lambda>
    label_cattr.register_unstructure_hook(Node, lambda x: str(self.nodes.index(x)))
ValueError: Node(name='head', weight=1.0) is not in list
Traceback (most recent call last):
  File "d:\sleap\sleap\gui\app.py", line 1314, in saveProject
    compress = compress)
  File "d:\sleap\sleap\io\dataset.py", line 803, in save_json
    d = labels.to_dict()
  File "d:\sleap\sleap\io\dataset.py", line 720, in to_dict
    dicts['labels'] = label_cattr.unstructure(self.labeled_frames)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 139, in unstructure
    return self._unstructure_func.dispatch(obj.__class__)(obj)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in _unstructure_seq
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in <genexpr>
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 204, in unstructure_attrs_asdict
    rv[name] = dispatch(v.__class__)(v)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in _unstructure_seq
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in <genexpr>
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "d:\sleap\sleap\instance.py", line 696, in unstructure_instance
    for field in attr.fields(x.__class__)
  File "d:\sleap\sleap\instance.py", line 697, in <dictcomp>
    if field.name not in ['_points', 'frame']}
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 139, in unstructure
    return self._unstructure_func.dispatch(obj.__class__)(obj)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in _unstructure_seq
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "C:\Anaconda3\envs\sleap\lib\site-packages\cattr\converters.py", line 225, in <genexpr>
    return seq.__class__(dispatch(e.__class__)(e) for e in seq)
  File "d:\sleap\sleap\io\dataset.py", line 700, in <lambda>
    label_cattr.register_unstructure_hook(Node, lambda x: str(self.nodes.index(x)))
ValueError: Node(name='head', weight=1.0) is not in list

Visualize model doesn't update video source

If you visualize model outputs for one video and then change the video you're looking at, it's still running the models on frames from the original video.

Gaussian blur on confmaps after upsampling

On multiscale models that output tiny confmaps, if we blur the confmaps we aren't finding peaks. For now I've disabled blur if upsample_factor is set. If we're upsampling, it would probably be better if we apply the blur after the cubic resize.

code in sleap/nn/peakfinding_tf.py

test case:

python sleap/nn/inference.py /tigress/MMURTHY/Brandon/FoxPdata/video/CS-males/190701_102018_18159211_20190701_102018/000000.mp4 -m /tigress/MMURTHY/Brandon/sLEAP/models/190812_144937.centroids.UNet.n=399.json -m /tigress/MMURTHY/Brandon/sLEAP/models/190812_135728.confmaps.UNet.n=794.json -m /tigress/MMURTHY/Brandon/sLEAP/models/190812_141241.pafs.LeapCNN.n=794.json --frame 123

Error handling when saving

Alert user in gui when a save fails. (This was happening with a file where the user didn't have write permissions on a file that I had generated on the cluster.)

update track spawn frame used by gui

currently gui uses track.spawned_on which doesn't get updated (e.g., when user removes instances from a given track). we should instead use Labels.get_track_occupany(video) since this is updated.

Toggleable options for training progress monitor

Checkboxes next to the Stop Training button for:

Log scale y-axis
Show all batches (vs recent batches)
Ignore outliers in chart scaling

Frame index in status bar disappears when opening active learning/expert controls dialogs

Better random node initialization

Explicit options when initializing (from scratch or when creating from predicted instance):

Copy from nearest frame
Random
Mean pose (average of pose coordinates for existing Instances relative to centroid; centroid in random location)
Force directed

Developer Guide adherence

Reference: Developer Guide

Tasks:

Reformat all code using black
Ensure full compliance with naming and docstring conventions (a linter like yapf might help for the first pass)
Set up coveralls + coverage.py + coveralls-python for AppVeyor.

Make divisibility a parameter stored in training job/model config like "scale"

https://github.com/murthylab/sleap/blob/ff6119056118bbf7d42c0e6d0d5714a3eb6dcef9/sleap/nn/datagen.py#L103-L105
https://github.com/murthylab/sleap/blob/0f5f6f6cbb2f0d57de8f998628ef46776db78324/sleap/gui/overlays/base.py#L31
https://github.com/murthylab/sleap/blob/ff6119056118bbf7d42c0e6d0d5714a3eb6dcef9/sleap/nn/inference.py#L242-L246

divisibility factor should be 2 ** down_blocks

Track identity swap

If the names of the tracks are changed so that the alphabetic order changes, then the track assignments get messed up during save/open. Maybe we're using index from before alphabetic re-order to access track list after re-order? It appears that bug only occurs when saving/loading labels in h5 format.

Memory leak in tf peak finding

I0912 01:07:02.182041 47450918620800 inference.py:624]   Inferred confmaps and found-peaks (gpu) [19.1s]

^ this line starts to increase in runtime, very noticeably for longer inference jobs

Relevant block:
https://github.com/murthylab/sleap/blob/ff6119056118bbf7d42c0e6d0d5714a3eb6dcef9/sleap/nn/inference.py#L607-L634

Selected instance mismatch between video and table

After turning a predicted instance into a regular instance, the instance selected in the video was different than the instance selected in the table.

Multi-scale inference bug when importing from saved models

Scale not being adjusted properly when using the new TF-based subpixel peak finder?

Suggestion sorting based on inference scores/metrics

Add suggestion sorting for suggested frames with predictions. Can just be mean score of all predicted instances.

In GUI: extra column in suggestions that is sortable.

This also enables a random --> predict --> sort by worst score type of workflow.

Unify Predictor inference modes

All modes should share a common interface to GPU-based peak finding
Image NMS vs global argmax should be a generic flag
Loading should happen asynchronously from downstream processing when possible
For modes with multiple models, a single TF execution graph should be built to avoid multiple transfers of the same data

points() should just be property method.

Exclude empty tracks from track occupancy h5

write_tracking_h5 currently includes all tracks, regardless of whether they have any instances

Refactor PAF matching

Decouple line integrals from matching stages. Line integrals could probably be trivially parallelized also.
Cover cases of disjoint subgraphs, cycles, etc
Tests

Enhancement:

Check for node centrality or use best scores to determine how to walk through the graph?

import sleap as slp

slp.Labels
slp.Video  # also subclasses? or maybe just through class/static methods?
slp.LabeledFrame
slp.Instance
slp.PredictedInstance
slp.Skeleton

Also inference, though Predictor might need some refactoring to be practical

Demo notebooks

Data input formats and I/O (basically demonstrating how to work with Video class and Labels for programmatic data import)
Interactive (synchronous) training (using Predictor in a notebook)
Programmatic inference and saving out data usable for analysis. Perhaps also with visualization if it's notebook compatible

Labeled frames counter in status bar

For current video and total for current Labels dataset

Proofreading line plots of auxiliary measures

These are intended to be displayed above the trackbar and used to guide manual proofreading

Sum/max of total displacement of all points from frame to frame between instances of the same track
Sum/min of instance matching scores
Sum/min of track assignment scores
Sum/min of predicted point confidences
Count of predicted points