Data labeling is a hot topic as a recent report from AI research and advisory firm Cognilytica found that over 80% of the time enterprises spend on AI projects goes toward preparing, cleaning and labeling data.

A new category of tools has emerged, called data labeling platforms, training data management platforms. Three years ago, this market did not exist. Today, the offer is plentiful and it is sometimes complicated to find one's way.

A good annotation tool must meet three needs:

• annotate quickly (productivity of annotators)
• annotate well (control the quality / consistency of annotations)
• annotate the right data (source diversity in the dataset)

Here is an attempt to do a fair comparison of data labeling / training data platforms.

I tried to be as accurate as possible, based on my experience with those solutions and their documentation. If you see any mistake (this is likely), please create an issue / push a PR.

• Tasks / Tools by Asset Type
  • Image
  • Text
  • PDF
  • Video
  • Audio
  • Timeserie
• Productivity Management Features
• Quality Management Features
• Dataset Management Features
• API
• Deployment
• Automation
• Workforce
• Security

The criteria are as follows:

• Classification
• Hierarchical classification
• Bounding Box
• Bounding Box
• Polygon
• Polyline
• Pose estimation
• Point
• Segmentation
• Vector Segmentation
• Image transcription / OCR
• Object relation
• Satellite imagery support
• High-resolution support
• DICOM support

The criteria are as follows:

• Classification
• Hierarchical classification
• Named entities recognition
• Relations extraction

The criteria are as follows:

• PDF document processing
• Bounding Box on PDF
• Object relation

The criteria are as follows:

• Classification
• Hierarchical classification
• Native Video/Frame
• Classification
• Object detection
• Transcription

The criteria are as follows:

• Voice transcription / Speech to text
• Diarization
• Ultrasound support

The criteria are as follows:

• Classification
• Range detection

The criteria are as follows:

• Interface Customization
• Asset preloading
• Keyboard shortcuts / Hotkeys
• Workload distribution
• Autosave
• Labeler productivity dashboard

The criteria are as follows:

• Issue management
• Consensus Image
• Consensus Video
• Consensus Text
• Consensus Audio

The criteria are as follows:

• Data hub
• Version control
• Public project
• ML export formats

How to programmatically interact with the tool. The criteria are as follows:

• GraphQL API
• Messaging bus API
• CLI SDK
• Python SDK

The different options to deploy the tool. The criteria are as follows:

• On-premise data
• On-premise deployment
• Air gap deployment

The offering in terms of labeling automation. The criteria are as follows:

• Model-assisted labeling
• Model based preannotation
• Queue prioritisation
• Model Training
• Model Inference

Availability of workforce to help labeling. The criteria are as follows:

• Human in the loop
• External workforce
• On-premise external workforce

Security certifications. The criteria are as follows:

• HIPAA Compliance
• SOC 2 Compliance