Introduction

When a company wishes to outsource its data annotation, it wants to be sure that the annotation will be carried out with a high level of quality and at a competitive price. If the need for annotation is recurrent, it is advantageous to look for a long-term partner.

The Importance of a POC

Because data annotation isn’t always quick and easy (see blog post on the subject), it’s crucial to take the necessary time at the start of the project to identify any potential difficulties, whether technical, functional or organizational.

That’s why we advise our customers to set up a POC (“Proof of Concept”) beforehand. A POC allows you to experiment with a realistic but limited dataset and identify potential questions or issues.

The POC is different from a 15-hour free test, more suited to simple annotation projects.

When to Use a POC

Two dimensions to consider are the project’s complexity and the recurrence of annotation projects within the company. Here is a table of the most suitable offers according to the context:

Advantages of a POC

A POC primarily helps to:

• Validate the choice of annotation tool. ✅
• Identify potential performance issues (e.g., large image or video files) and solutions. 🚀
• Validate the annotation method best suited to the project’s objectives, set up the annotation process inside the tool (interface, roles, review, …) and outside the tool (overall project management). ⚙️
• Identify and clarify ambiguities in instructions, update instructions based on the responses provided. 📝

It also helps validate the most effective communication method, quality KPIs (Key Performance Indicators), and convince internal stakeholders of the project’s usefulness for machine learning algorithms.

POC Execution

The POC is conducted with a small but representative team of annotators. It includes project setup on the annotation tool, annotation, review, and customer care (close communication with the client, management of Q&A and progress files).

The “Start Slow” Principle

The POC is based on the “start slow” principle, which is often neglected but essential. After the POC, a precise project plan is defined: team size, required skills, speed and quality objectives, client contacts, project meeting frequency, etc. The project can then proceed in “production” mode.

Risks of Ignoring a POC

Some companies are reluctant to conduct a paid POC. However, estimating annotation time on a complex project is challenging, and without a POC, the risks include:

• Paying too much due to excessive safety margins taken by annotation companies. 💸
• Selecting an unsuitable partner who cannot complete the project. 🙅‍♂️
• Facing price reevaluation after thorough testing. 📈

Conclusion

In conclusion, time spent on a POC ultimately saves time on the project, increases overall annotation quality, and maximizes the chances of success.