CRF-MA is an extension of the Java implementation of Conditional Random Fields (CRFs) available in the Mallet toolbox in order to handle multiple annotators. CRF-MA uses the Expectation-Maximization algorithm to jointly learn the CRF model parameters, the relia- bility of the annotators and the estimated ground truth. When it comes to performance, the proposed method (CRF-MA) significantly outperforms typical approaches such as majority voting. See the original paper for further details:

Rodrigues, F. and Pereira, F.C. and Ribeiro, B., Sequence labeling with multiple annotators, Machine Learning, Springer, 2013.

The tar.gz with the source code can be obtained here.

The datasets (including the Amazon’s Mechanical Turk data) used are also available here for download.

DOWNLOAD:

Source code // Datasets

CONTACT:

Please send questions and comments to fmpr [at] dei.uc.pt