Skip to content

Revise annotator count and dataset release dates #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
metabolean5 wants to merge 1 commit into
base: gh-page
Choose a base branch
from
+3 −3
Open

Revise annotator count and dataset release dates #185

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions _editions/2026/tasks/enthymeme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Given a tweet, determine whether it contains an implicit premise, an implicit co

An **implicit premise** is a supporting assumption left unstated that the argument relies on. An **implicit conclusion** is a claim that follows from the stated premises but is never explicitly made. When neither component is missing, the label is `none`.

Tweets in the train and dev sets are each annotated by five independent annotators; those in the test set by three. Individual annotator labels — prior to any majority vote — are provided alongside the data, making it possible to treat disagreement as signal rather than noise.
Tweets in the train and dev sets are each annotated by three independent annotators; those in the test set by three. Individual annotator labels — prior to any majority vote — are provided alongside the data, making it possible to treat disagreement as signal rather than noise.

Participants are invited to complete two tasks. While they may choose to complete only task 1, completion of task 2 is conditional upon prior completion of task 1.

Expand Down Expand Up @@ -80,8 +80,8 @@ Participants are encouraged to visit the [Enthymemes Dataset Portal](https://tur
The data will be released in three parts:

- **Data sample (1 March):** A small collection of tweets that have been annotated by two annotators, so that participants can read the data and understand the challenge of the task.
- **First data release (Mid-March):** A larger collection of data that has been annotated by five annotators. It is split into train and dev.
- **Final dataset (Beginning April):** The full dataset, which is split into train, dev, and test set. The train set is a superset of the training set released with the first data release. Likewise, the dev set is a superset of the development set released with the first data release. Train and dev data has been annotated by five annotators. The test set is the test set for the task: the participants are required to submit their predictions on the test set.
- **First data release (Mid-March):** A larger collection of data that has been annotated by three annotators. It is split into train and dev.
- **Final dataset (Mid-May):** The full dataset, which is split into train, dev, and test set. The train set is a superset of the training set released with the first data release. Likewise, the dev set is a superset of the development set released with the first data release. Train and dev data has been annotated by five annotators. The test set is the test set for the task: the participants are required to submit their predictions on the test set.

> ⚠️ Participants should be aware that the data contains language hurtful towards immigrants and should be ready for this when reading the data.

Expand Down