Skip to content

Incentive.md

Latest commit

 

History

History
145 lines (90 loc) · 7.62 KB

Incentive.md

File metadata and controls

145 lines (90 loc) · 7.62 KB

Incentive Mechanism

Benchmark Runs

Submitted discriminator miners are evaluated against a subset of the data sources listed below. Models are evaluated on cloud infrastructure -- miners do not need to host hardware for inference. A portion of the evaluation data comes from generative miners, who are rewarded based on their ability to submit data that both pass validator sanity checks (prompt alignment, etc.) and fool discriminators in benchmark runs.

Each modality (image, video, audio) is scored independently using the sn34_score metric, which combines discrimination accuracy (MCC) with calibration quality (Brier score).

Evaluation Datasets

Benchmark datasets are regularly expanded. Each modality includes a mix of real, synthetic, and semi-synthetic content from diverse sources (including continuously-updated GAS-Station data from generative miners).

Public datasets (available for training via gasbench):

Holdout datasets: In addition to the public datasets above, each benchmark round includes holdout datasets that are not publicly available during the round. Holdout data is critical to ensure models generalize well and to mitigate overfitting. At the end of each round, many of the holdout datasets are released and added to the public gasbench datasets for future training. Some holdout datasets cannot be released publicly due to licensing or other restrictions.

Generative Models

The following models are run by validators to produce a continual, fresh stream of synthetic and semisynthetic data. The outputs of these models are uploaded at regular intervals to public datasets in the GAS-Station Hugging Face org for miner training and evaluation.

Text-to-Image Models

Image-to-Image Models

Text-to-Video Models

Image-to-Video Models

Generator Rewards

The generator incentive mechanism combines two components: a base reward for passing data validation checks, and a multiplier based on adversarial performance against discriminators.

Base Reward (Data Validation)

Generators receive a base reward based on their data verification pass rate:

$$R_{\text{base}} = p \cdot \min(n, 10)$$

Where:

  • $p$ = pass rate (proportion of generated content that passes validation)
  • $n$ = number of verified samples (min(n, 10) creates a rampup of incentive for the first 10 samples)

Fool Rate Multiplier (Adversarial Performance)

Generators earn additional rewards by successfully fooling discriminators. The multiplier is calculated as:

$$M = \max(0, \min(2.0, f \cdot s))$$

Where:

  • $f$ = fool rate = $\frac{N_{\text{fooled}}}{N_{\text{fooled}} + N_{\text{not fooled}}}$
  • $s$ = sample size multiplier

The sample size multiplier encourages generators to be evaluated on more samples, similar to the sample size ramp used in the base reward.

$$s = \begin{cases} \max(0.5, \frac{c}{20}) & \text{if } c < 20 \\ \min(2.0, 1.0 + \ln(\frac{c}{20})) & \text{if } c \geq 20 \end{cases}$$

Where:

  • $c$ = total evaluation count (fooled + not fooled)
  • Reference count of 20 gives multiplier of 1.0
  • Sample sizes below 20 are penalized
  • Sample sizes above 20 receive logarithmic bonus up to 2.0x

Final Generator Reward

The total generator reward combines both components:

$$R_{\text{total}} = R_{\text{base}} \cdot M$$

This design incentivizes generators to:

  1. Produce high-quality, valid content (base reward)
  2. Create adversarially robust content that can fool discriminators (multiplier)
  3. Participate in more evaluations for sample size bonuses

Discriminator Rewards

Scoring: sn34_score

Each discriminator model is scored per modality using the sn34_score, which combines two metrics:

  1. Binary MCC (Matthews Correlation Coefficient) -- measures how well the model discriminates between real and synthetic content. Ranges from -1 (worst) to +1 (perfect).

  2. Brier Score -- measures calibration quality (how well predicted probabilities match actual outcomes). Ranges from 0 (perfect) to 0.25 (random baseline).

These are combined as follows:

$$MCC_{norm} = \left(\frac{MCC + 1}{2}\right)^{\alpha}$$

$$Brier_{norm} = \left(\frac{0.25 - Brier}{0.25}\right)^{\beta}$$

$$sn34_{score} = \sqrt{MCC_{norm} \cdot Brier_{norm}}$$

With default parameters $\alpha = 1.2$ and $\beta = 1.8$. The geometric mean penalizes models that are strong on one axis but weak on the other -- a model must be both accurate and well-calibrated to score highly.

Competition Rounds

The discriminator competition is organized into rounds. Each round introduces new benchmark datasets and evaluates all submitted models. Winners are determined per modality (image, video, audio) independently.

How Rounds Work

  1. New round begins: Benchmark datasets are updated (new GAS-Station data, potentially new static datasets). All modalities share the same benchmark version number.
  2. Models are benchmarked: All submitted discriminator models are evaluated against the current round's datasets and scored using sn34_score.
  3. Winner determined per modality: The highest-scoring model for each modality wins that round.
  4. Alpha reward: The round winner for each modality receives an alpha reward.

Winner-Take-All Per Round

Each round is winner-take-all -- only the top-scoring discriminator for each modality receives the alpha reward for that round. This incentivizes miners to continuously improve their models and push the state of the art in AI-generated content detection.

Rounds progress as benchmark versions are incremented, ensuring that models are always evaluated against fresh, evolving data.