ClimateModernBERT

Preprint manuscript · under review

ClimateModernBERT

Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

A family of climate-domain ModernBERT encoders. We continue-pretrain ModernBERT-Base on academic climate text, climate-filtered web data, and synthetic climate text, then ask which corpus composition — and which integration mechanism — actually helps.

Paper (PDF) Models on Hugging Face GitHub Explore models →
21
adapted variants7 Phase-1 · 7 Phase-2 · 7 merged
9
climate NLP benchmarksclassification, multi-label, retrieval
810+
fine-tuning runsn=3 seeds, n=10 for the synthetic ablation
6.42B
pretraining tokensacross three corpora

Key findings

Three results from the study

Each is stated as the manuscript states it — scoped to climate NLP, to nine benchmarks, and to the ModernBERT-Base encoder family.

  1. 01
    75.3 avg F1 · academic only

    Academic text is the strongest signal

    Academic-only adaptation reaches the highest average F1 of any jointly trained configuration. Gains concentrate on register-sensitive tasks — Sentiment +6.3, WXImpactBench +7.5, Commitments +5.9.

  2. 02
    −1.2 avg F1 · 𝒜 → 𝒜+𝒮+ℱ

    More sources are not better sources

    Adding web and synthetic text to the academic corpus lowers average F1 from 75.3 to 74.1. Synthetic text is the clearest case: it significantly helps TCFD and Specificity while consistently degrading Commitments & Actions.

  3. 03
    76.3 avg F1 · θSoup

    Merge afterwards, don't mix upfront

    Averaging three independently adapted checkpoints beats joint training on the very same corpus union. Their task vectors are near-orthogonal (pairwise cosine 0.06–0.18), so averaging keeps what each source learned.

Method

One base encoder, three corpora, two ways to combine them

Every variant starts from the same ModernBERT-Base checkpoint and sees the same climate text. What changes is where the sources are combined — in the training data, or in parameter space afterwards.

ModernBERT-Base 150M parameters · 8,192-token context · 4× A100
A Academic ~1.28B tokens · ~5M docs
F Climate Web ~5B tokens · 2.59M docs
S Synthetic ~0.14B tokens · ~20K docs
I

Joint continued pretraining

One training run over a mixture of corpora. All seven non-empty subsets are trained.

mixed corpus one CPT run θ{𝒜,𝒮,ℱ}

Full union: 74.8 avg F1

II

Parameter-space merging

One checkpoint per corpus, trained independently, then combined by weight averaging.

θ𝒜 θℱ θ𝒮 θSoup

Same data: 76.3 avg F1

Each configuration runs in two stages, following ModernBERT's own schedule. Phase 1 · CX context extension at a constant learning rate, then Phase 2 · CX_LRD learning-rate-decay specialization. The Phase-2 checkpoints are the paper's primary models; Phase-1 checkpoints are kept as an ablation isolating what the decay stage adds.

The ClimateModernBERT adaptation framework: three climate corpora feed either joint continued pretraining or independent continued pretraining followed by a parameter-space merge, on a two-phase schedule.
Figure 1 from the manuscript.

Results

Where the gains come from

Average F1 over nine climate NLP benchmarks, mean of three fine-tuning seeds under one shared recipe. Every figure is transcribed from the manuscript.

  • ClimateBERT prior climate encoder
    72.1
  • ModernBERT-base θbase · initialization
    73.5
  • Joint 𝒜+𝒮+ℱ Phase 2 · full corpus union
    74.8
  • Best single source 𝒜 Phase 1 · academic only
    75.3
  • θSoup weight-averaged merge
    76.3
Dashed line marks θbase, the checkpoint every variant is adapted from. Axis truncated to 71–77.5.
Model Retr.Comm.Det.Spec.Sent.NetZ.TCFDWFBWXI Avg.
Baselines
ModernBERT-base (θbase, stable-phase) The initialization used for every adapted variant. 86.7±1.9 65.9±1.3 94.5±0.3 67.3±0.7 71.8±4.1 98.7±0.6 60.1±0.7 96.0±0.4 20.4±5.0 73.5±1.7
ClimateBERT RoBERTa-based climate-adapted encoder, fine-tuned under the identical protocol. 82.4±2.6 70.9±0.0 97.5±0.0 67.8±0.0 76.6±0.0 99.1±0.0 51.4±0.1 92.5±0.0 3.3±4.6 72.1±0.8
Climate-ModernBERT · Phase 1 + Phase 2
𝒜 81.1±4.1 68.7±2.1 93.6±0.2 67.3±1.1 77.1±0.7 98.0±1.2 61.3±0.5 96.9±0.3 25.2±3.2 74.4±2.4
𝒮 84.5±2.1 66.4±3.4 93.8±0.2 69.4±0.9 74.8±2.7 99.0±0.1 61.1±0.2 96.5±0.3 25.0±2.4 74.5±1.4
ℱ 85.0±2.5 67.4±1.6 95.8±0.0 68.7±1.9 77.5±0.4 98.7±0.0 60.3±1.7 96.8±0.2 20.3±4.4 74.5±1.4
𝒜 + 𝒮 83.2±4.1 66.1±1.0 93.2±0.8 69.5±1.0 78.1±0.5 98.9±0.2 61.3±1.0 96.2±0.4 24.3±4.9 74.5±1.5
𝒜 + ℱ 83.5±4.4 69.5±0.6 95.5±0.6 70.2±0.7 76.0±0.4 99.1±0.1 63.0±2.5 97.0±0.5 18.5±5.0 74.7±1.6
𝒮 + ℱ 85.3±2.2 62.2±2.3 95.7±0.2 71.0±0.8 75.5±0.6 97.9±0.3 59.4±2.2 96.9±0.5 23.4±3.7 74.1±1.4
𝒜 + 𝒮 + ℱ 86.8±1.4 60.0±2.3 95.7±0.0 71.8±1.4 76.0±1.2 98.6±0.4 62.8±2.2 97.5±0.2 23.9±5.9 74.8±1.7
Table 2, upper block. Bold marks the column leader within this table.

Which corpus matters most

Average F1 change of a two-source Soup relative to the full Soup({𝒜, 𝒮, ℱ}).

  • without Academic -4.0
  • without Synthetic -1.5
  • without Climate Web -1.5

Removing 𝒜 costs four times as much as removing either other corpus — even though ℱ produces by far the largest parameter update.

The nine benchmarks

Six single-label classification tasks, two multi-label, one retrieval benchmark cast as binary relevance.

  • Det. Climate Detection Binary
  • Spec. Climate Specificity Binary
  • Comm. Commitments & Actions Binary
  • Sent. Climate Sentiment Multi-class
  • NetZ. Net Zero & Reduction Multi-class
  • TCFD TCFD Recommendations Multi-class
  • WFB WFB Nature Multi-label
  • WXI WXImpactBench Multi-label
  • Retr. ClimRetrieve Retrieval

Model merging

Combining sources after training, not before

Joint training makes one optimizer reconcile three distributions at once. Merging lets each corpus produce its own checkpoint first. Because those updates are nearly orthogonal, the combination keeps more of what each source contributed.

Soupbest

mean(θ𝒜, θℱ, θ𝒮)

Uniform weight averaging. No hyperparameter, and the best result here.

Task Arithmetic

θbase + λ · Σ δi

Sum the task vectors and scale by λ ∈ {0.5, 1.0}.

TIES

trim → elect sign → merge

Drop small entries, resolve sign conflicts, average the rest. d ∈ {0.5, 0.7}.

DARE-TIES

drop & rescale → TIES

Randomly drop and rescale before TIES, assuming task vectors are redundant.

ClimateModernBERT · Soup

Uniform average of the three single-source Phase-2 checkpoints, verified from the repo's own mergekit_config.yml. Gains are largest on Commitments (+7.0) and WXImpactBench (+9.3).

76.3 average F1 Merge_Soup_LRD

Model explorer

56 checkpoints, named by what they contain

The paper's checkpoints are republished under CMB-ClimateModernBERT with names that read as the paper's own notation — corpora in A_S_F order, then the training stage. The original sraj/* repositories are untouched, so every existing link still works.

Corpus
Training
Type
Status

9 checkpoints are marked unconfirmed — their mapping to a manuscript configuration could not be established from the paper, the collection and the merge configs alone. 8 are superseded; nothing has been deleted from the Hub. Reasoning in the naming guide.

Paper

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

Under review Unpublished — no venue, DOI or arXiv identifier yet, and no citation to give. The author list is anonymized for review.

We introduce Climate-ModernBERT, a family of climate-adapted encoders obtained by continued pretraining of ModernBERT-Base on academic climate text, climate-filtered web data, and synthetic climate documents. Comparing joint continued pretraining against parameter-space merging of independently specialized checkpoints across nine climate NLP benchmarks, the best model reaches 76.3 average F1 — 2.8 points over the baseline. Academic corpora provide the strongest adaptation signal, and merging outperforms joint multi-source training.

Read the PDF Code & pipelines