Benchmark report · updated 11 November 2026

The 800M run,
measured.

The completed 800M-token continuation improves every newstest2014 comparison against the preserved 350M full-data checkpoint. At 350M, ParaCrawl is the clearest data addition; Tatoeba helps mainly FR→EN. A fresh 800M ParaCrawl run also improves every cell. The gains are real; quality remains pilot-level.

Automatic metrics only No human evaluation Trained from scratch

Useful gist translation, not production quality.

Download · Windows 10/11 x64

Run the 800M-token model on your desktop.

The installer bundles the exported model and a Java runtime, so the app needs no Python, no separate model download and no network after install. It is a pilot: useful gist translation, not production.

Checking the latest release… All releases ↗

Latest Windows build

—

Checking… —

Reading the latest release from GitHub…

No public Windows release yet.

The installer is built and attached to a draft release before it goes public. Reload this page once that release is published to see the download.

View releases · Build workflow

Couldn't read the latest release.

GitHub's API may be unavailable or rate-limited. Use the releases page to check the current download.

View releases

The latest release has no Windows installer.

The release exists, but it does not attach Transcend-Translator-Setup.msi. Check the release page for the current state.

View releases

Installer facts

Installer SHA-256
—
Download
—
Installed
≈ 360 MB
Requirements
Windows 10/11 x64 · no admin
Offline
Model included; no network after install

The SHA-256 is read from the release asset metadata. The installer is unsigned; verify the hash and follow the SmartScreen steps below.

Install

  1. Download the MSI above.
  2. Run it. If SmartScreen appears, choose More info → Run anyway.
  3. No administrator is required: the installer is per-user and adds a Start-menu shortcut under %LOCALAPPDATA%\Transcend Translator.
  4. Launch it. The model loads locally; after installation the app works offline.

SmartScreen, honestly

This build is not code-signed, so Windows may show “Windows protected your PC” or “Unknown publisher”. That is expected for an unsigned pilot, not a detection.

Choose More info → Run anyway. Smart App Control on a fresh Windows 11 install may block unsigned installers entirely; a signed build is the permanent fix.

Verify and uninstall

In PowerShell, compare the download with the SHA-256 above:

Get-FileHash .\Transcend-Translator-Setup.msi -Algorithm SHA256

To uninstall, open Settings → Apps → Installed apps → Transcend Translator.

Model provenance

Model
19,825,920 parameters · 6+6 layers · 8,000-token byte-BPE
Training
800M-token continuation · dev-selected step 77,000 of 91,691
Data
Europarl v7 + WMT14 News Commentary v9
Export
11-file exported package · 237,211,735 B · encoder SHA-256 a3110fb2…4539

Model card · corpus terms · pilot quality, not production.

EN → FRBLEU
26.44
+2.36 vs 24.08

0–30 · beam 4 · final

FR → ENBLEU
24.38
+2.23 vs 22.15

0–30 · beam 4 · final

EN → FRchrF++
52.53
+2.21 vs 50.32

0–60 · beam 4 · final

FR → ENchrF++
51.90
+2.02 vs 49.88

0–60 · beam 4 · final

01 / Comparison

Every aggregate metric improved.

Preserved 350M step 38,000 versus 800M final step 91,691 on exactly the same retained newstest2014 rows. Beam 4 is the dev-tuned setting; greedy remains the model default.

Read this first

BLEU in the mid-20s is a gist-level score, not a percentage of correct sentences. The model can carry the main idea of a straightforward news sentence; it still drops details and produces wrong words.

Headline and tables use the final step-91,691 checkpoint. The dev-selected step-77,000 checkpoint is slightly lower but also above the 350M baseline; the comparison report includes both.

Full newstest2014 scores; positive deltas favor the 800M final checkpoint.
Direction Decoder 350M BLEU 800M final BLEU Δ BLEU 350M chrF++ 800M final chrF++ Δ chrF++
EN→FRGreedy 22.65525.367+2.712 49.04551.391+2.346
EN→FRBeam 4 24.07926.440+2.361 50.31852.531+2.213
FR→ENGreedy 21.26023.558+2.298 48.75550.830+2.075
FR→ENBeam 4 22.14524.377+2.232 49.88251.902+2.020

Paired document bootstrap

1,000 resamples over 176 newstest2014 documents. Every interval excludes zero.

Direction Decoder ΔBLEU 95% CI ΔchrF++ 95% CI
EN→FRGreedy+2.712 [+2.298, +3.077]+2.346 [+2.106, +2.587]
EN→FRBeam 4+2.361 [+1.978, +2.752]+2.213 [+1.998, +2.451]
FR→ENGreedy+2.298 [+2.022, +2.580]+2.075 [+1.859, +2.282]
FR→ENBeam 4+2.232 [+1.904, +2.559]+2.020 [+1.801, +2.253]

Sentence-level is mixed

Descriptive chrF++ win/loss/tie counts, not a significance test.

Direction Decoder Wins Losses Ties
EN→FRGreedy1,851978130
EN→FRBeam 41,814937208
FR→ENGreedy1,766944212
FR→ENBeam 41,740948234

02 / Data ablation

Two data additions. ParaCrawl wins at both scales.

At 350M, each arm keeps the 19.8M-parameter model and the 350M-token budget, then adds about 99k everyday-language pairs to the same 2.1M-pair baseline. ParaCrawl improves every cell; Tatoeba's gain is FR→EN-specific. A fresh 800M ParaCrawl run now confirms the same direction at the larger budget.

Tatoeba-only

direction-specific

99,135 added pairs, mostly short conversational sentences. The EN→FR point estimates are positive but both 95% intervals cross zero. FR→EN clears the interval in greedy and beam 4.

  • FR→EN+0.543 BLEU / +0.423 chrF++ greedy; +0.336 / +0.351 beam 4. Both intervals exclude zero.
  • EN→FR+0.173 / +0.179 greedy; +0.145 / +0.001 beam 4. Not statistically significant.
  • Best step39,000; 350,000,282 tokens.

ParaCrawl-only

consistent gain

98,034 added pairs from a bounded, filtered web sample. Every one of the four cells improves on both BLEU and chrF++, and every paired chrF++ interval excludes zero.

  • EN→FR+1.313 BLEU / +1.017 chrF++ greedy; +1.141 / +0.882 beam 4.
  • FR→EN+1.228 / +0.922 greedy; +1.136 / +0.888 beam 4.
  • Best step40,000; 350,001,418 tokens.
Full newstest2014 scores versus the full-data 350M checkpoint (step 38,000). Positive deltas favor the data arm.
Arm Direction Decoder BLEU Δ BLEU chrF++ Δ chrF++ chrF++ 95% CI CI excludes 0
Tatoeba
+99,135 pairs
EN→FRGreedy 22.828+0.173 49.224+0.179 [−0.076, +0.435]No
EN→FRBeam 4 24.224+0.145 50.319+0.001 [−0.239, +0.230]No
FR→ENGreedy 21.803+0.543 49.178+0.423 [+0.184, +0.665]Yes
FR→ENBeam 4 22.481+0.336 50.233+0.351 [+0.118, +0.568]Yes
ParaCrawl
+98,034 pairs
EN→FRGreedy 23.968+1.313 50.062+1.017 [+0.770, +1.275]Yes
EN→FRBeam 4 25.220+1.141 51.200+0.882 [+0.651, +1.119]Yes
FR→ENGreedy 22.488+1.228 49.677+0.922 [+0.661, +1.174]Yes
FR→ENBeam 4 23.281+1.136 50.770+0.888 [+0.618, +1.146]Yes

Baseline BLEU / chrF++: EN→FR greedy 22.655 / 49.045, beam 4 24.079 / 50.318; FR→EN greedy 21.260 / 48.755, beam 4 22.145 / 49.882. Each arm is a separate from-scratch run to 350M target tokens, not a fine-tune of the baseline. The additions are about 4.7% more training pairs, so this is an equal-token comparison, not equal-data.

800M ParaCrawl scale-up

confirmed at 800M

A fresh 800M ParaCrawl run completed at 800,004,736 tokens and step 91,754; the dev-selected checkpoint is step 91,000. Against the 800M continuation baseline at step 77,000, all four cells improve and all paired chrF++ intervals exclude zero. The effect holds at both scales.

The baseline is a continuation of the 350M full-data run, while this candidate is a fresh 800M run. This is an equal-token comparison, not an identical-schedule comparison. The 350M combined arm remains untrained.

Full newstest2014 scores for the 800M ParaCrawl arm. Baseline is the 800M continuation at step 77,000; candidate is step 91,000. Positive deltas favor the candidate.
Direction Decoder Baseline BLEU / chrF++ ParaCrawl BLEU / chrF++ Δ BLEU Δ chrF++ chrF++ 95% CI
EN→FRGreedy 25.021 / 51.06526.174 / 52.144 +1.153+1.079 [+0.757, +1.388]
EN→FRBeam 4 26.277 / 52.24927.709 / 53.260 +1.432+1.011 [+0.743, +1.283]
FR→ENGreedy 23.330 / 50.60024.806 / 51.685 +1.476+1.085 [+0.843, +1.334]
FR→ENBeam 4 23.898 / 51.46225.458 / 52.565 +1.560+1.103 [+0.903, +1.324]

Same retained rows and decoders as the headline comparison: 2,959 EN→FR and 2,922 FR→EN segments, with 1,000 document-level bootstrap resamples. The candidate adds 98,034 ParaCrawl pairs to the baseline training mixture and was trained from random initialization.

03 / Training run

Same data, more tokens, same 19.8M model.

The 350M baseline trained on the complete prepared Europarl/News Commentary split. The 800M continuation resumed from it with the same 2.1M pairs and added 450M target tokens. This is a token-budget comparison, not a clean data-only ablation.

2.10Mtraining pairs
800Mtarget tokens
91,691updates at stop
17h 49mcumulative training time

Development curve

Greedy dev chrF++ by training step

Best dev 56.08 at step 77,000

Greedy development chrF++ over training (800M continuation) Line chart of greedy development chrF++ over 91 validation points. Mean chrF++ rises from 21.21 at step 1,000 to 56.08 at step 77,000, then 55.84 at step 91,000. EN to FR and FR to EN are shown separately. 20 30 40 50 55 0 20k 40k 60k 80k 100k Best mean dev chrF++ 56.0833 at step 77,000Last validation mean dev chrF++ 55.8353 at step 91,000 DEV CHRF++ TRAINING STEP

Validation on 300 dev rows per direction; not the newstest2014 test score. Y-axis begins at 20. Best mean was 56.08 at step 77,000; the last validation, at step 91,000, was 55.84.

Run Train pairs Target tokens Best step Final step Dev chrF++ Parameters
350M baseline 2,101,321350,004,440 38,00040,113 54.809919,825,920
800M continuation 2,101,321800,002,117 77,00091,691 56.083319,825,920

The headline evaluates models/scaled_full/checkpoint.pt, the final 800M checkpoint at step 91,691. The dev-selected best is models/scaled_full/best.pt at step 77,000. Both stopped on max_train_tokens.

04 / Method

One benchmark. Same rows. Automatic metrics.

The preserved 350M checkpoint, both 800M continuation checkpoints, the two completed 350M data arms, and the fresh 800M ParaCrawl arm were decoded on the same retained newstest2014 segments with the same references. Nothing was truncated silently.

Evaluation setup

  • Benchmarknewstest2014 (wmt14/full), both directions
  • Retained rows2,959 EN→FR; 2,922 FR→EN; 44 and 81 overlength sources skipped
  • Documents176 per direction; bootstrap resamples documents
  • DecodersGreedy and beam 4 (length penalty 1.2, minimum 16 new tokens)
  • MetricsSacreBLEU 2.5.1, 13a cased BLEU; chrF++ with word_order=2
  • Uncertainty1,000 paired bootstrap resamples per comparison
Metric signatures and checkpoint hashes

BLEU: nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.5.1

chrF++: nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.5.1

Preserved 350M checkpoint SHA-256:fee0f72cfc4556f03fd6820157cc9c530c41f0a6db67153f8185d1e92b813277

800M dev-selected checkpoint SHA-256:44f963e715121d4efd4746a8631a674808d899b3ff72cd6cb9871649ec68de86

800M final checkpoint SHA-256:6fe833dd7c359d62c01935b5d7f117e2c490a69b83decf3550da7af059741342

800M ParaCrawl checkpoint SHA-256:4e578110c728465a708b357708a5dbbb640180308aa86a4922a584bec088575e

Reproduce and inspect

The 800M continuation wrapper evaluates the preserved 350M checkpoint and both 800M checkpoints on identical rows. The data-arm wrappers evaluate their baselines and candidates with the same rows, references, decoders and bootstrap. The 800M ParaCrawl wrapper is scripts/run_paracrawl_800m.sh.

SKIP_TRAIN=1 bash scripts/run_800m_comparison.sh
Report trail
  • reports/newstest2014.scaled_full_800m.comparisons.md
  • reports/newstest2014.scaled_full_800m.comparisons.json
  • reports/newstest2014.scaled_full_800m.{greedy,beam4}.{en-fr,fr-en}.full.json
  • reports/newstest2014.tatoeba.comparisons.{json,md}
  • reports/newstest2014.paracrawl.comparisons.{json,md}
  • reports/newstest2014.paracrawl_800m.comparisons.{json,md}
  • reports/newstest2014.paracrawl_800m.{greedy,beam4}.{en-fr,fr-en}.full.json
  • models/data_ablation/paracrawl_800m/training_report.json
  • reports/data_expansion_full.md
  • models/scaled_full/training_report.json
  • reports/review_packet_scaled_full_20/ — 20-row source/reference packet; original outputs from 350M; human review pending

05 / What it looks like

Aggregate gains, visible errors.

Examples from the same deterministic 20-sentence review packet, decoded with the 800M final checkpoint. These are illustrative, not a random quality estimate or a human evaluation.

FR → ENregression
SourceLe Nevada a déjà mené à terme un projet pilote.
ReferenceNevada has already completed a pilot.
ModelNevada has already conducted a pilot project in the long term.

The gist survives, but “in the long term” is unsupported. This sample regressed against the 350M output.

EN → FRclear error
SourceNevada has already completed a pilot.
ReferenceLe Nevada a déjà mené à terme un projet pilote.
ModelLa Nevada a déjà achevé un pilier.

Wrong article/gender and a wrong noun (“pilier” instead of “pilote”). Not acceptable as written.

FR → ENmixed
SourceCela leur laisse le temps de faire certains tests et de les comprendre, et un délai avant d'inscrire « masculin » ou « féminin ».
ReferenceIt gives them the time to do some tests and figure it out and a period of time before they write 'male' or 'female.'
ModelThis leaves them time to carry out some tests and to understand them, and a deadline before they inscribe “masculin” or “feminin.”

Clear gist, but “deadline” is wrong and the quoted labels remain French. Slightly better than the 350M output.

EN → FRclear error
SourceIn October, GM sold 42,660 Silverado and 16,503 Sierra pickup trucks.
ReferenceEn octobre, GM a vendu 42 660 pickups Silverado et 16 503 pickups Sierra.
ModelEn octobre, les OGM ont vendu 42,660 Silverado et 16,503 Sierra à reprendre des camions.

GM becomes OGM (“GMO”), and the pickup-truck phrase is still wrong. The 800M output did not fix this case.

Aggregate gains do not guarantee every sentence improves. In this packet, the 800M final output is better on the long FR→EN example but regresses on the short FR→EN Nevada example; individual sentences still need human review.

06 / Caveats

What this result does not prove.

The comparison is useful because it is controlled on the evaluation side. It is not a complete quality claim.

Two different 800M comparisons

The 800M continuation uses the same 2.1M pairs as the 350M baseline; its gain reflects about 450M additional target tokens. The 800M ParaCrawl candidate adds 98,034 pairs but starts from random initialization while its baseline is a continuation. The schedule difference remains.

No human evaluation

All results are automatic metrics. The 20-row packet has blank reviewer columns and remains pending; no qualified bilingual review was performed on any reported checkpoint.

Not production-ready

Mid-20s BLEU means the model still mistranslates names, idioms, numbers and sentence structure often enough to need heavy postediting.

Scope. The page reports the completed 800M-token continuation and its final step-91,691 checkpoint, the completed 350M Tatoeba and ParaCrawl arms, and the fresh 800M ParaCrawl scale-up. The 800M ParaCrawl candidate is the dev-selected step-91,000 checkpoint from a fresh 800,004,736-token run; its baseline is the dev-selected continuation checkpoint at step 77,000. The 350M combined arm remains untrained.

Source. Built from the project's evaluation reports in reports/, the completed continuation report in models/scaled_full/, and the data-arm report in models/data_ablation/paracrawl_800m/. Page source: docs/index.html.