The completed 800M-token continuation improves every newstest2014
comparison against the preserved 350M full-data checkpoint. At 350M,
ParaCrawl is the clearest data addition; Tatoeba helps mainly FR→EN.
A fresh 800M ParaCrawl run also improves every cell. The gains are
real; quality remains pilot-level.
Automatic metrics onlyNo human evaluationTrained from scratch
The installer bundles the exported model and a Java runtime, so the
app needs no Python, no separate model download and no network after
install. It is a pilot: useful gist translation, not production.
The SHA-256 is read from the release asset metadata. The installer is
unsigned; verify the hash and follow the SmartScreen steps below.
Install
Download the MSI above.
Run it. If SmartScreen appears, choose More info → Run anyway.
No administrator is required: the installer is per-user and adds a Start-menu shortcut under %LOCALAPPDATA%\Transcend Translator.
Launch it. The model loads locally; after installation the app works offline.
SmartScreen, honestly
This build is not code-signed, so Windows may show
“Windows protected your PC” or “Unknown publisher”. That is expected
for an unsigned pilot, not a detection.
Choose More info → Run anyway.
Smart App Control on a fresh Windows 11 install may block unsigned
installers entirely; a signed build is the permanent fix.
Verify and uninstall
In PowerShell, compare the download with the SHA-256 above:
Preserved 350M step 38,000 versus 800M final step 91,691 on exactly
the same retained newstest2014 rows. Beam 4 is the dev-tuned setting;
greedy remains the model default.
Read this first
BLEU in the mid-20s is a gist-level score, not a percentage of correct
sentences. The model can carry the main idea of a straightforward news
sentence; it still drops details and produces wrong words.
Headline and tables use the final step-91,691 checkpoint. The
dev-selected step-77,000 checkpoint is slightly lower but also above
the 350M baseline; the comparison report includes both.
Full newstest2014 scores; positive deltas favor the 800M final checkpoint.
Direction
Decoder
350M BLEU
800M final BLEU
Δ BLEU
350M chrF++
800M final chrF++
Δ chrF++
EN→FR
Greedy
22.655
25.367
+2.712
49.045
51.391
+2.346
EN→FR
Beam 4
24.079
26.440
+2.361
50.318
52.531
+2.213
FR→EN
Greedy
21.260
23.558
+2.298
48.755
50.830
+2.075
FR→EN
Beam 4
22.145
24.377
+2.232
49.882
51.902
+2.020
Paired document bootstrap
1,000 resamples over 176 newstest2014 documents. Every interval excludes zero.
Direction
Decoder
ΔBLEU 95% CI
ΔchrF++ 95% CI
EN→FR
Greedy
+2.712 [+2.298, +3.077]
+2.346 [+2.106, +2.587]
EN→FR
Beam 4
+2.361 [+1.978, +2.752]
+2.213 [+1.998, +2.451]
FR→EN
Greedy
+2.298 [+2.022, +2.580]
+2.075 [+1.859, +2.282]
FR→EN
Beam 4
+2.232 [+1.904, +2.559]
+2.020 [+1.801, +2.253]
Sentence-level is mixed
Descriptive chrF++ win/loss/tie counts, not a significance test.
Direction
Decoder
Wins
Losses
Ties
EN→FR
Greedy
1,851
978
130
EN→FR
Beam 4
1,814
937
208
FR→EN
Greedy
1,766
944
212
FR→EN
Beam 4
1,740
948
234
02 / Data ablation
Two data additions. ParaCrawl wins at both scales.
At 350M, each arm keeps the 19.8M-parameter model and the 350M-token
budget, then adds about 99k everyday-language pairs to the same
2.1M-pair baseline. ParaCrawl improves every cell; Tatoeba's gain is
FR→EN-specific. A fresh 800M ParaCrawl run now confirms the same
direction at the larger budget.
Tatoeba-only
direction-specific
99,135 added pairs, mostly short conversational sentences. The
EN→FR point estimates are positive but both 95% intervals cross
zero. FR→EN clears the interval in greedy and beam 4.
98,034 added pairs from a bounded, filtered web sample. Every one
of the four cells improves on both BLEU and chrF++, and every
paired chrF++ interval excludes zero.
Full newstest2014 scores versus the full-data 350M checkpoint (step 38,000). Positive deltas favor the data arm.
Arm
Direction
Decoder
BLEU
Δ BLEU
chrF++
Δ chrF++
chrF++ 95% CI
CI excludes 0
Tatoeba +99,135 pairs
EN→FR
Greedy
22.828
+0.173
49.224
+0.179
[−0.076, +0.435]
No
EN→FR
Beam 4
24.224
+0.145
50.319
+0.001
[−0.239, +0.230]
No
FR→EN
Greedy
21.803
+0.543
49.178
+0.423
[+0.184, +0.665]
Yes
FR→EN
Beam 4
22.481
+0.336
50.233
+0.351
[+0.118, +0.568]
Yes
ParaCrawl +98,034 pairs
EN→FR
Greedy
23.968
+1.313
50.062
+1.017
[+0.770, +1.275]
Yes
EN→FR
Beam 4
25.220
+1.141
51.200
+0.882
[+0.651, +1.119]
Yes
FR→EN
Greedy
22.488
+1.228
49.677
+0.922
[+0.661, +1.174]
Yes
FR→EN
Beam 4
23.281
+1.136
50.770
+0.888
[+0.618, +1.146]
Yes
Baseline BLEU / chrF++: EN→FR greedy 22.655 / 49.045, beam 4 24.079 / 50.318;
FR→EN greedy 21.260 / 48.755, beam 4 22.145 / 49.882. Each arm is a
separate from-scratch run to 350M target tokens, not a fine-tune of the
baseline. The additions are about 4.7% more training pairs, so this is
an equal-token comparison, not equal-data.
800M ParaCrawl scale-up
confirmed at 800M
A fresh 800M ParaCrawl run completed at 800,004,736 tokens and step
91,754; the dev-selected checkpoint is step 91,000. Against the 800M
continuation baseline at step 77,000, all four cells improve and all
paired chrF++ intervals exclude zero. The effect holds at both
scales.
The baseline is a continuation of the 350M full-data run, while this
candidate is a fresh 800M run. This is an equal-token comparison, not
an identical-schedule comparison. The 350M combined arm remains
untrained.
Full newstest2014 scores for the 800M ParaCrawl arm. Baseline is the 800M continuation at step 77,000; candidate is step 91,000. Positive deltas favor the candidate.
Direction
Decoder
Baseline BLEU / chrF++
ParaCrawl BLEU / chrF++
Δ BLEU
Δ chrF++
chrF++ 95% CI
EN→FR
Greedy
25.021 / 51.065
26.174 / 52.144
+1.153
+1.079
[+0.757, +1.388]
EN→FR
Beam 4
26.277 / 52.249
27.709 / 53.260
+1.432
+1.011
[+0.743, +1.283]
FR→EN
Greedy
23.330 / 50.600
24.806 / 51.685
+1.476
+1.085
[+0.843, +1.334]
FR→EN
Beam 4
23.898 / 51.462
25.458 / 52.565
+1.560
+1.103
[+0.903, +1.324]
Same retained rows and decoders as the headline comparison: 2,959
EN→FR and 2,922 FR→EN segments, with 1,000 document-level bootstrap
resamples. The candidate adds 98,034 ParaCrawl pairs to the baseline
training mixture and was trained from random initialization.
03 / Training run
Same data, more tokens, same 19.8M model.
The 350M baseline trained on the complete prepared Europarl/News
Commentary split. The 800M continuation resumed from it with the same
2.1M pairs and added 450M target tokens. This is a token-budget
comparison, not a clean data-only ablation.
2.10Mtraining pairs
800Mtarget tokens
91,691updates at stop
17h 49mcumulative training time
Development curve
Greedy dev chrF++ by training step
Best dev56.08 at step 77,000
MeanEN→FRFR→EN
Validation on 300 dev rows per direction; not the newstest2014 test score. Y-axis begins at 20. Best mean was 56.08 at step 77,000; the last validation, at step 91,000, was 55.84.
Run
Train pairs
Target tokens
Best step
Final step
Dev chrF++
Parameters
350M baseline
2,101,321
350,004,440
38,000
40,113
54.8099
19,825,920
800M continuation
2,101,321
800,002,117
77,000
91,691
56.0833
19,825,920
The headline evaluates models/scaled_full/checkpoint.pt, the
final 800M checkpoint at step 91,691. The dev-selected best is
models/scaled_full/best.pt at step 77,000. Both stopped on
max_train_tokens.
04 / Method
One benchmark. Same rows. Automatic metrics.
The preserved 350M checkpoint, both 800M continuation checkpoints,
the two completed 350M data arms, and the fresh 800M ParaCrawl arm
were decoded on the same retained newstest2014 segments with the same
references. Nothing was truncated silently.
Evaluation setup
Benchmarknewstest2014 (wmt14/full), both directions
The 800M continuation wrapper evaluates the preserved 350M checkpoint and both 800M checkpoints on identical rows. The data-arm wrappers evaluate their baselines and candidates with the same rows, references, decoders and bootstrap. The 800M ParaCrawl wrapper is scripts/run_paracrawl_800m.sh.
reports/review_packet_scaled_full_20/ — 20-row source/reference packet; original outputs from 350M; human review pending
05 / What it looks like
Aggregate gains, visible errors.
Examples from the same deterministic 20-sentence review packet,
decoded with the 800M final checkpoint. These are illustrative, not a
random quality estimate or a human evaluation.
FR → ENregression
SourceLe Nevada a déjà mené à terme un projet pilote.
ReferenceNevada has already completed a pilot.
ModelNevada has already conducted a pilot project in the long term.
The gist survives, but “in the long term” is unsupported. This sample regressed against the 350M output.
EN → FRclear error
SourceNevada has already completed a pilot.
ReferenceLe Nevada a déjà mené à terme un projet pilote.
ModelLa Nevada a déjà achevé un pilier.
Wrong article/gender and a wrong noun (“pilier” instead of “pilote”). Not acceptable as written.
FR → ENmixed
SourceCela leur laisse le temps de faire certains tests et de les comprendre, et un délai avant d'inscrire « masculin » ou « féminin ».
ReferenceIt gives them the time to do some tests and figure it out and a period of time before they write 'male' or 'female.'
ModelThis leaves them time to carry out some tests and to understand them, and a deadline before they inscribe “masculin” or “feminin.”
Clear gist, but “deadline” is wrong and the quoted labels remain French. Slightly better than the 350M output.
EN → FRclear error
SourceIn October, GM sold 42,660 Silverado and 16,503 Sierra pickup trucks.
ReferenceEn octobre, GM a vendu 42 660 pickups Silverado et 16 503 pickups Sierra.
ModelEn octobre, les OGM ont vendu 42,660 Silverado et 16,503 Sierra à reprendre des camions.
GM becomes OGM (“GMO”), and the pickup-truck phrase is still wrong. The 800M output did not fix this case.
Aggregate gains do not guarantee every sentence improves. In this
packet, the 800M final output is better on the long FR→EN example but
regresses on the short FR→EN Nevada example; individual sentences still
need human review.
06 / Caveats
What this result does not prove.
The comparison is useful because it is controlled on the evaluation
side. It is not a complete quality claim.
Two different 800M comparisons
The 800M continuation uses the same 2.1M pairs as the 350M baseline; its gain reflects about 450M additional target tokens. The 800M ParaCrawl candidate adds 98,034 pairs but starts from random initialization while its baseline is a continuation. The schedule difference remains.
No human evaluation
All results are automatic metrics. The 20-row packet has blank reviewer columns and remains pending; no qualified bilingual review was performed on any reported checkpoint.
Not production-ready
Mid-20s BLEU means the model still mistranslates names, idioms, numbers and sentence structure often enough to need heavy postediting.
Scope. The page reports the completed 800M-token
continuation and its final step-91,691 checkpoint, the completed 350M
Tatoeba and ParaCrawl arms, and the fresh 800M ParaCrawl scale-up.
The 800M ParaCrawl candidate is the dev-selected step-91,000 checkpoint
from a fresh 800,004,736-token run; its baseline is the dev-selected
continuation checkpoint at step 77,000. The 350M combined arm remains
untrained.
Source. Built from the project's evaluation reports in
reports/, the completed continuation report in
models/scaled_full/, and the data-arm report in
models/data_ablation/paracrawl_800m/. Page source:
docs/index.html.