My AI News Pipeline Has A Different Model Than My Top Test Pick

🔍 Read the full analysis: My AI News Pipeline Has A Different Model Than My Top Test Pick on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says his news pipeline will use GPT-6 Luna rather than the highest-scoring model in his test, GPT-6 Astra, because Luna delivered acceptable results at about one-hundredth of Astra’s reported per-article cost. In a test of 71 model configurations using real news sources and production prompts, Astra scored highest, while the evaluation also exposed two backup models that generated fabricated quotes or unusable output.

Meyer’s test ran from September 8 to October 13, 2026. Each configuration was asked to turn news sources into articles in English and German. The sources included a detailed product filing of 6,340 characters and a short political item of 619 characters. Meyer said short sources are common in his workflow, making them an important part of the evaluation.

GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and received a 5.00 source-fidelity score there. Across a further 12-article robustness run, it averaged 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. Meyer put Astra’s cost at about $0.12 per article and its typical generation time at around 50 seconds.

GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71 out of 5; all of its articles passed the pipeline’s validity requirement. Meyer reported a cost of about $0.0012 per article in the comparison, and about $0.0013 in production over the prior week, with a median generation time of 17 seconds. He said the roughly 100-to-1 cost difference led him to choose Luna for routine publishing.

Cost Shaped the Production Choice

The results illustrate a practical distinction for publishers using AI: the model with the highest evaluation score may not be the one that makes sense for a repeated, high-volume task. Meyer’s reported figures put Astra’s per-article cost at about 100 times Luna’s, while Astra’s robustness-run average was higher by 0.26 points on a five-point scale. That trade-off matters when a pipeline generates articles regularly, though the report does not establish what quality threshold other publishers should use.

The test also highlights that quality control must cover the entire route, not just its primary model. Meyer said the two fallback models were removed after tests found fabricated quotations in one and no valid outputs in the other. A backup model can become the source of a serious failure if it is invoked during routine operations and is not checked against the same standards as the main model.

These findings are specific to Meyer’s task, prompts, source samples, judges and output-repair process. They offer a case study in evaluating models against the work they will actually do, rather than assuming that a general leaderboard or a model’s performance in another field predicts newsroom performance.

How the News-Writing Test Worked

Meyer says the evaluation used the same prompt as his production pipeline and sources drawn from real news material. Articles were produced for an English edition and a German edition aimed at readers in Germany, Austria and Switzerland. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, reviewed the original sources alongside the drafts.

The judges scored source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles also received a language-quality score. Meyer counted an article only if it produced valid output after the repair step used in production. He cautioned that comparisons based on just two initial test articles could be noise when score gaps were below about 0.3 points, and described the additional robustness run as a more reliable ranking.

One comparison tested all six reasoning-effort settings for Luna. Meyer reported that the scores varied without a reliable advantage over the production setting, which uses no added reasoning effort. The maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting, according to his measurements.

“Public leaderboards tell you which model is smartest. They don’t tell you which model should write your articles.”

— Thorsten Meyer, in the report

Limits of the Model Comparison

The report describes one publisher’s workflow, not a broad evaluation of all news-writing tasks. It does not establish that Luna will perform similarly with other prompts, source types, languages, editing standards or publishing systems. The report also does not provide enough information to independently reproduce every result, including the full prompts, all source material and the complete article-level scoring data.

The judges were AI systems, and the supplied account does not describe a human editorial review or an independent audit of their scores. Although the test included two judges and a further robustness run, those methods do not settle how the results would compare under other evaluation approaches. Meyer’s cost and timing figures are reported measurements from his setup; actual costs and speeds may vary.

The report says Luna is now the production model but does not give a longer-term comparison of published article quality, correction rates or reader outcomes. It also does not explain the full current fallback configuration: the source text says Luna runs first and then repeats Luna as its backup, so the identity of any distinct secondary model is unclear.

Production Results Remain to Come

The next useful evidence will be performance over a longer production period, including whether Luna continues to produce valid articles and how often editors need to correct its work. The report gives a cost and median generation time for the prior week, but does not yet provide a sustained comparison of accuracy, corrections or publishing outcomes against Astra.

Meyer’s stated production choice is to keep Luna in the news route and remove GPT-4.1 nano and gpt-oss-120b as fallbacks. Further reporting would need to clarify what backup model, if any, now handles failures, and whether the pipeline has additional review safeguards for quotations and unsupported details.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across the additional 12-article robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer said Luna’s results were consistent enough for his quality bar while its reported cost was about one-hundredth of Astra’s. He reported Luna at roughly $0.0012 to $0.0013 per article and Astra at about $0.12.

Did higher reasoning effort improve Luna’s results?

Meyer reported that score differences across six reasoning-effort settings were within the test’s noise. The maximum setting took and cost about 12 times as much as the no-effort setting, without a reliable score gain.

What went wrong with the fallback models?

According to Meyer, GPT-4.1 nano scored 1.00 out of 5 for source fidelity and produced invented quotes. The local gpt-oss-120b configuration produced zero valid articles out of four in the test and also generated fabricated named quotes. Both were removed from the route on October 7.

Do these results show which model is best for every newsroom?

No. The findings apply to Meyer’s prompts, sources, languages, judges and production checks. Other newsrooms would need to test models on their own material and editorial requirements before drawing a similar conclusion.

Source: ThorstenMeyerAI.com