Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI

Highlights

• ToneBench evaluates AI creative writing across 120 models using 10 benchmark scripts and 5 generation drafts per model to measure quality and slop.

• Evaluating AI writing quality requires assessing nine metrics across five core aspects: tone, writing quality, instruction following, research accuracy, and slop score.

• LLM judges exhibit strong self-family biases, with OpenAI models showing a marked preference for their own family during evaluations.

• Modifying a benchmark rubric costs over $2,000, while altering an input prompt costs more than $4,000 per experimental iteration.

• Generating 10 drafts with a budget model like GLM 5.3 Flash and merging them using a high-thinking judge provides a cost-effective alternative to single expensive generations.

Timeline

1. 00:00 - 03:42

   Defining AI Slop and Introducing ToneBench

   AI-generated text remains plagued by predictable vocabulary like delve, lack of specific details, repetitive structures, hallucinations, and an absent or generic tone. Despite advancements in underlying models, sloppy output continues to scale alongside model capabilities. To address this issue within educational content operations, ToneBench is introduced as a new benchmark specifically designed to measure creative writing quality and stylistic alignment.

   • AI-generated text suffers from overused words, generalization, redundancy, hallucinations, and a generic tone.
   • ToneBench evaluates whether AI models can perform creative writing without sounding like generic generated content.
   • Educational content creators face constant challenges in reviewing and editing large volumes of AI-generated text.
2. 03:42 - 14:15

   Benchmark Architecture and Evaluation Methodology

   The benchmark pipeline sends detailed briefs containing personal writing styles, research data, and guidelines to 120 models. Generating five drafts per script allows researchers to quantify output variance, noting that weaker models exhibit wider performance ranges. Four LLM judges score the generations across tone, writing quality, instruction following, research accuracy, and slop score using detailed rubrics. Because LLM judges show strong self-family preferences, human reviewers manually evaluate 1,000 AB pairs to ensure scoring accuracy.

   • ToneBench uses 10 unreleased and published scripts to test models and prevent contamination.
   • Each model generates five drafts per script to measure intramodal variability and output variance.
   • Four different LLM judges analyze generations across nine metrics grouped into five core aspects.
   • Manual validation involving 1,000 AB pairs supplements automated scoring to mitigate family bias.
3. 14:15 - 18:55

   Development Costs and Practical Learnings for AI Writing

   Building the benchmark required significant financial investment, with prompt modifications costing upwards of $4,000 and rubric updates exceeding $2,000 per iteration. Practical application shows that achieving ideal AI writing requires extremely detailed prompts filled with concrete stylistic examples and explicit rejections of unwanted phrasing. Rather than relying on expensive single-shot generations, organizations can generate multiple drafts using cost-effective models like GLM 5.3 Flash and merge them using a high-thinking judge.

   • Initial kickoff costs reached $500 for building scripts and rubrics, while prompt and rubric alterations cost thousands of dollars per change.
   • Extremely long and detailed prompts containing positive and negative examples drastically improve generation quality.
   • Combining multi-draft generation from budget models with a high-thinking judge optimizes cost and output quality.