Written with AI assistance from the source video. The video is the authority.
Log in to leave a comment
No posts yet
Modern LLMs are already biting their own tails. When a model swallows the entirety of evaluation data scattered across the internet, the answers it produces are likely a product of memorization rather than intelligence. To witness true reasoning capabilities, we must pull data from an era where modern knowledge simply did not exist. I have outlined a specific process for creating a contamination-free training environment using public domain data from before 1931.
Before pouring a budget into data collection, you should first raid repositories where copyrights have expired. Project Gutenberg houses over 75,000 documents, and the Internet Archive's Sonny Bono Memorial Collection provides academic data from between 1923 and 1941 free of charge.
gutendex API, filter based on the author's death year and the first edition publication date rather than the metadata's Issued field to ensure only pre-1931 materials remain.LCC fields in the metadata to prioritize downloading texts related to philosophy (B), mathematics (QA), and classical logic.Century-old paper is worn, and newspaper layouts are complex. Running standard OCR results in a flood of typos. You need a process that deconstructs the layout before simply scraping the text.
LayoutParser framework to distinguish between titles and tabular regions within the document. Specifically for multi-column newspapers, the Newspaper Navigator model should be used to correct the reading order.LayoutLM to understand visual coordinate information and determine the logical sequence of text blocks before executing OCR by region.REVISE framework. Assign the LLM the role of a professional historical document editor to correct misrecognized words while maintaining period-appropriate spelling. This process can bring recognition rates—which often stall at 30%—up to a trainable level and cut refinement time in half.We must prevent the model from feigning intelligence by stealing modern knowledge. Build a system to monitor training datasets by creating a list of terms born after 1931.
Just because the data is old doesn't mean the intelligence is. On the contrary, literature like Bertrand Russell's Principia Mathematica (1910) serves as a superior textbook for teaching deductive reasoning compared to modern web data.
For evaluation, use past exam papers that aren't saturated with answers on the modern internet. Use the artificial language and logical reasoning questions from the very first SAT administered in 1926 as your evaluation data. Measuring zero-shot reasoning capabilities with questions from the 1916 revised Stanford-Binet Intelligence Scales clearly reveals whether the model has memorized an answer or is understanding and applying given rules on the fly. A model that can properly answer questions from 100 years ago is one possessing true intelligence, free from the suspicion of data contamination.