From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto
Transcript
00:00:00Hi, can everyone hear me? Sweet. We have a short 20 minutes here, and so I wanted to jump in and
00:00:20get right to it. My name is Deet, I'm the co-founder and CEO of Reducto, and today we wanted to talk
00:00:25about one of the, I think, really practical, but maybe less sexy parts of building agents that actually work in the real world, which is data. I'm sure you've seen plenty of talks about data today. We primarily have focused on building infrastructure for anybody working with some of the hardest sources of data, which is unstructured images, PDFs, spreadsheets, everything that humans are used to using day to day. Have any of you used Reducto already or trialed it? Cool.
00:00:55I guess helpful context maybe to start is we're an agentic document processing platform. So we help a lot of the world's leading AI teams build both AI applications and also workflows, depending on what they're trying to do.
00:01:08That includes a lot of the AI natives that you've probably seen today. It includes the Harveys, the Ligors, the Rogos, the world, but also some of the largest enterprises in the world. And I think that's important context for what we're going to talk about.
00:01:19We work with the largest tech companies, global financial institutions, insurance orgs, people that have decades worth of historical data that historically has been really, really hard to actually use outside of a demo context.
00:01:34And across all these companies, at this point, we've processed many billions of documents for our customers. I've gotten kind of lazy with that plus symbol at the end, but the number keeps changing, and so we'll let it sit there.
00:01:46But the main thing that we've learned across those and what I wanted to focus on today is not actually Reducto's product itself.
00:01:51It's the intricacies of what we've learned that hopefully you can actually take home and implement in the work that you're doing as well.
00:01:58And I think a lot of the work that we've done and a lot of the learnings we've had are in this sort of broader thematic context of the scope of AI applications has changed a lot recently.
00:02:07From just like information synthesis products to actual agents doing work.
00:02:13And with that, we found that there are a few things that are worth talking through.
00:02:16One is framing the problem itself, the bottleneck that people face, why PDFs in particular are hard, even though you've probably seen two dozen different PDF processing launches on Twitter.
00:02:26Where we find strengths and weaknesses with different tools.
00:02:31We think there's a right place and time for traditional CV versus BLMs.
00:02:35And then more interestingly, I think the latter half of this talk will actually be focused on the next frontier.
00:02:40What we've seen be possible as a result of having agents in a loop.
00:02:43Things that we found as a result of being able to use harnesses for different types of tasks.
00:02:47And our learnings around things like evaluations as you go from RAG to building agent products.
00:02:53But I'll start with the first thing, which is something that I assume has been harped on a lot.
00:02:58If you went to AI Engineer a few years ago, the word that you would have heard in every single talk would have been RAG.
00:03:04And everybody was building some form of RAG application.
00:03:07For a while, all applications were really some form of information synthesis, right?
00:03:12You would pull in information from some context, whether that's a perfect prompt or a file that a user uploaded.
00:03:19And you would have something like a search product.
00:03:22You'd have enterprise search.
00:03:23You'd have a chat bot that would do simple question and answer on top of the content.
00:03:27And that was it.
00:03:28But today, the buzzword that you've probably heard a million times is agents.
00:03:34And for those of you that are engineers, you've probably had Cloud Code or a similar tool do a lot of your end-to-end work for many, many tasks.
00:03:41And that same sort of shift is starting to happen for all sorts of white-collar work.
00:03:46Whether you're in finance or insurance or healthcare, people are starting to make autonomous decisions.
00:03:51They're trying to create end-to-end work products, not just answer questions from a PDF, but generate and modify PDFs as well.
00:03:58And that's a very different framing.
00:04:00The tools that you need, the problems that you face change a lot versus just trying to build a retrieval platform.
00:04:06And the common core for all of this is it actually ends up being an even more important problem to solve for when you start having these multi-step pipelines.
00:04:18If you're just doing question and answer, there is obviously a risk that comes with just answering the question.
00:04:24But when agents are making multiple decisions that compound over the course of a source of files, when they're pulling in multiple sources of data, the risk of bad inputs becomes really, really pervasive in your pipeline.
00:04:36And that's what we focus on.
00:04:38Because at the end of the day, a lot of the value of language model tools in the real world only applies in the context to which you apply that intelligence.
00:04:48So for a lot of enterprises, data is unstructured, it is scattered, and it is multimodal.
00:04:53It's not this like cleanly organized repository.
00:04:56You have random teams and different organizations that go through and put things in Google Drive and Box and wherever else you might have things.
00:05:04You're going to have data formats that are unstructured by default.
00:05:07You don't necessarily know what is in your corpus of information.
00:05:11And you're going to have all sorts of downstream problems that come with that.
00:05:14Some of the problems are going to be parsing extraction accuracy and that definitely matters.
00:05:18But it's also going to be about problems like are you retrieving the right context.
00:05:22It's going to be about how do you actually interact with that context and apply modifications to it.
00:05:27And the thing that you've probably heard reducto harp on and others in the space is PDFs are surprisingly still a very hard problem.
00:05:36I don't know if any of you follow Surge, which is a data lab that works with a lot of the foundation model companies.
00:05:42They have this really great benchmark called GDP PDF where even Fable, like current frontier of model intelligence, is at about 30% on their benchmark.
00:05:52And it's entirely predicated on this idea of can we have models go through and actually make determinations off of contents that would be in documents like PDFs.
00:06:01And the reason why they're hard, I'll come back to that benchmark in a second, is fundamentally PDFs as a file format are both very old but were designed in a very different context.
00:06:11I've genuinely met people that have worked on PDF processing longer than I've been alive.
00:06:16I've met the people that worked on printer drivers for printers to print PDFs in the early 1990s.
00:06:21And that's what it was.
00:06:22Like you wanted to be able to represent what was faithfully on the documents when you originally created it and have it be printable at the end of it.
00:06:30A lot of the considerations today are not around that.
00:06:33Ultimately, what we want is something like a markdown representation, something that agents will effectively reason on.
00:06:39And in the real world, humans encode so much context visually.
00:06:44Like the average financial analyst that is a new grad in their IB role is not going through and thinking in terms of will agents consume this deck.
00:06:54They're creating these really creative slides.
00:06:57I'm sure you've seen the soft bang slides with goose laying eggs.
00:07:01Those details matter.
00:07:02Right?
00:07:03Like a lot of the data that you're going to reason on is going to be tabular structures that maybe don't have clean grid lines and separate out the merge cells.
00:07:10You're going to have things like line charts and graphs.
00:07:12You're going to have messy handwriting that even I as a human would often struggle to read.
00:07:17And that's the sort of problem that you need to solve for if you're dealing with that long tail.
00:07:22And we started the company in 2023 because we felt that there was a sort of step change in what was possible here.
00:07:29For a while, as people would think about any sort of PDF processing problem, it used to be some modified version of an NLP pipeline.
00:07:37You'd do a simple OCR pass and then you'd try to post process that text.
00:07:41And that worked when you would have really consistent layouts.
00:07:44Right?
00:07:45Like if you knew that a W2 is always going to look like a W2, that's a constrained problem space that you can template your way around.
00:07:51But VLMs are interesting because they are fundamentally horizontal in nature.
00:07:56For the first time, you can have this premise of read the document the way that a human would have.
00:08:01You can have this premise of we want to address the long tail.
00:08:04And so we found that they were incredible for all sorts of things like handwritten text in a way that traditional OCR just never was.
00:08:12But the flip side is that we don't think that they're a one size fits all solution.
00:08:16And if you're solving this problem at scale, if you're a company dealing with hundreds of millions of documents,
00:08:21there are all sorts of secondary considerations like determinism.
00:08:25You care a lot about the efficiency of your processing.
00:08:27And so we find that there are some things where traditional CV is actually still really, really strong.
00:08:33And this has been really underappreciated as autonomous vehicle research has gotten better.
00:08:37Techniques like object detection are more sophisticated than they were a decade ago.
00:08:42And so we find that sub-hundred million parameter models are really effective for things like detecting the layout of a document.
00:08:48You can go really, really far without even needing a large foundation model.
00:08:52And these are models that can actually run on CPU.
00:08:54You can run them at scale.
00:08:55You can make sure that you understand on a region level what are the hard things for you to process.
00:09:00And VLMs introduce this notion of semantics.
00:09:03You can go through and actually identify and correct the sorts of mistakes that you were likely to have in your pipeline.
00:09:09And off of that idea of semantics, when you're deconstructing this problem and you have a clear sense of where the more nuanced things are,
00:09:18if you've segmented the text on your page and you understand where the handwriting is,
00:09:21you can introduce this notion of a sort of agent in the loop.
00:09:25Whereas historically you would have had a human review team go through and annotate and correct mistakes,
00:09:29VLMs can now present this sort of idea of what we call agentic OCR.
00:09:33Really for us what that looks like is if you've ever used a tool like cursor that's applying fast edits in your IDE,
00:09:41there's this notion of speculative decoding where you're applying token level edits to your output.
00:09:45You can apply a similar sort of principle here that is not just sending OCR to Gemini,
00:09:51writing a really pretty prompt, asking it nicely to not deviate too much from the original.
00:09:56Because we find that when you're doing that sort of next token prediction,
00:09:58you introduce net new loss cases where models that are really intelligent
00:10:02will start actually correcting things not faithfully to what was in the document.
00:10:05They'll see the word total and if the human made a mistake in that table,
00:10:08models will actually sometimes go through and add up the values in the table themselves.
00:10:13What you want is to correct the token level edits that you want.
00:10:16Maybe you mess up a period versus a comma, a zero versus an O,
00:10:19those sorts of details really, really matter.
00:10:21And that's a question of how do we actually represent what a human would have seen
00:10:25if they had read that document.
00:10:27So the way that we see this is agentic OCR is almost like that human in the loop analogy,
00:10:34where you have the first inputs go through with a CV plus VLM parse,
00:10:39but then you have a verification and correction layer that ends up leading to a high confidence output.
00:10:44But I mentioned earlier that we don't see the range of problems as purely just parsing and extraction.
00:10:50And a lot of what this looks like is I think it's important to think through the details of your pipeline,
00:10:56even if you have a great documents to markdown pipeline.
00:10:59And a great example of this is if you've built any sort of RAG platform,
00:11:04you've probably had some consideration around things like tables.
00:11:07There are a lot of things that you can encode well in something like markdown.
00:11:13But things like this table where the merge cells actually encode a lot of meaning,
00:11:18it matters that you're preserving that sort of structure.
00:11:20And it's not a model limitation.
00:11:22LLMs are incredible at reasoning through like an HTML structure of the same table,
00:11:26but you're also wasting a lot of tokens, and that gets expensive quickly.
00:11:29And obviously on the other end of the spectrum,
00:11:31you probably don't want to go through and encode simple tables in HTML,
00:11:35because then you have a lot of HTML tags that are erroneous.
00:11:38And so what we ended up doing was looking at this as sort of like a dynamic problem
00:11:42of when you have a simple table, great, we can approximate that data in markdown.
00:11:47When you have a more complex table, you may want to use something like HTML,
00:11:50but it's not only language models that should be a consideration in your pipeline.
00:11:55If you're doing anything related to embedding,
00:11:57you're also going to have the secondary problem of retrieval of that context.
00:12:01That same table that I showed you earlier,
00:12:03if you look at the HTML representation is really, really messy.
00:12:08The vast majority of that snippets is just HTML tags.
00:12:11It's just classifying the structure of the document.
00:12:14And the unfortunate thing is, whereas in some blanket evals,
00:12:17you might have contents that is really trying to do the work for the model
00:12:22and say exactly what you're looking for.
00:12:24A real world person does not enumerate the values in the table.
00:12:28They just say how did revenue change over time?
00:12:30And they assume that you're going to retrieve the right table when it's relevant.
00:12:34And whereas language models can reason through that text effectively,
00:12:37if you're pulling from a large corpus,
00:12:39we find that embedding models really struggle to correlate that natural language human prompt
00:12:44with this messy blob of HTML tags and numbers.
00:12:47And so a big thing that you can do that actually takes very little effort
00:12:51is creating a representation that's more so designed for the embedding model itself.
00:12:55Taking the same table,
00:12:56you're creating a natural language representation of that table,
00:12:58so that you have the best of both worlds.
00:13:00When you're actually passing this into the model for reasoning,
00:13:02you're using the HTML table representation,
00:13:04and when you're trying to make sure that you retrieve the right snippets,
00:13:07you're using that natural language block.
00:13:12Off of that, there's also this idea of there's a lot to do that is not just parsing and extraction.
00:13:18And I think a lot of the industry's focus has been on parsing and extraction historically
00:13:22because we do think that there's a massive uplift there.
00:13:25And I talked earlier about the GDP PDF benchmark,
00:13:30which I think is a great illustrative example of what you can see
00:13:34as a result of improving your data pipeline.
00:13:37So what we found is that if you take the same exact benchmark
00:13:40that I mentioned earlier, unfortunately we couldn't test it on Fable
00:13:43because our access was cut.
00:13:46But if you test on other models and you give it both the original PDF
00:13:50but also a structured representation of the PDF,
00:13:52like the parse results here, across models,
00:13:55whether it's Gemini, whether it's Anthropic, or OpenAI,
00:13:58you find that you actually improve end LLM performance just from better inputs.
00:14:04And it's to an extreme where models like GPT 5.5 and Opus
00:14:09actually outperform something like Fable out of the box,
00:14:12not just on an accuracy basis, but as a result of giving better inputs,
00:14:17the models end up needing to use fewer reasoning tokens as well.
00:14:20They're focused less on representing the data and more on the actual outputs.
00:14:24And as a result, they end up driving down latency
00:14:27and also getting to the correct answer more quickly.
00:14:30But even once you have that sort of pipeline
00:14:33and you've gone through and you've actually inspected everything
00:14:37in your parsing layer, a lot of human work is going to require
00:14:41actually understanding the range of what you have in your corpus,
00:14:44routing it to the appropriate pipeline and sort of decomposing that problem,
00:14:48or even at the end editing and modifying your documents.
00:14:51And so what we tried to do is look at this as this problem
00:14:54of how do you make sure that every interaction that a language model has
00:14:57with the document is as effective as if a human would have done it.
00:15:00If you're filling out a form, how do you make sure that you have precision
00:15:03in where you fill out fields?
00:15:05And a really good example of this on the orchestration side
00:15:08is I think classification splitting are a very underappreciated way
00:15:12to have an LLM do its best work.
00:15:14Obviously, you can just go through and dump as much context as you want.
00:15:17And if you're doing a sort of needle in the haystack test,
00:15:19that might be fine.
00:15:20But in practice, there is erosion that you find in quality outside
00:15:24of just the token economics as a result of passing in too much.
00:15:28And instead, what we find is you can get a lot of headroom by thinking
00:15:33through things like how do you classify the right documents
00:15:36to the right sort of pipeline?
00:15:38And even for large documents, how do you make sure that you're passing
00:15:41in the snippets that are actually relevant?
00:15:43We see use cases where people will have things like paper mail,
00:15:47and these paper mail packets can be hundreds of pages long.
00:15:50You don't necessarily know what is going to be contained within it.
00:15:53You might have issues like a person interleave the content,
00:15:57and having the model do that sort of work is almost like a distraction
00:16:01from the work that you're actually trying to achieve,
00:16:03which might be extracting the data from the paper mail,
00:16:05reasoning on it, or making a decision.
00:16:10Again, I mentioned earlier that the second half, I think,
00:16:13is a more interesting piece, which is once you have the sort
00:16:16of initial classification and splitting layer,
00:16:20and you've figured out sorting.
00:16:22I think the thing that we are really excited about as a team
00:16:26is agent harnesses have been this really, really interesting frontier
00:16:29to push past what canonically used to be hard unsolved problems.
00:16:34One good example of this that I'll talk about in a second
00:16:37is things like line charts.
00:16:39We work with many of the largest hedge funds in the world,
00:16:41and things like line charts historically have been really,
00:16:43really difficult because one, they're in imaged format.
00:16:46But two, there's a lot of pixel level granularity that if you're
00:16:49doing anything with a traditional vision encoder,
00:16:52you're probably going to lose.
00:16:53You're going to get a rough plot of how revenue trended,
00:16:55but you're not going to get the individual data points.
00:16:57And so we've been thinking through how do we give agents the ability
00:17:00to have the right tools to solve for the specific type of problem
00:17:04that you're looking at.
00:17:05In the case of chart extraction, the chart on the left encodes
00:17:09a massive table of data.
00:17:11If you actually went through and tried to plot every single pixel,
00:17:14it would be really, really difficult.
00:17:16But it would also be hard for a model to even approximate
00:17:19the intricacies of the lines in between.
00:17:22And there's no model that out of the box can do this
00:17:24as a single shot problem.
00:17:25What you're seeing on the right is a reconstruction
00:17:28of the markdown table that we're able to generate
00:17:30off of the initial line chart.
00:17:32And the only way that we were able to get there
00:17:34was to have an agent with all sorts of tools.
00:17:36It has its own code interpreter.
00:17:37It has the ability to visualize the chart that it's generating.
00:17:40And it's iteratively going through.
00:17:41It's finding mistakes in the line chart again and again and again
00:17:45until it's able to get to the final output.
00:17:47That applies for problems like structured extraction as well,
00:17:51where for a while we've had this document
00:17:53to structured output feature.
00:17:55But you can really take it a step further
00:17:57by having an agent harness around that same sort of task.
00:18:00You can have a parent agent go through set validation criteria
00:18:03for sub-agents to follow.
00:18:05And this means that if you have something like a CBP form
00:18:09with tens of thousands of fields,
00:18:11that's the sort of problem where you end up finding
00:18:13a lot of issues that are silent in nature.
00:18:15Like you drop contents, you drop rows.
00:18:17And MicroOne actually released a really good benchmark
00:18:20in the space this morning where there's this bifurcation
00:18:23in the markets.
00:18:24Frontier models with max reasoning are really, really precise.
00:18:28Like provided that they extracted a row,
00:18:30odds are it's not a hallucinated row.
00:18:31Like they actually got it correct.
00:18:33But they silently drop a lot of the contents across the benchmark.
00:18:37Recall really, really struggles.
00:18:39On the flip side, a lot of dedicated document processing services
00:18:42are actually behind Frontier models from a precision perspective,
00:18:46but close that gap on a recall perspective.
00:18:48And so there's always been this sort of trade-off.
00:18:51And it was only with an agent harness that we were able to find
00:18:54that sort of local maximum of both precision and also recall
00:18:57for this sort of task.
00:19:00The last thing, and maybe the most important thing from this talk,
00:19:04is that I think at the end of the day, evals should underpin
00:19:09all of your decisions.
00:19:10And it's been a big part of how we think about our product.
00:19:12That applies both to off-the-shelf data sets that you eval against,
00:19:16but also to things like real-time production monitoring.
00:19:19Because your production data is going to differ
00:19:21from whatever else you have in your contrived sets.
00:19:25And I really think it's important to think of evals
00:19:28not as just this like macro level view,
00:19:30but also the best teams that we work with
00:19:33look at evals on a granular level for each step of their pipeline.
00:19:36The first thing might be that you want to make sure
00:19:38that the inputs to your pipeline are great.
00:19:40And of course you should eval things like your parsing pipeline.
00:19:43But even perfect parsing with a horrible retrieval pipeline
00:19:47is not going to help if you're not passing in the right context.
00:19:50And so it's important that you're thinking through details
00:19:52like your retrieval pipeline,
00:19:54your formatting at the end of the pipeline,
00:19:56and also ultimately the most important thing
00:19:58is are you able to improve end agent performance.
00:20:03I'll close off just with a sense of where we are headed
00:20:06and where we've seen the industry head.
00:20:08The most important thing I think is as agents get better and better,
00:20:12you can deviate from the sort of deterministic pipeline
00:20:15that you would have had a few years ago.
00:20:17A lot of our customers will actually create effectively a file system
00:20:20for their agent to go through and navigate
00:20:22and let the agent decide what sorts of tools it wants to use.
00:20:25So we create a CLI where instead of people creating an end-to-end pipeline
00:20:28where documents always follow one specific flow,
00:20:32the agent will decide if it needs to read a certain type of document,
00:20:35and they'll split that into two sets.
00:20:38One is a content field which the agent can read as it needs to,
00:20:41but the other is all the metadata that you would have wanted.
00:20:44If you're doing things like citations, you may want bounding boxes,
00:20:47so on and so forth.
00:20:50I'll skip this part on editing.
00:20:52I think there's a lot of interesting work being done here.
00:20:54We've already released some of it, but in the next few months,
00:20:57you will see us look more and more towards things like document generation.
00:21:01But the recap for today, and I really appreciate your time,
00:21:04is one, I highly recommend that you decompose the parsing problem.
00:21:08To the extent possible, you should think of it as the right tool for the right task
00:21:11so that you can hit that period of frontier of accuracy, cost, and latency.
00:21:16Two, I think agentic verification is the biggest step change that the industry has had for a while,
00:21:21and it's a really good opportunity for you to make sure that you're building pipelines that work in production.
00:21:26Three, I think it takes very little effort, but there's a lot of headroom from details,
00:21:31like formatting the data for its consumer.
00:21:34Or similar vein, I think it's really, really important to think about not just the data processing,
00:21:39but the data orchestration.
00:21:41And so you should always think about tools like classify and splits
00:21:44as a way to augment your pipeline.
00:21:46Five, make sure that you eval at every stage.
00:21:49And six, think through what that next frontier looks like for you,
00:21:52because I think most successful companies in today's era have deviated a lot
00:21:56from what we used to do two, three years ago.
00:21:59But if you have any questions, please feel free to reach out at any point.
00:22:03My email is just firstname@reducto.ai.
00:22:06And you can also reach out on our website if we can be helpful for your use case.
00:22:10Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video