Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語PortuguêsРусский中文
AAI Engineer
Computing/SoftwareInternet Technology

Transcript

00:00:00you know, being continually updated within your SharePoint, just for, like, rag knowledge-based
00:00:04search. You know, in these cases, you obviously want it to be not, like, terribly inaccurate,
00:00:08but even if it's a little bit messed up, it's okay, too, because in the end, if you have a
00:00:12sufficiently good agent, it can always dive deeper into the document and surface the right
00:00:16information with the right citations and grounding. So, for these, you know, being able to
00:00:20create some sort of scalable offline indexing pipeline that has the best, like, cost constraints
00:00:25is something that is optimal. One thing about VLM-based approaches, though, is that, you
00:00:31know, they're typically not very fast, and I think a lot of times, if you have, like, real-time
00:00:36file uploads, let's say you're using Cloud Cowork, you upload 1,000 documents, and you need to
00:00:41process it within a minute, like, having a bunch of VLMs process that at scale is really tough
00:00:47for basically every single OCR service out there, and to be fair, that includes ours, too. I
00:00:52think, in general, there is also some sort of need for an extremely low-latency solution
00:00:57so that you can actually process stuff in real-time, even if you have, like, deeper VLM-enabled
00:01:03processing for kind of, like, deeper visual inspection and analysis. And so, that's what I call
00:01:08kind of being in the agent loop. So, besides LamaParse, which is kind of our commercial service
00:01:12around, like, document processing and extraction, we also created this tool called LightParse.
00:01:17It is surprisingly really, really good. I don't know if you've been following some of the Twitter
00:01:21threads, but it is Rust-based. It is the fastest open source parser out there. It is completely
00:01:27free, and there's basically no strings attached. I think it's, like, MIT or Apache license. And
00:01:33it basically is the most accurate, like, markdown parser out there that doesn't use a VLM or
00:01:37any sort of kind of, like, deeper model. And so, this is kind of nice because you can use
00:01:43it as a default in the assistive agent loop. Let's say you're uploading a bunch of documents
00:01:48to Cloud Code, Cloud Codework, Codex. And you want it to process, like, a thousand PDFs extremely
00:01:53quickly. You can always do that. And then, you know, equip a VLM-based parser, like LamaParse
00:01:59or other frontier models, as a tool. So, what these agents will do is they'll do, like, a fast
00:02:04pass over all the documents first. Kind of, like, just scan through all the context extremely
00:02:09efficiently. And then, if it actually needs to dive into a page with, like, tables, with, like,
00:02:13charts and actually needs to more deeply understand the values, it'll use a VLM-based tool, slower
00:02:19processing, to actually make sure it reads the information correctly. This is available
00:02:22as a one-click installable skill. Compliments kind of any other deeper VLM-based OCR tool
00:02:27you want to use. And we kind of designed to make it as fast as possible and also easy to
00:02:32plug in to your favorite AI agent.
00:02:35So, I kind of speed ran through a bunch of this stuff, but basically, you know, we spent
00:02:42a bunch of time on the parsing layer. There's also other general components around, like, the
00:02:47semantic and storage layer in terms of document extraction and search, and, of course, like,
00:02:51document workflows. I think due to time, I'll probably kind of just skip some of the unexplored
00:02:57areas, like agent-native document formats, hill-climbing as a service, and others. But I'll share
00:03:01the full set of slides online. In terms of the semantic and storage layer, you know, besides
00:03:06document parsing, a lot of use cases also require actually getting back structured information
00:03:12at scale from documents. Whether you're processing, you know, a million invoices or expense reports
00:03:16or receipts or claims, you need to make sure that, you know, you want to actually get back
00:03:21structured outputs that you can put into a downstream database or system. And so, a lot of these use
00:03:26cases basically revolve around the form of, you know, how do you automate a lot of workloads
00:03:31that humans typically do in scanning a lot of paperwork and doing data entry. Whether
00:03:36it is, kind of, again, invoices, claims, contracts, receipts, or others. We kind of created these
00:03:41capabilities within LamaParse as well. A lot of our capabilities are actually tuned towards,
00:03:47like, low cost while extremely high accuracy. And you get back granular citations all the
00:03:52way back to the source document for every extracted output. And, of course, you can run this in
00:03:56the pipeline at scale with confidence scores. And also, you know, being able to actually flag
00:04:02whether or not we're certain about a certain value before deciding to put it into some sort
00:04:06of the software system. There's also document search, which is a basically expanded toolset, as I mentioned,
00:04:12around retrieval, VM25, grep, vector search, reading, and scrolling. And so, all these capabilities are
00:04:19available within some of our commercial platforms as well as open source offerings. But I also just wanted
00:04:23to paint a picture of the general concepts out there today. So, I'll skip this section about kind of what's next and then maybe just go all the way to the end. I know I'm a little bit over time, so really appreciate you all spending time today. And then let me just
00:04:36just, how do I get to this part really quick. Oh, right. I'm going to skip this piece. I'll put this online. Our booth is at LG 47, if you guys are interested in stopping by. And we're hosting a giant pickleball tournament today. So, thank you for your time.
00:05:06Thank you.

Key Takeaway

AI agents achieve optimal speed and accuracy by using LightParse for a low-latency initial pass over bulk PDFs, invoking deeper vision-language models like LlamaParse only when complex tables or charts require visual inspection.

Highlights

  • LightParse is a free, Rust-based, open-source Markdown parser that operates without vision-language models (VLMs) or heavy deep learning architectures.

  • Real-time file uploads of 1,000 PDFs strain traditional vision-language model OCR services due to high processing latency.

  • A two-tier parsing workflow uses LightParse for a rapid initial document scan and reserves VLM tools like LlamaParse for complex pages containing tables or charts.

  • LlamaParse extracts structured tabular data from invoices, claims, and contracts with granular citations back to the source document.

  • LlamaParse incorporates confidence scoring to flag uncertain values before injecting extracted data into downstream software systems.

Timeline

Real-time PDF processing demands two-tier agentic parsing

  • Processing bulk uploads like 1,000 PDFs in real time exceeds the speed capacity of visual-language model OCR services.
  • LightParse processes documents as a free, open-source Rust-based Markdown parser without deep learning model overhead.
  • Combining LightParse with VLM-based tools allows AI agents to scan large contexts quickly before triggering deep inspection on complex pages.

Offline indexing pipelines for RAG knowledge bases require scalable, cost-optimized solutions. While VLM-based approaches handle visual elements well, high processing latency creates bottlenecks during real-time bulk document uploads. LightParse addresses this low-latency requirement by operating as a fast open-source Rust parser under MIT or Apache licensing. AI agents implement LightParse as an installable skill to execute an initial fast pass over thousands of files, delegating pages with tables or charts to LlamaParse or frontier VLM tools only when necessary.

Structured extraction and document search at scale

  • Structured data extraction from high-volume forms replaces manual paper-scanning and data-entry workflows.
  • Extracted outputs link back to source document coordinates through granular citations.
  • System pipelines use confidence scores to flag uncertain values before saving them to downstream databases.

Automating paperwork workloads like invoices, expense reports, claims, and contracts requires converting unstructured files into reliable downstream database records. LlamaParse balances low processing costs with high accuracy, attaching granular citations to every extracted field. Built-in confidence scoring verifies individual data values, flagging uncertain extractions prior to software system integration. The framework supports hybrid document retrieval strategies by integrating standard grep, vector search, BM25, reading, and scrolling capabilities.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video