Your Agreements Are a Database You Can't Query — Hiral Shah, Docusign & Sean Sodha, NVIDIA
AAI Engineer
Computing/SoftwareManagementInternet Technology
Transcript
00:00:00Welcome everyone to our session. I'm Haral, I'm a senior director of products at DocuSign,
00:00:19and I'm joined by Sean. Hello everyone, I'm a product manager at NVIDIA.
00:00:23So today, Sean and I are going to talk about a massive problem that every enterprise faces,
00:00:29which is agreement data, large-scale agreement data. Agreements are a big part of any relationship,
00:00:37any B2B organization kind of goes through, day in, day out, and a lot of that data is captured inside
00:00:45that agreements, and it's very critical, whether it's pricing tables, a lot of it, and it's all in
00:00:51a lot of different unstructured format, and that's kind of what we're going to show is how DocuSign,
00:00:57partner with NVIDIA, are fixing that on making that data available, readable, usable for a lot of our
00:01:04kind of organizations. So here is kind of just a quick map of our talk today. We'll start with
00:01:10just the stakes. Why does this matter? Why the scale is so large? And then we'll dive deep into the
00:01:16technical architecture of how we are approaching it, how we have tackled this thing, especially this
00:01:22document processing at scale. And then finally, we'll cover what we have learned from our evaluation
00:01:29of all of the different models we've tried for different purposes and share our learnings with
00:01:34you. So why, like, you know, just to understand why this is a big problem. When you think about
00:01:42DocuSign, raise your hands, how many of you have used DocuSign? Anyone who's employed, probably the HR docs,
00:01:47right? So it's a massive scale. Everyone uses DocuSign. For us, it's a massive engineering problem as well,
00:01:55because just look at the scale. We have 1.9 million customers who are paying us and a billion users.
00:02:02What does that imply? We process a million agreements a day that needs to now structurize, make it readable,
00:02:10make it queryable, usable. And, you know, in the past we've worked with Deloitte on a study, and it says
00:02:16that there's $2 trillion captured in this agreement negotiated value that no one capitalizes, no one goes
00:02:24back and gets that data back, right? And why? It's because they have to do a lot of human reading, human reviews,
00:02:31there's disconnected systems, a lot of manual workflows that are there. So that's kind of why
00:02:37DocuSign built IAM, an intelligent agreement management platform that takes the entire, like,
00:02:43applying in an AI-first way, the entire agreement life cycle, whether you are creating agreements,
00:02:49Gen AI helps a lot with that, whether you're negotiating to understanding and redlining, all the
00:02:55way to after signing, storing, and making a lot of insights from this data. So when you think
00:03:01about the challenges involved, right, in an agreement, there is the unstructured data, it could
00:03:07be a PDF, it could be a PNG, and what are people wanting to do is simple questions. They can't get
00:03:13that, that is the data that is trapped inside one agreement, but also the whole corpus of millions
00:03:20of agreement, 10 years, 20 years of business has kind of put into that. So for agreements aren't flat,
00:03:28they're also hierarchical in nature. They are like, you know, one agreement governs the other, the other
00:03:33kind of does something else. So you always are needing a lot of things to answer this question. Simple
00:03:39thing, I'm sure you all are using a lot of, right, Claude and all, especially at your company's
00:03:43organization. Simple thing, what did we contract for the total tokens, right, with Claude? No one knows.
00:03:48That's captured inside this agreement in different forms and fashion. So you need to be extracting this
00:03:54data to find things, but also insights and push it downstream where you're tracking, doing more
00:03:59things. And when you analyze an enterprise contract, a big set of things are captured, what I call vital
00:04:06terms like pricing tiers, the SKUs, the information, SLAs, rate cards, they're all in table format. Now,
00:04:13traditional document extraction tools or a generic VLMs completely fail here. Like, you know, we've tried,
00:04:21we've definitely done this, because they're reading text line by line, which breaks a lot of that concept
00:04:27within the table. Emerge sells. Some things are not boundaried. So this makes a massive operational
00:04:33overhead for the downstream legal teams, procurement teams, sales teams to get that queries and get that
00:04:39answers done. And they spend hours and hours digging through this to even just locate a basic thing.
00:04:45And that's kind of where we partnered with NVIDIA and leveraged a purpose-built model, like, you know,
00:04:51tool, which is really, like, you know, making our architecture for table extraction take us to make and
00:04:58solve these complex use cases. So with this, we are making things scalable. We can understand it with the
00:05:06layout, but also deliver really accurate results. And to share more about how we're leveraging the
00:05:12NemoTron, I'm going to hand it to Sean.
00:05:17All right. Hello, everyone. So real quick on the NemoTron Retriever initiative. So for those who
00:05:23here knows about NemoTron, maybe raise your hand real quick. Awesome. So NemoTron is all about building
00:05:29world-class open source models and publishing the data sets, the techniques, the quantization
00:05:34approaches, distillation approaches, pruning approaches, every technique possible, blueprints
00:05:39to go with that, you name it. Throughout the NemoTron portfolio, we have specifically NemoTron
00:05:44Retriever, which is building embedding models, re-ranking models, and document extraction models.
00:05:50So real quick here, sort of our first initiative is, if you're a large-scale enterprise that deals
00:05:55with petabytes-scale data, our first initiative is, how do we make sure that you find the right
00:06:00document given a certain query? Your agent sends, you know, a set of queries to the corpus afterwards.
00:06:05Once you find those top five documents, whatever it may be, then we say, okay, you found the right
00:06:10document. Now, how do you then find the right information within the document? And this is where
00:06:14the work with the DocuSign team has gone really great, where we've worked with them to build the
00:06:18NemoTron parse model to focus specifically on table extraction, which is a really complicated technique.
00:06:24If you think about it, the number of permutations of tables are quite vast when you think about nested
00:06:28tables, merged cells, merged columns, merged rows, whatever it may be, and that can get really,
00:06:33really complex and really hairy of a problem. So real quick, as I mentioned before, right, our team is
00:06:38responsible for leading a lot of leader boards in the retrieval space. So Vidori, V1, V2, V3,
00:06:44MTEB, MMTEB. So our team knows how to build world-class retrieval models, given a lot of
00:06:50leadership, given a lot of leaderboard winnings that we've had in the last year or so. And then,
00:06:54of course, as I mentioned before, we open source everything, right? So we share the open source
00:06:59model weights, the techniques, and then we release with those blueprints and skills that agents can use
00:07:04then afterwards. So touch a little bit on the actual model that we are working with with DocuSign was
00:07:09the NemoTron parse model. So when you think VLM, you generally think a multi-billion parameter
00:07:14model. It's very heavy. It's high latency. This is a very small, tiny C-radio VLM. It's about
00:07:22850, 900 million parameter model designed to kind of be that all-in-one package sort of model where
00:07:28you deploy it. And instead of having small, let's say, YOLOX models that do table extraction or page
00:07:35element extraction, where it may be, this is a single-shot model that you can feed a document in,
00:07:39and out comes the semantic formatting layouts, the text, the reading order, the preserved structure of
00:07:46the table, et cetera. This can be served via the NVIDIA NIM or via VLLM as well, too.
00:07:52And so it's a tiny, small model that you can use. It's not a generator. It's more of an extractor at the
00:07:56end of the day.
00:07:59So real quick, as well, too, we always want to make sure that we're building towards benchmarks
00:08:03that matter most to the enterprise space. So we want to make sure that both on the Pareto curve of
00:08:08accuracy versus performance, we'll make sure that we're going to be releasing world-class models to
00:08:12the ecosystem, too. So you'll see here, generally, is just a very standard benchmark of table extraction.
00:08:17I believe this one was RD TableBench. And we compare some popular open-source models here,
00:08:22and then we compare how our Neumatron Parse model does compared to that industry. And we continue to
00:08:26kind of strive to improve this as time goes on. So that, I think we believe we have a demo as well.
00:08:32Yeah, just press one. Yeah, there we go.
00:08:36Let me show you how easy it is to turn any agreement into structured usable data with Agreement Manager,
00:08:42which is a central repository of every agreement an organization has ever signed.
00:08:47Let's look at this. So when we look at the Agreement Manager view here, you know, we have an
00:08:54ability to see the entire list of agreements, but also go and upload a new agreement. So I'm uploading a
00:09:00new order form into Agreement Manager. As you can see, I can select from a computer, import from other places.
00:09:07The moment I select the agreement, it starts uploading and starts processing with AI. And just like that,
00:09:14you can see that the jobs engine has processed it. Let's take a closer look at this agreement. So when
00:09:21you go into the action, you can go and browse the file. Within seconds, Agreement Manager has extracted
00:09:27a rich set of metadata. Everything from key terms to commercial details are automatically structured,
00:09:33highlighted, highlighted. And immediately you can jump to that section where the details are found.
00:09:40Built-in goes deeper. This is where the NVIDIA model comes in, that it's extracted all the
00:09:45structured pricing data around this agreement. It goes in, breaks down these complex tables into order details.
00:09:55And as you can see, we can break it down, we can download all of this data. This is powered by the
00:10:02advanced parsing, leveraging NVIDIA's NEMOTRON model, turning every even dense tables into something that
00:10:07is instantly usable. And of course, you can take this data with you. You can see when we've downloaded into
00:10:13CSV, how we've structured all of it for your finance team, procurement team, even further analysis.
00:10:19All of it is also available through API. And that's how Agreement Manager has transformed agreements into
00:10:25actionable insights in seconds, leveraging NVIDIA. So I think, you know, what you saw there from a demo
00:10:33perspective, we've tried to shorten it. It's like we have a whole repository. What you see a list, we get
00:10:38customers which has thousand agreements to all the way millions of agreements within. But the big piece is,
00:10:44how do we understand and get that data that makes it very valuable to an end business user, right? A
00:10:50legal person, a procurement person, a salesperson who's doing a lot of the deals, or even a leader,
00:10:55right? Like a business unit, the CTO goes and asks, "What did we do?" This is how we are making each of
00:11:00the things a lot more structured. So we have our own proprietary agreement data model, which we are
00:11:06structurizing each agreement, but also at a whole organization level, and leveraging a lot of the NVIDIA
00:11:11things, we've been able to do a really good job, especially with all of those tables, like pricing,
00:11:16SLAs, and then make that available. And then we also have a robust kind of search that is on top of it.
00:11:23So when you think about what have we learned, right? When you think from a NemoTron plus DocuSign,
00:11:30one of the biggest things for us, we definitely have done a lot of different models for different
00:11:35purposes. So a purpose-built model for the job you're trying to do is a big, big part of how we've been
00:11:42thinking about, and that's kind of where we've been able to accelerate, bring things to market
00:11:46much faster. The second big piece around the model efficiency. So as Sean was talking about,
00:11:52the number of parameters, yes, context and stuff matters in a different environment for different
00:11:57things. For us, the lower kind of context basically also meant lower latency, lower cost, to deliver the
00:12:05scale that we are talking about. Last, around the faster extraction. So we ran this against a lot of
00:12:12the other open source models. When we think about how many tables can it extract per seconds,
00:12:18NemoTron was 20x faster, which helps us when we're talking about the millions and billions of scale that
00:12:25we're kind of serving for all of our customers. So a lot of it is like having that smaller purpose-built
00:12:31things is the way for an enterprise as an organization to go and leverage and then serve that from an end
00:12:39user perspective. And then what's next? So I'll let Sean talk through those. Yeah, so working with the
00:12:45DocuSign team has been awesome so far and we're going to continue to deepen that partnership as
00:12:50well over the next few months. So with them we started with the how do I extract as much possible
00:12:55information from a page and now we'll scale to how do I now find that page to begin with. So we'll
00:13:00start a little bit with the NemoTron, NemoRetriever effort and then of course we'll talk a little bit
00:13:05about the NVIDIA agent toolkit with them over the next few months and then actually start scaling into
00:13:10more production-scale agents then. Perfect. I think that's what we had. We have time for
00:13:15a couple questions and one in the room. Okay, I have someone there.
00:13:31So just to recap for everybody if you didn't hear it was the question is right like OCR is always a
00:13:49thorn in the whole process. So are we thinking about letting the go of that and starting from agentic
00:13:54from the get-go? I can talk from my perspective. So I think for us right like there are different
00:13:59use cases at different points in time. Many times if you are reactive you have a question and you're
00:14:05coming some of that can work dynamically at a smaller scale. The question is the latency. When I am querying
00:14:12at that scale of thousands I do need to have pre-processed have identified so that's one. I think the
00:14:18second big part of the use case for us a lot of times businesses want to use this data to do a lot of
00:14:24downstream work. So an example is a procurement team. This is my pricing table. I want to put it into
00:14:29Coupa to make sure when I'm paying that works. At that time there is like you know the agent is kind
00:14:33of helping but I can't do that on a one document by document. That said there is ways that we are
00:14:38compressing. That's kind of why Nemotron worked for us is like how do you do it from a layout understanding
00:14:43just for that purpose but I would like let you add. Yeah I think it depends on the use case a little
00:14:49bit. I think for this specific instance right you have petabytes of documents that you want to be
00:14:54queryable at some point right. So you are heavy on the compute at the upfront side with all the OCR.
00:15:00So you don't have to worry about it later on right. And there are some instances where people may
00:15:03upload a contract to begin with for Q&A and that's a very high that's a very low latency use case right.
00:15:08So you have a high throughput versus a latency use case. And in that scenario your different batch sizes,
00:15:12your concurrencies, your different techniques on how you process the document will be different.
00:15:16And where you spend that compute in that cycle will be changing between the different use cases.
00:15:22One more there.
00:15:35We do a lot of like more of what I call hybrid approach and a purpose built for like the needs
00:15:41and the use cases. So from a table piece it does kind of you know do the whole layout along with
00:15:45extracting. We still do OCR from a lot of other fields and metadata and the clauses like all of
00:15:51the text kind of things. So I think we had an architecture where we have a pipeline going through
00:15:56two different routes for that. As of all we have a blog out there how are we really solving this at
00:16:01scale across. And if you look at that there's a lot of different piecemeal modules and stuff together.
00:16:07So we this model is currently on FP16 but there are paths towards going on to FP8 and
00:16:19then the FP4 in the next few months as well too.
00:16:21What about the data center? Are you using the 16 or do you want that to download?
00:16:25We do use that and then we are also kind of using some of the older ones and that's the journey
00:16:30as a partnership is to kind of go tweak as you get more of the customers.
00:16:34Yeah so for this there are many techniques on how to improve the performance side right so quantization
00:16:38right so we're trying to move everyone to Blackwell right so that's why NVFP4 is the big thing now.
00:16:43As well as multi-token generation for this it's a VLM architecture right so your encoder decoder
00:16:48techniques can definitely be further optimized so not right now this model just generates one token
00:16:54at a time you can do multi-token generation of course too so there's plenty of performance things
00:16:57right now we're focusing on the accuracy side like are we adding value to the system and then from there
00:17:01we'll then push out that Pareto curve on the performance side.
00:17:03I believe they just deploy the VLM directly.
00:17:18It's coming from that agreement data that we've kind of extracted and stored yeah maybe I can chat with you offline and how our architecture kind of works fully as well.
00:17:28We're almost coming up on time there but I think that's kind of all we have but I think that's kind of all we have I'm happy to hang around in the back with more questions and good luck.
00:17:44With a lot of your challenges with a lot of your challenges with AI.
00:18:00Thank you.
00:18:00Thank you.
00:18:00Thank you.
00:18:02Thank you.
00:18:14Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video