Transcript
00:00:00[BLANK_AUDIO]
00:00:30.
00:00:53>> Hi, welcome everyone to another Purcell community session.
00:00:57My name is Jacob Paris, I'll be your host today.
00:01:00We do these sessions to highlight cool projects from community,
00:01:03from customers, and just anyone building cool stuff on Purcell.
00:01:08We're streaming this live O2X on LinkedIn, but
00:01:10if you want to participate in the chat, come to community.com/live, and
00:01:15you'll be able to see the session at the top of the page.
00:01:18At the end, we'll have some time for questions and answers, so
00:01:21feel free to drop any questions in the live chat along the way.
00:01:25I'd like to introduce our guest.
00:01:27We have Shane Smith from Saltbox Management.
00:01:30>> Hi.
00:01:32>> Hey Shane, how are you doing today?
00:01:35>> Doing great, how are you doing?
00:01:37>> I'm doing pretty well.
00:01:38Yeah, so you've got a demo for us?
00:01:42>> I do, I've got some content to walk through and
00:01:46a little bit of context and then a live demo at the end.
00:01:48>> Awesome, I can help you share your screen here if you'd like to get started.
00:01:53>> Sounds perfect.
00:01:57Okay, as that's pulling up, I'll just start with a quick background here.
00:02:04So my name, as Jacob said, is Shane Smith, I'm the CTO at Saltbox Management.
00:02:09Saltbox is a services firm.
00:02:12We actually spend most of our time implementing Salesforce
00:02:16technologies for our customers.
00:02:18And over the last couple of years, we have been building a product for ourselves.
00:02:24And as we've been doing that, we are going to the point where we're starting to roll
00:02:28it out into our customers and to the world of larger standpoint.
00:02:31So today, I'm going to be talking about how we did that, specifically with Vercel,
00:02:37and some of the different technologies that Vercel gives us.
00:02:41As you can kind of see on the screen here,
00:02:42we're going to be talking about the AI Gateway as well.
00:02:44So getting a little bit into the problem statement here,
00:02:49we have a set of customers who are using Salesforce.
00:02:54And especially in this agentic world, these users are wanting to engage with
00:02:58Salesforce in a more natural language way.
00:03:02And so on the first use case here, the problem statement we're trying to solve is
00:03:06for the business users.
00:03:07They want to be able to talk to their ecosystem of Salesforce products with plain
00:03:10language and really get through the questions they have quicker.
00:03:16And on the second one, we are trying to solve for the builders, people who are
00:03:20actually configuring and building customizations inside of Salesforce,
00:03:23which largely is us.
00:03:25That's where we started on this journey is we wanted to make building a set of
00:03:28Salesforce easier with agents.
00:03:31And so those are the two problem statements that we're trying to solve for ourselves
00:03:34and for our users.
00:03:35And the context behind this is, doing this is actually pretty challenging.
00:03:41I mean, it's why people hire services firms like us and others.
00:03:45Because understanding the context of Salesforce, understanding the company and
00:03:49the context of what project they're trying to do, and try to move that into
00:03:54a production scalable code with best practices is really difficult.
00:03:58And so you can't just go out onto any LLM and
00:04:01just ask a question about Salesforce that probably will get you something, but
00:04:05it won't get you something that is production ready.
00:04:07And so that was the goal that we are trying to achieve.
00:04:12So as we're going down this path, we have a relatively small product team that we're
00:04:16trying to build this pretty ambitious product.
00:04:19And so we wanted to focus on the product and not so much on the infrastructure.
00:04:24We wanted the infrastructure to just work and give us the capabilities we need.
00:04:28And so that was kind of our challenge and our bet was that we could build this
00:04:33scalable platform on the Vercel infrastructure and enable us to go and do that.
00:04:40I'm going to talk a little bit about how we actually managed to do that.
00:04:44We kind of have three levels when it comes to kind of our high level architecture.
00:04:48At the very top level, we've got an XJS application that sits on the kind of
00:04:53the board facing UI capability.
00:04:56And that has all the normal streaming UI capabilities you would expect in
00:05:02an agent in this day and age.
00:05:04And then the second layer of that we have our edge and server capability here.
00:05:09We use the AI SDK, specifically version six now, to control the agent and
00:05:16have the agentic loop and all the tooling that is built into that.
00:05:21If you're not familiar with this SDK and you're getting into the agent world,
00:05:25I'd highly recommend going in using that.
00:05:28It gives you the ability to use any LM across the board.
00:05:34It kind of agnostically and provides kind of the framework and
00:05:37scaffolding you need to build the agent loop and
00:05:40really give it some powerful tools through that process.
00:05:43And the third layer here is the actual AI, so plugging into whatever model we want.
00:05:51And that was really what we were deciding when we first went down this path,
00:05:56is we wanted to understand what was the right model to use.
00:06:00And our verdict was that there isn't always one right model to use in
00:06:04every situation.
00:06:06And so we wanted to build it in a way that is agnostic, that we could plug and
00:06:10play the right model for the right situation as new models come out.
00:06:14Plug those in as well without huge scaffolding changes behind the scenes.
00:06:20So we chose the AI gateway for that, and
00:06:23I'll talk a little more about that in a moment.
00:06:25Then we have kind of our background or backing services,
00:06:30as you can kind of see at the bottom, that support the rest of the infrastructure.
00:06:35So kind of going into the AI gateway a little bit more here,
00:06:38you can kind of see on the left hand side is just a pseudo script here of what this
00:06:43would look like, but you essentially have one line that says,
00:06:46I'm going to plug into the gateway.
00:06:49And once you have that, you have the gateway and
00:06:51you can essentially use whatever model you'd like.
00:06:55So in a lot of cases, we use Opus 4.6, but in some cases, we use Sonnet and
00:07:00GPTs and some Grox in different situations.
00:07:04And that's really one of the powerful things about the AI gateway is that
00:07:08you can use whatever model you want, you're not locked in,
00:07:10you're able to have that flexibility.
00:07:12And what this allowed us to do is have some pretty intuitive
00:07:18based routing here around complexity and type of questions.
00:07:22And so we have this classifier in here that allows us to determine how complex is
00:07:27that question the user is asking?
00:07:29Are they asking for a simple response to a hello message?
00:07:32Or are they asking something more complex like trying to build an entire flow for
00:07:37the user?
00:07:38Depending on that, we might route it to different models.
00:07:40And that allows us to have fast responses,
00:07:44choose the right level of complexity on the model we provide it, and
00:07:49give the right response to the user.
00:07:50And in today's age, kind of around the experience on the agent,
00:07:59there's really two different capabilities that most users expect nowadays.
00:08:04And that is something that's streaming back to the user so
00:08:07you can see what's happening in real time.
00:08:09But also what actually happens when you have a really complex
00:08:14request that might take more than a couple of minutes or tens of minutes or hours.
00:08:20And so where we started out from a product perspective is on the streaming side.
00:08:23So that's really where AI SDK shines, it starts kind of on the streaming side.
00:08:28And on the left-hand side, you can kind of see a sample questionnaire,
00:08:31generate user stories from a meeting.
00:08:33That was one of our first use cases as we were building this tool,
00:08:37is taking all the context and providing user stories for developers.
00:08:41But as we started to get into more complex requirements like spinning up an entire
00:08:45B2B storefront in Salesforce or designing an experience cloud site with custom pages,
00:08:51those take quite a bit longer than a few minutes that can be streamed to a browser.
00:08:56So as we started to mature in the Vercel platform,
00:08:59we started to have this single experience that the user has in the UI,
00:09:05but be able to both stream and spin up background processes,
00:09:09depending on the complexity.
00:09:11And that's all supportive inside of the Vercel infrastructure,
00:09:14both on the actual platform itself, but also using some capabilities around
00:09:19sandboxes and a few others that I'll talk about in a moment.
00:09:24So from a tool perspective, this is I think one of the superpowers of using
00:09:28something like AI SDK is you have really just a framework that you can plug into.
00:09:34So we started out with just a few tools and then over time have built more and
00:09:39more tools in these categories that allow us to really have hands to interact
00:09:44with the user and Salesforce environments as well.
00:09:48So you can kind of see we've got document generation, we've got search,
00:09:52we've got stories, integrations, and primarily this Salesforce capability here,
00:09:57which allows us to understand the environment, validate it, query it and
00:10:01make deployments into those environments as well.
00:10:05So really brings to life this agent outside of just going straight to an LLM.
00:10:10So getting into kind of the agent has hands, right?
00:10:16How does that actually work?
00:10:18Well, inside of the Vercel ecosystem, this is actually one of the newer
00:10:22capabilities that they've been rolling out is this ability to have a sandbox.
00:10:27So if you think about, if you're familiar with Salesforce and
00:10:30you think about how do you interact with Salesforce, a lot of times developers and
00:10:34kind of the technical side of the house, they will use Salesforce's CLI.
00:10:39And that allows them to, at a terminal level, interact with Salesforce,
00:10:43spin up sandboxes, spin up B2B commerce storefronts, do deployments.
00:10:49And that makes it really easy to interact with Salesforce.
00:10:51But that's really difficult in an ephemeral browser, right?
00:10:55You log into just any sites and you have to log in every single time and
00:10:59it makes it difficult to interact with Salesforce.
00:11:01And so we started to leverage this Vercel sandbox capability where we
00:11:06actually allow the user to authenticate to their environment.
00:11:10Spin up a sandbox that would allow us to actually interact with a file system and
00:11:16would allow us to actually deploy our code that we're making or files that we're
00:11:20trying to do into that sandbox and then use the SS CLI
00:11:26in that sandbox to push those files into Salesforce.
00:11:30And this is maybe a use case specifically in the Salesforce ecosystem because they
00:11:35have these concepts of scratch orgs and sandboxes, but I think as we
00:11:40get more and more into kind of the coding side of the agentic ecosystem,
00:11:43this idea behind sandboxes is absolutely critical because
00:11:49it requires or it provides us the ability to have this file structure
00:11:55that agents and LLMs are traditionally built on.
00:11:58So I think it gives you a lot of flexibility when it comes to
00:12:01what you actually can do.
00:12:06And then kind of wrapping up kind of the pillars here that we have been building
00:12:10on V0 has been a huge capability for our team, right?
00:12:17We actually use it on two different sides.
00:12:18We use it from a product perspective as we're building our product, Saltbox One.
00:12:23We also use it as a services side when we are building Salesforce.
00:12:27And so what we've actually done is we've plugged the two of them together.
00:12:31Where if you're in S1 and you have the context of what you're trying to do,
00:12:35you have all the Salesforce information, you have the ability to actually go and
00:12:41gather all that about your Salesforce org, we have a tightly coupled integration with
00:12:45V0 where you actually can pass over that information to V0 as context.
00:12:51And that allows V0 to actually help you generate code.
00:12:54So it's not just for React, but given the right context and
00:12:58the information there, it can do things like Apex and LWCs and flows.
00:13:01As we've been able to kind of couple those together and
00:13:05create a really nice seamless integration there.
00:13:09So if we had to kind of pause for a minute and say, what would we tell another team?
00:13:14These are kind of my four learnings here.
00:13:17One is use a gateway, stop arguing about those models,
00:13:21start choosing the right models for the right tool on the right occasion.
00:13:25That gives you a lot of flexibility and capability there.
00:13:28Streaming is just the start.
00:13:31That's where a lot of these applications start that are kind of more startup
00:13:35applications in the agentic world.
00:13:38But really you need kind of that persistent background running capability as well.
00:13:42And so that's the ceiling of where you're going to.
00:13:45Third is that sandbox as I was just talking about here.
00:13:48Making it real, giving the CLI access to your agents and
00:13:52that really gives you a lot of superpowers there.
00:13:55And then last one is find the right platform.
00:13:57So for us, that was building on Vercel and
00:14:01allows us to not focus on the infrastructure.
00:14:04And allows us to focus on the features and
00:14:07the capabilities that we're trying to roll out in Saltbox One at scale.
00:14:12So with that, I'm going to jump a little bit into the demo here and
00:14:19go through kind of a conversational Salesforce experience.
00:14:24Talk a little bit about the story generation and
00:14:26kind of see the agent in motion here.
00:14:29So give me one moment as I switch over to that screen here.
00:14:34Okay, cool.
00:14:38So jumping into Saltbox One here, this is one of our demo orgs here.
00:14:44But you can see that you've presented on the left hand side with what we call our
00:14:48projects and our projects give us different contexts.
00:14:51And then in the center, we have our new conversation pane here.
00:14:56So we can kind of focus in on here.
00:14:58We can see all of our different projects as we scroll down.
00:15:01This allows us to load in the right context into the conversation.
00:15:05We have our tools across the bottom here,
00:15:08which are where all of our integrations are passed into.
00:15:12And then down below, we actually have the ability to add context.
00:15:16We actually can drop in a meeting in here, a user story, an artifact,
00:15:20a Salesforce org, or something specific in that Salesforce org.
00:15:23That allows us to have a really rich conversation like that.
00:15:27So I'm actually going to start with a more complex one to begin with.
00:15:33And then we're going to switch over as that's running to something a little bit,
00:15:37or something I've actually ran ahead of time so we can talk through that.
00:15:40So in this example here, we want to create a screen flow in Salesforce that allows
00:15:45us to enter case comments and we want that to be able to give us confirmation screen.
00:15:51So not a super complex requirement, but it does require that you know flow and
00:15:56that you know how to configure that and you know how to get there and
00:15:58you know how to debug it.
00:16:00So we're going to allow our agent to actually go and work on that as we go and
00:16:03talk about some of the other conversations.
00:16:06And before I hop away, you can kind of see it's retrieving some of the context files.
00:16:09That's some of the secret sauce with Softbox One is we have a lot of
00:16:13information about Salesforce and how to do it and
00:16:16the playbooks to make it actually work correctly here.
00:16:19And you can actually see that it was pretty quick and how it came back and said,
00:16:23hey, good news, I checked your instance.
00:16:25There are no automations on the case comment right now and so we're clear to go.
00:16:29It's going to ask me a couple of follow up questions here.
00:16:32And I'll say yes, you can use the default org and
00:16:37I'll let that go, it'll start to put together a plan here.
00:16:41But as that's doing that I'm going to actually hop over into
00:16:44a different conversation here.
00:16:47So this is actually one of the most frequently used capabilities we have in
00:16:52our projects for our customers is the ability to take a meeting.
00:16:56So in this case, I have a demo brand called Halston.
00:17:00I recorded a meeting where we talked about some new requirements about
00:17:03building out in the Salesforce capability.
00:17:07And I am asking it a little bit about our org, I wanted to do it based on what's in
00:17:11there today and I wanted to do it based on what the out of the box Salesforce
00:17:16functionality is.
00:17:18So on the surface of a pretty straightforward request, but
00:17:21as you think about it, it actually is pretty complex because it is asking for
00:17:26the details of your environment and you have to know what's out of the box and
00:17:30what is custom to make the right decisions.
00:17:33So in this case, we asked that question and you can kind of see it come back with,
00:17:37here are the meeting requirements that came out of that discussion.
00:17:41And here's what you have today.
00:17:43Here's your data model, put together a nice mermaid diagram for us to look at and
00:17:46confirm this is what it is today.
00:17:48And then it actually starts to go into recommendations here.
00:17:52So here's how S1 would approach it, would break it out into these phases.
00:17:57It would start with phase one as a data model, store from visibility subscriptions,
00:18:02and then kind of give you some recommendations and some key questions.
00:18:06And I'm able to go back and forth with it.
00:18:07And so I've given it kind of this response of here is the answers to these questions.
00:18:12And at the end of the day, it's given me this nice document that's broken out
00:18:16the requirements into those phases with a lot of detail.
00:18:19And I can take this and turn it into code.
00:18:22I can turn it into user stories or any other requirements.
00:18:27So that's kind of one of the superpowers of self box one is we have built this from
00:18:31a services standpoint.
00:18:33So we know how to implement Salesforce.
00:18:35We know the best practices.
00:18:37We know how to approach it.
00:18:38And all that's baked into kind of this agentic context that we have given
00:18:43softbox one.
00:18:44So as we hop back over to our original conversation here,
00:18:49we can see that we've said yes to the default org.
00:18:52It's gone through all that context that is talked about,
00:18:55figured out how to best do this.
00:18:57And we can see that it has created this nice little card for us to review.
00:19:02Now, this card here is our way of controlling what the agent can do.
00:19:07So you want the agent to go and put together this plan, but
00:19:11you don't want it to run by itself.
00:19:13We still want the human in the loop here.
00:19:15And so we've gone ahead and put together this component where you can see exactly
00:19:19what is going to deploy and how it's going to deploy.
00:19:22And what you're able to do is go ahead and click approve and execute.
00:19:25And what that will do, of course, I'm doing a demo so it's going to not work.
00:19:29But what you can do is you can actually go and deploy that.
00:19:33You can click retry after you've gone back and forth with the agent and
00:19:37resolve that issue.
00:19:38And if it's deployed, you actually can revert back to the previous version if you
00:19:43didn't like that version.
00:19:44One last use case I'm going to hop up on the screen here before we do one more
00:19:50live one is just a quick review of the architecture of your org.
00:19:55So this is a really common question that we get asked all the time is
00:20:00help us understand the architecture or help us understand the technical debt
00:20:03that's in there.
00:20:04How do we get ready to actually go down this agenda path
00:20:09in the Salesforce environment?
00:20:10Those are all really simple questions, but really in depth questions as well.
00:20:15And so what you can do is you can ask a question like this and
00:20:18have it actually go through your license environments, key findings,
00:20:22put together what your object model may look like across opportunities.
00:20:27And then again, one of the things that we're going to always do in our response
00:20:30is to start to flag things that you may need to review.
00:20:34So in this case, we have documentation that it looks like,
00:20:38undocumented fields across all of our objects.
00:20:40We've got a lot of custom fields it looks like, recommendations on how to approach
00:20:45it, next steps on what we actually can dive into.
00:20:48So in this case, I could go ahead and say, I want to go and do that.
00:20:53And what it will actually do is start to narrow down its focus specifically around
00:20:57the B2B commerce flow and how everything starts to interact together.
00:21:02It's going to go ahead and make queries into the environments.
00:21:05Look at all the ecosystem around it and
00:21:07then pull together kind of a synthesis of the response and
00:21:10you can continue having that conversation.
00:21:12Now as that agents go ahead and running across,
00:21:16I'm going to do one more use case here just to kind of show one last example.
00:21:21So if we hop back over into a new conversation, what I can do is go ahead
00:21:27and drop in a confidence page that I have that was actually
00:21:32the output of one of our previous conversations.
00:21:35And this has that full long detail of exactly what that solution will look like.
00:21:39And I can quickly go ahead and say I want to create user stories or
00:21:43phase one in this document.
00:21:46And again, what this is really special is that it's pulling together not only
00:21:50the context of this document, but of your project, your Salesforce environment,
00:21:55our best practices, Salesforce ecosystem, and
00:21:59deciding what is the best way I should break up this phase one into
00:22:03manageable user stories, write the initial statement of as a user,
00:22:08I would like to with a few bullets of what the acceptance criteria actually is.
00:22:14This gives you a really nice starting place for those user stories.
00:22:18And once those user stories are actually generated, then you can kind of go into
00:22:22extra levels of detail in the back and
00:22:24forth of the agent if you'd like to kind of refine what those look like.
00:22:28So now that that one is running,
00:22:32I'll hop back over to our previous conversation where we just finished our audit.
00:22:36We went through a couple of queries, pulled together kind of the B2B audit flow here.
00:22:40You can see we've pulled together a pretty good assessment of what that looks like,
00:22:45and kind of recommendations on what the technical debt is or
00:22:48what we should actually go back and prioritize on.
00:22:51So you can kind of start to see how all of this pins together.
00:22:55As I hop back over into my example here,
00:23:00looks like my screen is frozen, screen one second here.
00:23:11Here we go.
00:23:14So you can kind of see in our last example, we pulled together these user stories.
00:23:18And you can see these user stories are pulled up on the left hand side with
00:23:22the details on the right hand side.
00:23:24And this gives us the ability to integrate into Asana or Jira and
00:23:27allows a kind of full development life cycle of what this looks like.
00:23:32So that's a quick overview of Saltbox One, how we use the Vercel ecosystem,
00:23:39how we use things like the sandboxing capability, the gateway capability,
00:23:44and V0 to pull together an application like this and
00:23:49really focus on the features and functionality for our customers.
00:23:53And not so much on the infrastructure behind it.
00:23:56So with that, Jacob, was there any questions that we want to hop into?
00:24:02>> Yeah, yeah, thanks for giving the cool demo.
00:24:06I had a few questions here.
00:24:08So first off, I was curious how the agents permissions work.
00:24:12Does S1 have its own permissions, like when you set up the app,
00:24:16it gets certain access to Salesforce Confluence and so on?
00:24:20Or is it based on the person who's asking the questions?
00:24:23Because that's sort of a tricky thing to do once you have all these kind of
00:24:26enterprise SSO integrations here.
00:24:29So I'm just curious what decision you landed on for that.
00:24:32>> Yeah, we actually went back and forth a little bit on this as we were thinking of
00:24:36that same question and where we landed was user based permissions.
00:24:40So when the user comes in, they have the ability to authenticate as their user.
00:24:45And so any action they do in Salesforce or any of the other applications is
00:24:50their specific user with either their OAuth or
00:24:53their API key depending on the platform that we're supporting.
00:24:57And this gives us the control of your user was the one that performed it.
00:25:03But then on certain platforms like Salesforce,
00:25:06we actually have another layer of protection as well,
00:25:09which is each time you connect a Salesforce instance,
00:25:12it starts as a read only connection.
00:25:15And then you have the ability to flip it to a write permission.
00:25:19And once you do that, you will always still see that approval screen for
00:25:23any changes that are going to be done in your Salesforce environment.
00:25:26>> Okay, so on the Salesforce side for auditability,
00:25:31any of the actions that the user did, they clicked.
00:25:37So when you had a write action, it popped up the approval.
00:25:39The user explicitly had to say accept and continue to get the agent.
00:25:44>> Exactly.
00:25:44>> On Salesforce, it would show as being done by the user.
00:25:49>> Exactly.
00:25:50>> Okay, perfect, perfect.
00:25:51I was also curious, how does the agent use Sandboxes itself?
00:25:57Like, did you have certain tool calls that implicitly invoke
00:26:02a Sandbox as like an implementation detail of themselves?
00:26:06Or does the agent have Sandbox tools and it knows there's certain things it
00:26:11must do in a Sandbox that that's where it has its CLIs or APIs and so on.
00:26:16I'm curious how you set that up.
00:26:19>> Yeah, we've actually got two main tools that use the official
00:26:25Vercel Sandbox and I'll talk about a separate Sandbox in a moment.
00:26:28So the two official tools is when you're spinning up a scratch org,
00:26:33we always spin up a Vercel Sandbox and we spin up that scratch org using the SF CLI.
00:26:37The second use case is when you're interacting with Salesforce to do a
00:26:43validation or deployment of something new into an environment.
00:26:48That's always done with the Vercel Sandbox with the Salesforce CLI.
00:26:51That's the easiest way in our opinion to do deployments and
00:26:54do other activities is to use the CLI and the Vercel Sandbox makes it really easy for
00:26:58us to install that and continue on with the existing Sandboxes.
00:27:04So we have those two using the official Vercel Sandbox and then we actually have
00:27:08a smaller in-memory Sandbox that every conversation has,
00:27:14as they have each turn on the agent that we use to pull together all the files and
00:27:20allow the agent to kind of understand what's in that in-memory Sandbox.
00:27:25So kind of two different versions of that.
00:27:26>> Okay, are you doing any sort of network
00:27:32restrictions on the Sandbox to limit what it has access to?
00:27:36What sort of security profile look like on that?
00:27:40>> On the Vercel Sandbox side?
00:27:42>> Yes, yes.
00:27:43>> Yeah, the Vercel Sandbox, I mean, they get spun up pretty blank.
00:27:47So we load in the files that we need to and allow it to just have access to that.
00:27:52So it's not like our code base by any means, it's like S1 will go and
00:27:56actually take whatever it's gone and put together the plan essentially.
00:28:02So let's take the flow example I showed, it would take those files.
00:28:05It would create the directory it needs that Salesforce is expecting when it does
00:28:10deployments, it would put those files into that directory.
00:28:13And it would then go and deploy from there.
00:28:14So really, it's pretty limited in what it has, just what we give it at the right time.
00:28:20>> We have a question from YouTube, what's the pricing model set up?
00:28:25How do you handle pricing?
00:28:28Is it usage-based, seat-based, or Soapbox?
00:28:32>> Yeah, it's a great question.
00:28:33So we are still in the process of finalizing that as we're rolling this out
00:28:38to our customers and trying to figure out the right way to doing that.
00:28:41We are currently at a user-based or a user-seat-based pricing
00:28:48with kind of limits inside of that tier of how many conversations you can have,
00:28:53how much data, how many interactions.
00:28:55So that's kind of where we're starting.
00:28:59But as everyone has experienced in this ecosystem,
00:29:02things change quite frequently and quickly when it comes to token-based tools.
00:29:07So that's where we're starting, is kind of that threshold of the user-based seats.
00:29:11>> Yeah, yeah, I think it's really hard to find the perfect model that kind of scales
00:29:16with your usage, with your customers, and with your own expenses as well.
00:29:20So here's some of that.
00:29:23Let's see, another question here.
00:29:24How do you decide what questions to route to each model?
00:29:29So there were certain things you were routing to Opus.
00:29:31There were certain things you were routing to Sonnet.
00:29:33How do you decide what those are?
00:29:35Do you use like an eval system to kind of data back it or just kind of testing manually
00:29:41and seeing, hey, Sonnet is good enough for these classes?
00:29:45What was your approach there?
00:29:46>> Yeah, it's an ever-evolving answer to that question.
00:29:52Where we're at right now is we have a deterministic classifier on the initial questions
00:29:57that helps us to understand, based on what the user asks, how complex is that question?
00:30:03Do they say things like plan?
00:30:04Do they say things like investigate?
00:30:07So using that, because those are quick and free, right?
00:30:12You're able to kind of pick those up pretty quickly.
00:30:15So we start that with kind of a, you know, Regex or deterministic-based approach.
00:30:20If we don't find a match there, then we lean into on an LLM classifier, and we go and say,
00:30:27okay, based on what's in here, if it's over a certain threshold of words and et cetera,
00:30:32let's pass it over to an LLM to actually classify how complex it is.
00:30:38That model's pretty quick, about 300 milliseconds to kind of do that classification of that type.
00:30:42And then it will help us route it to the right model, whether that's Opus, Sonnet, or another one.
00:30:47And then we kind of have a fallback, too.
00:30:49If things get too complex and the user, you know, asks the same question again
00:30:54or there's other rules we have in there, then it will fall back to our most powerful model
00:30:59for a minute threshold in order for the user to be able to have the right responses in their questions.
00:31:06Okay. And you're just using Haiku at the moment for that classifier for the routing model?
00:31:11Yep, for the initial one, yeah.
00:31:13Cool, cool.
00:31:15All right. So yeah, you're also a part of the v0 ambassador program.
00:31:20How has that helped you? Would you recommend it?
00:31:23Have you gotten value out of that program so far in building this?
00:31:28Yeah, definitely.
00:31:30I'd highly recommend it if anyone listening is interested in kind of being closer to the community.
00:31:36I've done a couple of events now for people in different areas.
00:31:41I did one a couple months ago or maybe one month ago out at Mexico to actually do an onsite
00:31:48and kind of walk through like v0 with a bunch of people there.
00:31:51Personally, I get a lot out of the v0 ambassador program
00:31:56because I get access to the product team members,
00:31:59but also other people who are just as dedicated in the space as I am.
00:32:03So kind of bouncing ideas off of each other has been really valuable.
00:32:08Cool. So yeah, if you have access to the product people directly,
00:32:13I'm sure you send a lot of feedback that way.
00:32:16Anything you want to ask for feedback out here in public to really put them on the spot
00:32:22and get the pressure going?
00:32:25I'll be nice. I think maybe the biggest one and we've talked about this
00:32:30is like just bringing more capabilities into v0
00:32:34when it comes to what developers are used to from an IDE local environment perspective.
00:32:41You know, we've over the last couple of months got sandboxing capability inside of v0
00:32:45and the ability to kind of, you know, spin it up and have like an actual environment there
00:32:49and that's a huge step in the right direction.
00:32:51I think as we continue going down that path, that will continue to get better.
00:32:55So I think just continue going down that path would be my shout out.
00:33:00Amazing. Yeah, it's been a long ongoing process.
00:33:02It used to be this very, very minimal little browser IDE
00:33:06and now it's kind of full VS code, Monaco integration.
00:33:10It gets better and better every month.
00:33:14That's right.
00:33:15All right. Well, I think that's all we have for questions here.
00:33:18So, yeah, I just want to thank you for coming on and doing such a great demo here.
00:33:24Where could people find you if they want to follow up or learn more about stopbox?
00:33:32I'm most active on LinkedIn.
00:33:34So if you want to go ahead and connect with me, put a message in there.
00:33:37So you watch the session.
00:33:39I'll know who you are and happy to connect with you and answer any questions or connect that way.
00:33:44Amazing. Yeah, we'll drop a link to your LinkedIn in the chat here.
00:33:48So, yeah. Anyway, thank you. Thank you so much for coming on.
00:33:52All right. Thank you.
00:33:55All right. Yeah. Thank you, everyone else, for joining.
00:33:59We will have another community session coming up soon here in...
00:34:05Let me check the calendar.
00:34:10I'm supposed to do this ahead of time.
00:34:15Oh, it's coming up on Thursday, but it's not actually added to our calendar yet.
00:34:20So, yeah, come back in a couple of days.
00:34:22We have another session going.
00:34:24Until then, we'll see you all in the community.
00:34:27Have a good day, everyone.