Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind
AAI Engineer
컴퓨터/소프트웨어AI/미래기술
스크립트
00:00:00-
00:00:12- Hi everyone, thank you for coming.
00:00:14I know it's the fourth day,
00:00:15last session before the keynote starts again,
00:00:17and we are going to do something fun.
00:00:20We're going to look into how files
00:00:22are basically replacing Python.
00:00:24And before we begin, I would like to start
00:00:26with my favorite definition of what is an agent,
00:00:29from Simon, and LLM agent runs tools
00:00:32in a loop until it leaves the goal.
00:00:34And what we are going to do is we are going to build
00:00:36the same agent, the same GitHub PR review agent,
00:00:40in three different ways, and we are going to delete code
00:00:43on the way, each new version, less code,
00:00:45more files basically.
00:00:47Before we begin, I would like to quickly introduce you
00:00:51to the Interactions API, which is our new Gemini API.
00:00:54It's a unified interface for running models and agents,
00:00:58so you can use the Interactions API to call the Gemini model
00:01:01directly, or to call our new agents,
00:01:03which also comes with a sandbox.
00:01:06It supports server-side state management,
00:01:08background execution, so it's perfectly suited for all
00:01:11that's coming in the next years.
00:01:13And the capabilities, it's the same API for tool call,
00:01:18multi-modality understanding, multi-modality generation,
00:01:22so you always have the same interface.
00:01:24Might look very familiar if you're using other LLM applications,
00:01:27we really try to build something for developers which you like
00:01:30to use to build, and that's something we are going to do.
00:01:34So something a little bit different in the Interactions API
00:01:37to other LLM applications or APIs is that we moved away from
00:01:42this term-based conversation history to steps.
00:01:46So until, I would say, a few months ago, most of the applications were really turn-based.
00:01:51Normally, you had a user input and then a model output, a user input,
00:01:54a model output, which definitely works for a normal chat application,
00:01:58but as soon as you start to build agents, use reasoning model,
00:02:01we have more than just a user role and a model role, right?
00:02:04So we have different inputs, we have different types, we have reasoning.
00:02:08So we decided to make a cut, make a change, and build something really for agents,
00:02:14and that's what you see on the Flat Steps timeline on the right where you have a user input,
00:02:18then you have reasoning, you have a function call, you have a function result,
00:02:21and you no longer need to abuse the user role for passing back data from an environment.
00:02:27So, roughly a year, one and a half years ago, writing agents mostly meant writing a loop in Python,
00:02:35you needed to define a JSON schema, you needed to define Python functions,
00:02:39you needed to look at the output from the LLM, need to check if it was a function call
00:02:44or if it was a text response, and then needed to match it against the type,
00:02:48and then call the tool, look if you get an arrow, and then go back and forth.
00:02:54And let's look at some code examples on how this would look, and also run it,
00:02:59and hope that the demo gods are great to us.
00:03:03So I built, or I let Gemini build, a basic implementation of this Python loop.
00:03:09So we have our class, we have a run function which uses the interactions API,
00:03:15we have all of the weird complex parsing with function calling,
00:03:18with appending the arrows, checking if you get an arrow,
00:03:21and then we have the result again, and what we need, of course, for an agent is we also need a system instruction,
00:03:27so there's a separate file for the system instruction, very basic,
00:03:31you are GitHub PR reviewer, and then, of course, we need tools.
00:03:35And for tools, we needed to write those JSON schemas, specifications of description,
00:03:41exactly define which actions the agent can take, and then, of course, we need the implementation,
00:03:47in this case, using the basic GitHub API, just sending some requests.
00:03:52So we can run this, and basically, the main implementation is a very simple input interface,
00:04:03and we can say something like, "Hello," and, yes, we get back, "Hey, I'm an agent,"
00:04:10and then we, yes, ask it to review a pull request on the Gemini skills repository,
00:04:15and what we should see is, like, the agent should hopefully start soon sending function calls,
00:04:20function results, function calls, function results, but it's very limited to -- yes, great, it works --
00:04:27very limited to the tools we define.
00:04:29So if we ask the agent to do something which it does not have the capabilities to,
00:04:34it just says, "Hey, I cannot do this," which is unfortunate,
00:04:38but that's how we were building agents.
00:04:40Raw Python code, a lot of files, a lot of things which can go wrong,
00:04:44a lot of code to manage.
00:04:46So what happened afterwards, or what we need to do?
00:04:51We have, like, a token generation, we have the native function calling,
00:04:54and we must execute the loop, we must handle the tool routing,
00:04:57we must create a JSON schemas, we must write the Python code,
00:05:01we need to execute the Python code, we need to manage the state,
00:05:04so there's a lot of things we need to do to get an agent running.
00:05:07And then we got agent frameworks.
00:05:09There were many different agent frameworks which abstracted away
00:05:12some of that complexity.
00:05:14One example here is the ADK framework,
00:05:17where you have an agent class now which handles all of the tool loops,
00:05:21the function calling, the retries, the error handling,
00:05:24and it made it a little bit easier.
00:05:26We basically removed all of the boilerplate code,
00:05:30which we always needed to write for agents,
00:05:32put it into a framework, and help people build with it.
00:05:35So back to the demo, and same example, so we go into the 02,
00:05:43and what is very interesting, if you let me open both,
00:05:47so we still have our, we don't have our agent file anymore,
00:05:51so the agent went away.
00:05:52We still have our prompt, same system prompt,
00:05:56we still have our tools, in this case also no JSON definitions anymore,
00:06:00because those agent frameworks now use the signature of our functions
00:06:05to create those JSON schemas on the fly to provide the model.
00:06:09So let's stop our agent, now let's run our second agent,
00:06:17similar interface, similar prompt,
00:06:20and we should see a similar expected behavior,
00:06:23where we have function calls, we try to get the PR data,
00:06:26we try to get the diff, we try to get all of the code we need,
00:06:31and it works, and we wait for the agent to, yes, continue,
00:06:36but similar difficulty here is,
00:06:38I ask it, like, what's the weather in San Francisco?
00:06:46We should get back, hopefully, a result, like, hey,
00:06:49I cannot do this, I don't have access to the weather API,
00:06:51which obviously makes sense, because we did not define any tool,
00:06:54still very unfortunate, because we need to be very explicit
00:06:57on what our agent can do, and we all know nowadays
00:07:00that we just want to prompt something,
00:07:02and we wanted the agent to do whatever it takes
00:07:04to achieve that goal.
00:07:05So what is left for us to do?
00:07:08What does the framework solve?
00:07:09The framework solves the turn-taking loops,
00:07:11the routing, the execution mapping,
00:07:14the JSON schema creation for, like,
00:07:16the different function calls,
00:07:17but we still own the Python plumping,
00:07:19so we still need to write those tools with Python code,
00:07:22we still need to add specific rules or requirements
00:07:27to, like, make sure whatever we want the agent to do,
00:07:29and we need to provide the environment
00:07:32where all of the tools are running,
00:07:33where we want to host it.
00:07:35So what comes afterwards?
00:07:37Afterwards, hopefully, comes remote agents,
00:07:40and at Google I/O, we launched the anti-gravity remote agent
00:07:43on the Gemini API.
00:07:45The anti-gravity agent is powered by the same agent harness,
00:07:49which powers the anti-gravity IDE.
00:07:51Here, the same harness, very important,
00:07:53does not mean the same agent, because the anti-gravity agent
00:07:55is a coding agent at the moment,
00:07:57and the agent available on the Gemini API
00:08:00is a general purpose agent.
00:08:01So there might be different system instruction.
00:08:03There might be slightly different tools,
00:08:05because the Gemini API already has a Google search tool,
00:08:08so we use that what we have built.
00:08:09And, but very importantly, it comes
00:08:11with this new environment parameter.
00:08:13And this environment parameter here
00:08:15allows the agent to get access to a hosted, isolated cloud
00:08:20sandbox, where it can run tools, where it can run bash commands,
00:08:23and where it can save files.
00:08:24And those environments can be configured.
00:08:28So you can provide sources.
00:08:30And sources can be a GitHub repository.
00:08:32It can be a GCS bucket.
00:08:34It can be inline files.
00:08:35And of course, very important, we
00:08:37want to make sure that those agents are secured
00:08:40and cannot use our credentials in any way possible.
00:08:43So we created a network proxy around the agent sandbox, which
00:08:47basically injects the credentials when the agent makes
00:08:50a request from inside the sandbox to outside the sandbox.
00:08:54So the agent never really sees your credential.
00:08:56It just knows, hey, I can call the GitHub API.
00:08:59And then on the fly, we make sure
00:09:01that it receives the correct token, which you define.
00:09:03And you can also limit which domains the agent has access to.
00:09:06So if you want to restrict the agent completely
00:09:09on which network access it can-- or which website it can access,
00:09:12you just leave it blank.
00:09:13By default, the agent can access all because, I mean,
00:09:16it's a hassle if you first need to define where to go.
00:09:19So we try to stay simple.
00:09:20And of course, making an API call is nice.
00:09:23But we thought, hey, people want to reuse their configuration,
00:09:26want to reuse their agents.
00:09:28So we added the agents API where you
00:09:30can define your own custom ID.
00:09:32You have the same system instruction, the same base agent,
00:09:34the same base environment.
00:09:36And then you can create that agent.
00:09:38And then you can use that agent in the same exact way
00:09:40as we use Gemini models or as you use the Antigravity agent
00:09:43by providing the ID.
00:09:44So all of the existing code can be reused
00:09:47with your own custom agent, with your own custom tools,
00:09:49with your own custom credentials, environments,
00:09:52whatever you need for it to run.
00:09:54So let's look at how this will look for SS code.
00:09:58And as a demo.
00:09:59And OK, now 0.3.
00:10:03And what might be very obvious is that we no longer
00:10:07have a source directory.
00:10:08So the code went away.
00:10:11We have now an agents folder with an agents MD file
00:10:15with system instructions.
00:10:17So very similar system instruction.
00:10:19The only difference here is that we tell the agent, hey,
00:10:22you have access to the GitHub CLI.
00:10:25So we no longer create specific tools
00:10:28for reading files from a GitHub pull request,
00:10:31for accessing a GitHub pull request.
00:10:32We just tell the agent, hey, you have a GitHub CLI.
00:10:35You have a bash tool.
00:10:36You have file systems.
00:10:38Try to use it whenever you think it's important.
00:10:41And since we don't have the CLI installed,
00:10:43we have a very basic bash script in this case, which
00:10:45checks, hey, if the GitHub CLI is installed, please use it.
00:10:48If not, download it and install it on the first turn.
00:10:51So we go into our terminal.
00:10:54And we run our agent here.
00:10:56In this case may be important.
00:10:57I use a stream version, because otherwise we
00:10:59would wait a few seconds and we would not get anything back.
00:11:05So same prompt.
00:11:07And we should soon see our function calls and function
00:11:12results coming in.
00:11:13So in this case, since we run inside a sandbox,
00:11:16the agent first explores the sandbox to really make sure, hey,
00:11:19do we have this GitHub CLI installed and then tries to run it.
00:11:23It did not find it on the first turn.
00:11:25So it installs it.
00:11:26And then we can see the agent doing its work.
00:11:28And in this case, it's not using the predefined function
00:11:31calls.
00:11:31It's using the GitHub CLI.
00:11:33And it's already existing knowledge about how it works.
00:11:36I have a bash tool.
00:11:37I have access to the file system.
00:11:39And I do all of that work to see or to review the pull request.
00:11:44Let's wait a little bit.
00:11:47OK.
00:11:48And I think the amazing part here is if we ask the same
00:11:52question as before, what's the weather in San Francisco,
00:11:59we should hopefully see that the agent tries to use--
00:12:02ah, it uses Google search in this case on 2nd of July.
00:12:06Let me quickly check here.
00:12:07That's today.
00:12:08And we have around 20 degrees Celsius.
00:12:11And it works.
00:12:12So the agent became more of a general purpose agent.
00:12:15And we don't need to specify all of the tools.
00:12:18we basically trust the model on understanding, hey, I have a
00:12:21specific set of very atomic general purpose tools to solve my
00:12:25task or a task for the user.
00:12:27And if we look at the code for, like, the input or, like, the--
00:12:34sorry, the interface, we have our sources here.
00:12:36So we have the bash script, which install the GitHub CLI.
00:12:40We have the agents MD file.
00:12:42And then we say, hey, you can use the GitHub API with credentials.
00:12:47So I want to access or use GitHub credentials in a secure way.
00:12:50So I created a token for the API and also for GitHub.com.
00:12:54Since you need both URLs, one is used for the Git commands.
00:12:58The other one is used for HTTP commands.
00:12:59And then domain all is basically, hey, in addition to the GitHub URLs, you can use all of the web.
00:13:05But you don't have credentials for it.
00:13:06And then it's a simple, single API call to the anti-gravity agent with our user input with the environment.
00:13:14And then also with the previous interaction ID that we keep the multi-turn going.
00:13:18And that's all it takes.
00:13:19And it's a single API call on the back-end side.
00:13:22We start that Cloud Sandbox.
00:13:24We load the agents MD file and the skills from the environment provided to the model.
00:13:28And then the model between the API and the Sandbox does all of the looping, calling the function,
00:13:34returning the function results, calling the function, returning the function results.
00:13:38And that is all it takes.
00:13:39So where does it leave us?
00:13:42We no longer need to execute loops.
00:13:44We no longer need to do two routing.
00:13:47We have a server-side conversation and session state.
00:13:49So we only need to provide new inputs.
00:13:52The context window and the compaction is also automatically managed by the agent.
00:13:56So if we continue our conversation at a certain point, the context is compacted.
00:14:00And we can continue without the need to manage anything.
00:14:03And we also get an isolated remote Linux sandbox, which we can use to run our code.
00:14:08So what is still left for us?
00:14:11We need to define instructions.
00:14:12We need to define rules, behaviors in an agents MD file.
00:14:16We need to provide capabilities or context and skills MD.
00:14:19And we need to own the evals.
00:14:20So all of the heavy lifting, the infrastructure management, all of the same code, which probably
00:14:26everyone has written of us here like 20 times is no longer needed.
00:14:30And you can start really building your product instead of needing to rewrite the same code over
00:14:34and over again.
00:14:35And very important is like, hey, that's great, but what about extending?
00:14:40And I think looking into how extending previous agents to like those new agents work, it's
00:14:47very obvious that previously, if we want to do like some kind of security scanning on a pull request,
00:14:52we would need to define or write a Python function.
00:14:55We would need to understand, okay, which CLI tools do we need to use?
00:14:58We need to define a new function schema.
00:15:00And then we needed to add it to our tools, need to run it.
00:15:03And then so there's a lot of things we need to do on agents powered by files.
00:15:07We write a skills MD file, maybe with some additional information on which CLI tool to use,
00:15:12or maybe provide a CLI tool inside the environment, and then we extended the capabilities.
00:15:16We don't need to change our code.
00:15:17We just provide more files to the agent, and the agent decides on what we want to do.
00:15:22And I like to bring up some very good examples.
00:15:25So at AI engineer in Europe, Curso did a great talk on how they replaced roughly 12,000 lines
00:15:31of TypeScript code with 200 lines agent files to create something similar.
00:15:35So they had a very hard-coded code orchestration for doing Git work trees,
00:15:41and they were able to replace it with just a skill and markdown files.
00:15:46And there are more, I would say, better lessons of agent engineering.
00:15:49Manus has refactored their harness five times in six months last year.
00:15:54LangChain has re-architected their OpenDeep research three times a year.
00:15:57And then also Wersel has removed 80% of their tools to achieve fewer steps,
00:16:02faster responses, and better accuracy.
00:16:04So there's an obvious trend that with better model capabilities, we can remove orchestration code.
00:16:10But if your harness is getting more complex as the model improves,
00:16:14you are most likely overengineering your harness.
00:16:17So if you struggle with model improvements and adding new capabilities which lead to more complexity
00:16:22and more code, you might need to rethink a little bit on how your agent harness looks.
00:16:26And so where does it end up?
00:16:32Agents are just files.
00:16:34We write markdown files to extend capabilities.
00:16:36Agents can learn from those, can create their own files.
00:16:40So if you have a session and tell the agent to remember something, to take notes of rules of preferences,
00:16:46you just write it to disk, and then can reuse it in the later session.
00:16:50And you can also externalize context.
00:16:52So if you have a very long-running session, and during that session you notice,
00:16:56hey, maybe I want to additionally work on another feature, you can like just write that information,
00:17:00that handoff to a file, and like tell the agent to later pick it up.
00:17:04And so what are the takeaways?
00:17:07We should not fight the model, like we should stop micromanaging the execution paths,
00:17:11provide general tools to the agent, and let the model explore, reason, and discover the right solution,
00:17:16own what is yours, meaning focus on your domain instructions, focus on the workflows,
00:17:21focus especially on the evals, define clean tools, and verify the outcomes, and really build to delete.
00:17:27Like we have seen in the past many, many times, the better the model gets, the more code we can remove,
00:17:32and the more things we need to change.
00:17:34And obviously we all want to benefit from better models.
00:17:37So the things for you to get to do on Monday, you can scan that QR code,
00:17:42which brings you directly to AI Studio, where you can immediately try out the anti-gravity harness,
00:17:48so you can already start prompting it, it will start your own custom sandbox.
00:17:52If not, start or create your API key.
00:17:55We are currently working on a free tier for the API, so hopefully you can start exploring faster soon,
00:18:01and then definitely start building files and skills.
00:18:04And that's it, thank you for coming.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기