500 Skills, Zero Fine-Tuning: LinkedIn's Playbook for AI Agents — Ajay Prakash, LinkedIn
AAI Engineer
Computing/SoftwareManagementInternet Technology
Transcript
00:00:00Hey, everyone. Good morning. Thanks for being here. I see people are still coming, but yeah,
00:00:21my name is Ajay and I am a software engineer at LinkedIn. Today I'm going to be talking about
00:00:30how we are doing context engineering to improve the performance of coding agents at LinkedIn.
00:00:39Okay, imagine you are a software engineer in a big tech company and your products are being used
00:00:47by millions of users on a daily basis. You happen to be on a team which owns a set of very critical
00:00:55services and you are on call. You get an alert saying that there is an error spike in one of
00:01:03your services. By the time you are trying to figure out how to deal with this issue, you take the link
00:01:13to the alert, give it to a coding agent like Cloud Code or GitHub Copilot. While you are trying to
00:01:21figure out how to deal with the issue, the coding agent is working in the background. It will fetch
00:01:27the instructions on how to debug such issues in your company and identifies that based on that
00:01:33instruction, it identifies that this alert is happening in a specific service. Then it fetches
00:01:39instruction and context on how to debug that particular service. Based on those instructions,
00:01:47it will take actions like fetching logs, metrics, and then it uses those logs to identify the root
00:01:54cause of the issue. It identifies based on the error logs where the issue is happening.
00:02:01It doesn't just find the root cause, it also figures out the steps to mitigate the issue. Once it finds all the
00:02:08details, it summarizes and gives it to you, saying this is the error and this is the issue and this is the
00:02:15actions that you need to take to mitigate. Once you confirm, it also goes ahead and takes those actions
00:02:21on your behalf to mitigate the issue. It doesn't just stop there. It updates your incident management system
00:02:29with all the details, error metrics and dashboards, etc. And also it checks out the code and creates a PR for
00:02:36you to fix the root cause of the issue. All of this happens in a matter of a few minutes, which would have
00:02:43easily taken a few hours if you were to do it manually. This is not fiction, so this is how teams at
00:02:50LinkedIn are using coding agents as effective co-workers with deep understanding of LinkedIn's
00:02:57internal systems and code to help the teams be really productive.
00:03:05And this is possible because of a system that we built called as contextual agent playbooks and tools
00:03:12at LinkedIn. And today I'm going to talk about why we built the system, how we built it, and what are our
00:03:19learnings from the success.
00:03:23To understand why we built the system, we have to go back to the early days of coding agents, right?
00:03:30So just like any other company, even at LinkedIn, we wanted to use the coding agents to be for our
00:03:37engineers and everyone to be really productive with the AI. So we started using, started giving
00:03:44these coding agents to all of the engineers, and the problem was the coding agents doesn't really,
00:03:51or the wide coding doesn't really work in a large enterprise like LinkedIn.
00:03:59So the biggest problem is the coding agent or the LLMs are trained on open source repos, right?
00:04:06So they don't have the context of how we are mature code bases at LinkedIn or our internal frameworks,
00:04:15our internal systems. So what used to happen was the engineers used to do wipe code or try the agentic
00:04:23coding, but because the agents lacked context, they used to hallucinate and get stuck in between, or
00:04:33even more dangerous, they used to make up things which is not correct. So the engineers had to prompt these
00:04:40agents manually to do the right thing, which used to take more time than the manual coding itself.
00:04:48So a lot of people, a lot of engineers went back to manual coding. So coding agents was not effective.
00:04:56To understand the problem, to get more perspective, so if you look at the LinkedIn stack, we have over
00:05:04thousand repos, which make up thousands of microservices and apps. And we have a lot of, all of these apps and
00:05:13services are built on a lot of internal frameworks and libraries. And we also have a lot of custom built infra.
00:05:21For example, we have our own databases. We have our own experimentation and tracking platform. We have our own
00:05:28configuration management system, which is purely internal to LinkedIn and coding agents doesn't have any idea about them.
00:05:37And engineers go through a week long boot camp whenever a new engineer joins. So just to get familiar with these
00:05:44systems.
00:05:49So we looked at this problem and we asked ourselves the question, how can we make any coding agent like
00:05:58Cursor or Cloud Code or GitHub Copilot understand our LinkedIn's internal systems so well that they can
00:06:08ship the code that our engineers can trust. By trust, I mean the code should be correct. And also,
00:06:16the quality of the code should be as good as it is written by an actual engineer.
00:06:22So that is the bar we set up and wanted to figure out how do we get there.
00:06:30So in early 2025 last year, so Anthropic released MCP and it quickly became the
00:06:38standard for industry standard for building tools to the agents.
00:06:43We leveraged that and pretty early on, we built our own internal MCP.
00:06:49And the first tool that we built was code search.
00:06:54So we have a pretty sophisticated code search system at LinkedIn where engineers can go and search for
00:07:01code. It will search for any code across thousands of repos using keywords and custom filters and rejects,
00:07:09etc. So we made that available to the coding agents via MCP. This was a really powerful unlock because now you
00:07:18don't have to manually figure out how to do better search. The agent, you ask a question,
00:07:24"Hey, how do I set up a particular thing?" And the agent can use the code search tools to figure out the right
00:07:31way to do that. So we created examples of how we do things at LinkedIn and use that to give you an answer
00:07:37and also implement it based on its findings. This was really powerful.
00:07:43So we added more tools. We added docs, Jira, Slack, even connected to all of our data platforms
00:07:55and even feature flags. So every tool that we added to our internal MCP, it created more value.
00:08:03It's almost like a compounding effect because now an engineer can bring in the PRDs, product
00:08:11requirement documents, and design docs, and also their Jira tasks, which has different contexts,
00:08:18and use all this to give to the coding agent to automate or help with their coding.
00:08:27But there was a problem. So you connect all these tools, but it's not enough. So even with a slightly
00:08:36complex workflow, the agents used to not do really well. For example, if you give a context, it is
00:08:46with the tools, the agent was able to answer basic questions and find code examples, but it cannot do
00:08:55a complete job reliably end-to-end. The main problem was, to do a specific job end-to-end,
00:09:03it needs to have a lot of tribal knowledge, right? So all of, for example, how to fix a particular error,
00:09:11or how to configure, how do you debug a particular error log, right? So all of this knowledge, even
00:09:19though you have access to the tools, it is scattered across a lot of different surfaces. For example,
00:09:24docs, docs, wikis, and Slack conversations, etc., and most of the times, you may have experienced that
00:09:33docs and wikis might be outdated, written, and there might be like duplicate docs, right?
00:09:39So the problem is the agents, even though they have access to the tools, they used to get lost.
00:09:45The second problem was context overload. As agents use more and more tools, their context gets overloaded,
00:09:55which means every tool output, it takes up space in the context, which will eventually cause the agent to
00:10:02compact its, while it is working, compacts its context, which causes it to lose some of the information.
00:10:10Then it has to do all over again. And the third problem was, even though, even if the agent was
00:10:19able to figure out all these details, it can, it doesn't have a way to retain this information,
00:10:26it doesn't have a durable memory. So every time an engineer asks the agent to do a certain task,
00:10:33they have to start from scratch.
00:10:37So how do we solve this problem? So we give these instructions right away, right? So we built a system,
00:10:44we invented a system in early 2025 called as Playbooks, where we not only provide the tools
00:10:52to the agents via MCP, we also allow the agents to access these instructions and prompts via MCP.
00:11:01We call it Playbooks. And Playbook, it just appears just like any other regular tool.
00:11:10And they have names and description on what it does. And the agent can decide to invoke that Playbook,
00:11:17just like any other regular tool. And when the Playbook is invoked, the instructions and the context
00:11:26within that Playbook are returned as the tool output to the coding agent. So that way, the agents have
00:11:32both tools, as well as instructions on how to use tools to set up or perform a task, right? For example,
00:11:42if the engineer goes and asks, like, how do I set up an Airflow DAG at LinkedIn, the agent will first
00:11:49decide, okay, so I have a Playbook for creating that specific task. And it will use that first effect,
00:11:58uses that Playbook to get the information. And then it calls the necessary, follows that instructions and
00:12:05calls the relevant tools to get the job done. This was really powerful.
00:12:13Mainly because now anyone at LinkedIn can go ahead and create a setup of Playbook and check it into a
00:12:20repository and make it available for everyone else at LinkedIn.
00:12:27So as people started creating more Playbooks, so we wanted, so this is one of two foundational principles
00:12:37we want everyone to follow when creating a Playbook. The first one is a Playbook should be self-contained,
00:12:45which means it should do a very specific task. Only, for example, if it is for setting up an Airflow DAG,
00:12:53it should be about the instruction and the construct should be about one specific task. This helps the
00:13:00agents pick the right Playbook for the right task. And the second most important one is to break a big
00:13:08Playbook into multiple smaller Playbooks. So this has, and reference those smaller Playbooks from a bigger
00:13:16Playbook. This is a really powerful principle because just like, so it has two main advantages, right? So
00:13:26the first one is reusability. So if you have a small self-contained Playbook, it can be used from multiple
00:13:33reference from multiple Playbooks. And if you, the another big advantage is progressive discovery of
00:13:41context, which means the agent only when it needs to read a smaller Playbook, instead of reading the entire,
00:13:48all of the Playbooks at once, it can progressively go and read the Playbooks as it wants. So this is the same
00:13:54concept as skills as well. So Playbooks are very similar to skills, but we developed this entire
00:14:01system around Playbooks even before skills was a thing. And Playbooks are a little bit more nuanced
00:14:08because it helps us, it helps us seamlessly capture all of the organizational context
00:14:16and serviced via MCP without much of a setup.
00:14:23And another cool thing about this Playbooks is the self-improving loop. So you have engineers creating
00:14:32these Playbooks and checking into the repository. And one of the main problems with any knowledge base is
00:14:39it gets outdated. The biggest problem is how do you keep the context fresh, right? So the great thing about
00:14:46agents is they can improvise. So we encourage the agents to, whenever they use a particular Playbook,
00:14:55at the end of the session, to identify the learnings. So any outdated information or any discrepancy or any
00:15:03missing information. And we also encourage the agents to figure out how to improve the Playbook and use that
00:15:11context to check it, to update the Playbooks, check out the repository and update the Playbooks and create a PR.
00:15:20And that once it gets uploaded, the Playbooks get updated, right? This creates a really seamless flywheel of self-learning loop.
00:15:33So what is the architecture of a MCP server looks like? So this particular system, we have one local MCP server
00:15:43and it is automatically installed on all of the LinkedIn laptops by default. So if you join LinkedIn
00:15:50and you get a laptop, it is pre-installed. And any updates to the MCP server or the Playbooks or the tools,
00:15:58it automatically gets updated every one R on all the laptops. And we have a concept of two local
00:16:07playbooks and central playbooks. So central playbooks are the playbooks which are cross-cutting
00:16:14in nature, right? So these playbooks apply for multiple repositories, not just one code repository.
00:16:23And then you have local playbooks, where these are the playbooks which are very specific to your code
00:16:29repository. And you can just have them checked in with your repo. And only when the coding agents are
00:16:39working in your repo, those playbooks will be automatically picked up. So this helps us scale
00:16:45the local playbooks which are very specific to repo without having to worry about changing the central
00:16:52repository. And also, this is one MCP server which is serving all of the playbooks and tools. So this
00:17:01helps us do a lot of central things like seamless authentication, telemetry, and that we can use
00:17:10for learning to make the whole ecosystem better.
00:17:16You may be wondering, like, how many tools and playbooks it can support, right? So this is a common
00:17:21problem with MCP. We cannot scale it beyond 30 or 40 tools without degrading the context or degrading the
00:17:32performance of the system. So what we do is, instead of surfacing all of these playbooks and tools
00:17:40through MCP, we replace them with three meta tools. So the first one is search. The agent first uses this
00:17:48tool to search for the relevant tools and playbooks using keywords and tags. So we also control the
00:17:56system instructions. So every coding agent is pre-configured with system instruction on how to use
00:18:03these tools and how to use the search really efficiently. And once it finds the right set of
00:18:09tool or playbook, it can then get the more details about that particular tool using get schema and then
00:18:16execute that tool or playbook. So this has allowed us to scale to thousands of tools in playbook.
00:18:27So this is the growth chart. So now we have over 8000 users daily using the system daily, using tools and
00:18:38playbooks. So we have over 1300 tools and over 600 playbooks.
00:18:46And it's not just engineering, right? So it is not just engineers, but also product managers, designers,
00:18:53TPMs. So across different functions, they are using the tools and bringing their playbooks
00:18:59to automate their workflows.
00:19:05So I'll leave you with this takeaway, key takeaways based on our learning. The first one is the system
00:19:13was successful because we thought about quality and reliability from day one, right? So even before
00:19:20creating an MCP server, we thought our fundamental principle should be how do we ensure not just
00:19:27productivity, but how do we ensure the quality and also reliability of the system so that it doesn't
00:19:33degrade as we move fast?
00:19:37And the second one was build the right infrastructure for agents. In a large enterprise like LinkedIn,
00:19:44it's not just enough to give all of the engineers all the latest and greatest tools and models.
00:19:52They are not very effective if you don't build the right infrastructure for the agents to operate
00:19:57within your enterprise.
00:20:01Yeah, that's my time. Thank you for attending and feel free to connect with me on LinkedIn.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video