Transcript
00:00:00armin bronacher i don't know how to pronounce it in english but it's a german or austrian name
00:00:04anyway so it's fine i guess who is a member of the pi team published a pretty interesting article
00:00:11which is worth reading obviously that's why you find it below and this article i think
00:00:17has implications for all of us and it is about a topic which i personally did not think too much
00:00:23about until now so i want to share it with you now pi in case you don't know is a pretty great
00:00:30very minimal agent harness which you can totally make your own it has an amazing extension system
00:00:36and it starts minimal and you can turn pi into exactly the ai agent you want i'm a big fan i've
00:00:42been using it for many months and coincidentally indeed because i didn't know that the article
00:00:47would come out i have created and released a pi course a couple of days ago it's currently on offer
00:00:52because i got asked about it a lot it's an amazing tool and in this course i share all about the setup
00:00:57the configuration the core features built in how to make pies yours and extend it and yeah all that
00:01:03stuff so if that's interesting you can also check this out it's an amazing coding agent either way
00:01:08and this article is also pretty interesting now in this article armin talks about an issue you can see
00:01:14if you're using pi for example with your open ai subscription for example with a model like gpt 5.6 salt
00:01:22you can analyze your pi session logs so to say on your system so these are simply json l files which
00:01:30are created for each pi session you start and in there you find all the turns in that session all
00:01:36the messages sent so the message sent by you to the model for example so here that's the message i sent it
00:01:43and then also the responses you get back the results of the various tool calls that were made and so on
00:01:49that's all logged in there now why does this matter well of course when you use a coding agent like pi and
00:01:57it's not just pi that's in general how they work the idea is that the agent runs on your machine or
00:02:04maybe on a remote machine but typically on some machine you own or control and of course the large
00:02:11language model the brain behind the agent typically does not run on your machine but instead on some
00:02:18remote server served through an api for example by openai if you're using their subscription or their
00:02:24api or anthropic and so on so there is this communication between your machine and their
00:02:30server and we've all seen this when their servers were overloaded and down again and in order to have a
00:02:36real conversation and a session where the context builds up over time and the model can work its way
00:02:42through a problem by gathering more and more information and performing more and more steps
00:02:47in order for that to happen you need to manage the context of your session so you need to keep track of
00:02:56all the things that happened and that context here must be sent to the large language model because
00:03:03on its own by default every new request this large language model receives is just one request some
00:03:10text input where it can produce some text output now if it wants to call a tool that only happens because
00:03:17ai agents ai harnesses like pi do send tool descriptions along with your prompts so tool
00:03:26descriptions are part of that context in the end and are sent to the llm in addition to your prompt and
00:03:33other instructions so it's this package here that is being sent to the large language model with those
00:03:39requests and then the large language model can make decisions and can ask to execute a tool so it can
00:03:45respond with a tool call request but it's then in the harness on your machine for example where the tool
00:03:52is actually executed and i guess that makes sense if the model decides that it wants to read a file or
00:04:00write a file that file is on your system so it can't do it on the open ai servers and even if for some
00:04:08reason the file should end up on those servers even if that would be the case the model the large language
00:04:14model can only produce text that has not changed since the release of chat gpt they take text input they
00:04:21produce text output so all they can do which still makes them very powerful though is describe tool use
00:04:28intense you could say the model can describe that it wants to read a file but it can't do it on its own
00:04:35anyways that tool call request is then received by the agent or by the harness if we want to be precise and
00:04:41then the tool is executed on your system the file is read and then the result of that so the tool call
00:04:50result is sent back to the large language model and is also part of that session context because
00:04:56ultimately it's of course that session context that builds up with all those prompts with all
00:05:01those answers with all those tool call results and so on and that is always sent to the model so that's
00:05:08important to understand it's this session that's in the end sent with every turn you could say to the
00:05:14model so that it has more and more information available as this session builds up okay now but now
00:05:20what's the problem with that the problem with that is a session entry like this the assistant so the model
00:05:28replied with some thinking tokens you could say and that is a feature all these frontier models have these
00:05:36days they are reasoning models which means they don't just output final responses or tool call requests
00:05:43they can also output intermediate thought processes you could say thought processes that then lead to
00:05:51tool calls or to the final message they may send back the problem is that what you get these days by some of
00:05:59these models like gpd 5.6 sol on openai you only get a short summary of what it reasoned about
00:06:06and not the full thought process output instead the only details you get is a bunch of unreadable stuff
00:06:15here you get some encrypted content but the kicker is you can't decrypt that encrypted content only openai can so
00:06:24you're not getting the full thought process you're just getting a summary and some unencryptable
00:06:31encrypted content now of course when that session here when that context here is sent to openai with the
00:06:39next turn in the conversation then openai can decrypt that content on their servers so the thought process is
00:06:47not lost it's still there but it's invisible to you and why is that a problem that is one of the main
00:06:55things armin talks about in this article and again there are more interesting things in there you should
00:07:00read it but why is this specifically a problem well one problem i have with that is of course
00:07:07transparency simply you can't see what the model thought about you can't see if it evaluated some
00:07:15options you're not too happy about and you could then maybe tell early that it's going in a wrong
00:07:20direction you have no idea of what it thought about because it's transparent all you get is a summary
00:07:26so transparency is one problem related to that i have a problem with billing you are built for all those
00:07:36thought process tokens not just for that summary you see so you are paying for tokens you don't see
00:07:43you don't see if openai is billing you correctly now you don't have to assume that they're cheating on
00:07:49you but you have no way of auditing or controlling this and that's also not a great feeling paying for
00:07:57something where someone tells you yeah this is the right price not great but one of the biggest
00:08:04problems i have with that and that is also the main problem armin mentions in this article is that
00:08:09suddenly you don't own the session anymore i mean if that session as it is stored on your system
00:08:19no longer accurately represents what was going on it's not worth a lot and what do i mean with that
00:08:27this will work as long as we keep on sending this to open ai and keep on using this model it may also
00:08:34work if we switch to different models but that's already one part where we have to rely on
00:08:39the provider to ensure that this works cross model that this encrypted content can be decrypted when
00:08:45using a different model by them it absolutely will not work when moving to a different model by a
00:08:52different provider now why would you switch model in session well there are reasons for that one could be
00:09:00that open ai is offline again another reason could simply be that you maybe plan or discuss something with
00:09:09let's say gpt 5.6 sol but then you want to implement it with grog 4.5 because it's faster it's cheaper
00:09:17whatever so you may want to switch the model because of reasons like this and another reason could also
00:09:24be that maybe you worked on a feature a couple of months ago by now there are new models out
00:09:30suddenly you want to continue an old session because maybe you need to fix a bug related to the feature
00:09:36that was implemented in this session obviously you want to do this with a new model at the point of
00:09:41time where you want to fix the issue but maybe even when sticking to the same provider but using a
00:09:47different model this session doesn't work anymore or does not fully work anymore because maybe that
00:09:53content can't be decrypted anymore and it certainly as mentioned doesn't work if you switch to a different
00:09:58model provider and these may all sound like niche issues but combined they can become real issues
00:10:06and it is not great that you're losing and that i'm losing ownership of my sessions on my system
00:10:13now why does open ai do that and by the way it's not just open ai and fropping is the same
00:10:19they're doing this this encryption stuff here to prevent distillation or make distillation harder
00:10:25because they don't want those chinese or other companies to train their models on their models
00:10:32and i could make a deep dive here um about how weird this stance is uh given the fact that they
00:10:43trained their models on all the open internet essentially uh and beyond that but that's not
00:10:51the focus of this video there is a lot of hypocrisy here but this is not what i want to get into here
00:10:57but that is why they are encrypting it now this is not the only problem you could run into and not the
00:11:03only thing armin mentions in his article but it's one main point he makes here another interesting point
00:11:13is related to tool calls remember i mentioned that large language models make tool call requests
00:11:20and then it is your harness or running on your system that actually performs the tool call that reads a
00:11:27file and so on for some tools this could be skipped let's think about a web search tool web search doesn't
00:11:40have to happen on your machine and indeed outsourcing web search may sound interesting because it's not
00:11:47trivial if it happens on your machine you will most likely work with some third-party api or provider
00:11:55because you i guess don't have your own web search index you have no crawler that crawls all the websites
00:12:03right so if you want to have a web search tool or some web search capabilities in your agent you will
00:12:12likely use a provider and you can you can absolutely build your own web search tool on top of a provider
00:12:19like brave or google and a bunch of possible options there but what many of these model providers like
00:12:26google but also open ai on offer is that they have a web search tool and they offer you that when you
00:12:35send a request to a large language model they will use that tool automatically for you if needed by the
00:12:42request if the model makes the decision that it should perform a web search and then this entire cycle of
00:12:49the model requesting the tool call and the tool call being made and the results being analyzed all that
00:12:56would then happen on the open ai servers and what you get back for your initial request is not a detailed
00:13:02report of what happened is not a list of the searched urls along with the per page content but instead what you get back
00:13:11in the end is just a summary a summary of the web search results along with the source urls which you
00:13:18then can of course inspect on your own but you don't get all the details you don't see what actually
00:13:24happened what was considered what was analyzed in that web search request now obviously you don't have
00:13:32to use these capabilities but you may have turned it on by accident or you may find it convenient to
00:13:39outsource that and let them do it on their servers and there are good reasons for doing that but of
00:13:43course if you do use that feature for example here at the web search example you again lose some ownership
00:13:52over your session suddenly i don't know if i have a web search to call in there i don't think so but
00:14:01suddenly you just get a summary and not the details so again some information lost and if you would port that
00:14:09context here to a different model the model would only see that summary and not all the details from
00:14:15that web search as it happened so another important point here also important is this point here about
00:14:22compaction you know if you have a conversation with a model or if you're building up the context during a
00:14:28session simply it may eventually for bigger more complex tasks run out of context window space all these models
00:14:35have different context window sizes and depending on the task you're tackling and what's going on in there
00:14:41that context window may fill up if that happens a process called compaction kicks in which simply means
00:14:49the agent harness normally takes the current context so takes all the content in here and summarizes it
00:14:58creates a summary with help of a large language model and creates a fresh context window and puts that
00:15:05summary in there that's in a nutshell what happens it's a bit more elaborate than that but that's in a
00:15:10nutshell what happens now some model providers like openai offer a feature called server-side compaction
00:15:16which means they manage the context for you they can compact for you on their servers why would you do
00:15:23that well this compaction can be more powerful than the trivial compaction you perform on the client
00:15:30side with the summary they can store more complex data there simply not just a summary text but maybe a
00:15:37more complex object they can store on their servers and reference that from the new context so more
00:15:43information can maybe be kept and maintained um despite compaction happening so it can be a useful feature
00:15:50but if you are using that feature and it's not like you have to use it right that's important but if you
00:15:56do use it again suddenly you lose information suddenly that compaction information lives on the open ai
00:16:04servers and all you get back is a key to that compaction summary or an encrypted content so suddenly you no
00:16:13longer own that compaction summary you just own a key to some point in the open ai servers or to some
00:16:22database and you're on the open ai servers and that again means your session lost information a similar
00:16:30problem can occur if you are relying on open ai and again our providers have similar features
00:16:40on open ai's conversation state feature and armin mentions this too you know in these sessions
00:16:47we obviously keep track of everything that happened that's the idea that the agent harness
00:16:54manages that for us so that we can easily port this to different models and so on and of course also
00:17:00in case of pi to enable its amazing tree command and so on so the agent harness keeps track of the session and everything that happened in there on our system
00:17:08everything that happened in there on our system as i explained now you could outsource this open ai for
00:17:14example offers their conversation state api which means instead of manually keeping track of what happened
00:17:21and sending that to the model with every new request you let open ai handle it you can set a store property
00:17:29when using their sdk to true if you set it to false then they don't store the conversation you have to store
00:17:39it you have to store the session if you set it to true on the other hand and indeed it is set to true by
00:17:46default so you have to set it to false to manage it on your own the default is that they manage it for you
00:17:52if you set it to true if you set it to true then they actually will manage the conversation
00:17:58on their servers the context on their servers and they will give you a response id with every response
00:18:07and you can send a new request with just the next message in a conversation
00:18:13and just reference a response id you got earlier so whenever you get back a response you just store the
00:18:19response id and with your next message you reference that last response id and that way open ai on
00:18:26their servers knows to which conversation this new request is referring and they keep on managing that
00:18:32that context that that session um on their servers of course the advantage of this approach is that less
00:18:40data for example needs to be sent over the wire less data because if you're managing the context and
00:18:46the session then this entire chunk of information has to be sent to the open ai servers for example with
00:18:53every request your agent hardness makes obviously because otherwise the model on the server wouldn't know
00:19:01what's going on because by default not by default anymore because by default it stores it but if that
00:19:06is turned off it is stateless that's the way i should put it but if store is not set to false if you use
00:19:13the default settings then they do store that context so yeah you only have to send your new message you
00:19:20don't have to send the entire context with every new request convenient right sure convenient but suddenly
00:19:27you don't know what's going on you're not storing the the session anymore you can't switch at all you have
00:19:32basically no information about what was going on because the context is not there for you and that's
00:19:39why in this article armin also says that store falls is available and should be used because otherwise the
00:19:47default is true and they manage it for you but all these things all these little improvements that are
00:19:54happening here to make your life easier have the impact i described earlier that there is less
00:20:01transparency that you have to wonder if billing is correct that you can't audit it and that maybe most
00:20:08most importantly you don't own the session anymore with all its implications like reduced or no portability
00:20:15that you may not be able to come back to a session in the future um if the model provider doesn't
00:20:21support it with newer models and that's all a bit of a worrying trend a trend as i said initially i
00:20:28didn't fully anticipate or think about to be very honest because i'm not in the business of building
00:20:34agent harnesses um and it's it's just not something i spent a lot of time worrying about i do worry a lot
00:20:42more about it now and um i don't see these model providers going back especially since they want
00:20:50to defend against distillation in their perfect world they certainly have all your data all your
00:20:56conversations on their servers and you only send your tasks that is certainly their perfect world because
00:21:03that gives them way more control and in general i am the kind of person who likes to stay in control
00:21:12and these trends are making me have less control over my work with ai these are my thoughts on that please share yours