The End of the Static Screen: Architecting Intent-Driven UX — Gus Iwanaga, commercetools
AAI Engineer
Computing/SoftwareManagementInternet Technology
Transcript
00:00:00How's it going? The end of the conference. How is everybody feeling? Tired? Drinking from the
00:00:19firehose as well? Are you guys a little bit tired of hearing loop engineering, harness engineering,
00:00:24software factory, pre-training, post-training data, and what not? But anyway, those topics
00:00:33were more than valid, right? Hi. Some really common faces here. By the way, I'm super excited
00:00:40to be here and talk to you guys about this title. And I'm sorry. When I submitted the
00:00:47application, I was thinking of potentially a catchy title. But I don't like this at all.
00:00:52So, with all the respect to the organizers, I'll have to make a change. And the actual
00:00:58topic that I want to focus here today is lessons learned from a team that is building proper
00:01:05generative UX and UI. And I was going to touch on agentic orchestration. But come on. Over
00:01:13the last three days, this is what we heard all the time. So, I'd rather focus on what I didn't
00:01:19hear enough about here in the conference. And hopefully, you walk away, if not with something
00:01:24very tangible but with a new mental model that can spark meaningful discussions down the road.
00:01:29Sounds good? All right. Very good. So, hi, everybody. I'm Gus. I'm a general manager at Commerce
00:01:36Tools. I lead product UX and engineering for zero to one products. I tell people I'm in a very privileged
00:01:43position because we get to build really cool stuff. So, we cook really interesting stuff at Commerce
00:01:48Tools. Fantastic. And this is what you guys can expect, at least during the presentation. I'd like to make sure that we're on the same page with respect to the problem space.
00:01:52Followed by a quick demo of the product. Because I'm not sure if you guys agree, an image speaks more than 1,000 words. So, it would just make it more tangible for everybody here. Followed by a rapid discussion on the emergence of UI protocols. And I'm not sure if you guys joined, maybe some of the talks here. Even the founder of some of these protocols were here this week. And that was really cool. Followed by a quick demo of the image.
00:02:11And last but not least, the challenges that my team and I faced and we still face. And some of the mitigation tactics that we put in place to overcome some of these challenges. So, with that, let's continue. The problem space.
00:02:36So, this is more like a statement. And we're still adapt to the software that we ship. Not the other way around. Right? And then, although you could argue when GPT came out, this was November 2022. We had a really good glimpse of real personalization. But everything else remained static.
00:02:59And even with AI. And even with AI, we keep shipping a lot of stuff much faster. But to a significant extent, it is still static. And my question is, why?
00:03:10So, over the last 40 years, we kept shifting static experiences. And if I put myself in the shoes of some of my customers, they need several SaaS applications for the day-to-day work. And each with its own mental model and its own way to get anything done.
00:03:33And over time, it just kept getting worse. Just accruing debt. The cognitive load that we wanted to remove and that we wanted to transfer to the machine, it's on us. And now with AI, things can be different. I don't know if you guys agree. But this is what we think.
00:03:54And I do have a couple of examples. I'm not going to say out loud the name of these apps. But let's take a look. Here. First one. You know, that icon is very well known. This is probably the most famous CRM of all times. But when I look at this screen, there's a lot going on. I don't even know where to start. Right? Not only the information overload. How many features? How many teams do you think are somewhat involved just to ship this? Many.
00:04:22Just to ship this. Many probably. Right? And this is just one. Let's have a look. I have a couple of other examples. So this here. Look at that. Oh my God. This is a fancy table. I don't even know where to start. But anyways. One more. Does everybody know this one here? Beautiful. Beautiful UI. Right? It's a fantastic, super intuitive.
00:04:48And something else just to highlight. Do you guys know how much time these companies need to invest in onboarding people? So that was the trade-off. Right? So you've got to allocate a lot of time for a lot of people just to onboard newcomers as a result of this complexity that has been introduced over time.
00:05:10So this is just at least the hardcore evidence. So different apps, different logic every single time. And then more apps are coming out. So imagine, just put yourself in the shoes of an average user. And then, oh, now I have five apps.
00:05:26And then every single one I need to learn how to navigate, how to browse, and so on and so forth. And then this has been the history up until now. But then, this was August last year, I sat down with my boss, happens to be the founder of the company. So big shout out to my boss.
00:05:44And then we asked this question. And then we asked this question, because at Commerce Tools, we are an API first company, 300 plus still counting. And we asked this posing question, through the lens of artificial intelligence, what are the foundational shifts that could be made, if we could change drastically the way that we interact with software?
00:06:02Not in a static fashion. And the answer to that question led to the product that I'm going to demo right now. Let's go. How about quick demo? Guys like the idea? Give me a thumbs up. I know everybody's tired. Let's go. Yay.
00:06:17All right. Cool. Look at this. I know I'm going to zoom in. No worries. So I have this query here, create a sales report for Q1. And the UX side of me, when I look at what was generated, and by the way, everything here on the right side has been auto-generated, guided by us, but this is AI, right?
00:06:41Deciding on the placement on the information architecture, deciding which components had to be actually retrieved from the catalog. But I don't like it at all. Right? It's a, even if you don't know a lot about UX, just let's, you know, I have at least four different variations, because those were four different terms for the same query. Let's check it out.
00:07:04Okay. First one here, it was Q1, urban thread, whatever that is, Q1. I see all of these KPI cards. There's a lot going on here. And then my intuition tells me, man, this doesn't add up.
00:07:17Okay. Second one, it's not Q1 anymore. Now this is January and March. There's no consistency. Does that help? Yes or no?
00:07:27No, right? This will only create confusion. If this is a heavy personalized experience for the user, imagine if every single time you need to prompt and then at least the model will output something different.
00:07:41This is not good. And this is just the second turn. Let's have a look at the third one. Oh, my God. Now there is even more stuff here on the right side.
00:07:51So this is, I have this component, the KPI cards, a bunch of text, a bunch of charts. And then this, this was the beginning of our journey. One more? Yes. Okay.
00:08:04Okay. Now it's still Q1. But still, for me, there's still, it's confusing. And because this has been a very experimental journey, at least, okay, let's, let's move on here.
00:08:18My feedback to the team and to myself was no, no, and no, there's no way that I would ship this to, to prod, uh, whatsoever. Right. And then, uh, I have my colleagues here just to, uh, confirm what I just said, but then things evolved.
00:08:33And, uh, I like to demo the current state of the product. It's much more sophisticated and let's have a look. I, I like to plan a campaign and for what it's worth, I'm going to save you from all the nitty gritty details for everything that is domain, uh, specific, but I'm going to pick this query here, uh, and then let's see what happens.
00:08:54And this is the, the, the organic orchestration part that I was going to highlight underneath the hood. We have the orchestrator and the orchestrator can then just extract the intent of the query based off of the intent of the query.
00:09:05It can locate the tools, right? Those can be a first party, third party tools and the outputs of this different, it could be agents on MCP servers combined will give, uh, uh, enough what I call ammunition in context for the UX agent to eventually render something, uh, that we call meaningful.
00:09:24So compared to the previous terms, this is, this is decent, right? I want to just remove my bias, but the overall aesthetics, the looking field of this query, uh, it, it resonates with me.
00:09:38Would you agree? Give me a thumbs up if you, if you agree. Okay. At least, um, the vast majority here. And then see, it is decent. And let me just continue here.
00:09:48And then once again, right, this was decided by AI guided by us. I would just want to make, make that clear here. And then I'm going to touch on the UI protocols and then how you can make this happen.
00:10:01But okay, let's see if I approve here and then this is already life and this is pre prod, right? So, uh, this is great. Let me go back to my presentation.
00:10:10Perfect. I have one more question. Are you guys skeptical that this is possible because I can tell you this, it is possible. If you're still skeptical, don't worry.
00:10:20I have all of these guys here, uh, also, uh, every day just looking at me and then challenging whether, uh, this, uh, can be made possible at scale. Right.
00:10:29And then, uh, okay. And then this is the part that I would like just to touch, uh, touch base on the, uh, on the three ways that you can render what you just saw. And there are different UI protocols.
00:10:41Are you guys familiar with, uh, generative, uh, UI? Have you guys played with it? Let me see here. Okay. Wow. That's really cool. Uh, okay.
00:10:49So, uh, what I would like just to share with you guys, it's all about how much control you want to exercise over the experience.
00:10:57And this matters a lot because, uh, you know, as a non-deterministic solution, uh, you can decide if you want something really, um, like this.
00:11:06So let's have a look here. This is, uh, Chad GPT. And my query was, help me find a Japanese restaurant, uh, in SF today.
00:11:16If you guys see here, this component, this component is very opinionated.
00:11:22Would you agree with that here? Right. So, uh, you can have complete control over this, uh, component.
00:11:29So depending upon the nature of your business, this works really well, right.
00:11:35And for that, let me just go back here. And then this is what I call, uh, control.
00:11:40Essentially, you ship the component, uh, as it is, the agent will pick and it will display the component.
00:11:45And it will display exactly the way that you describe. However, depending upon the nature of your business,
00:11:51at least for us, right. At my company, we're a B2B SaaS. The, there's so much configuration that we don't want to be over, uh, prescriptive.
00:12:00Because, uh, the feedback that I keep getting from my customers, oh, the flows are so confusing.
00:12:05There's so much configuration. How can you remove the cognitive load, uh, for me?
00:12:10But, uh, if you're like booking, for example, this approach, uh, works, uh, really well.
00:12:15And then let's see, uh, how it works. So, essentially, you have the agent.
00:12:21The agent would just pick the component from your catalog and then it will render, uh, as it is.
00:12:26Right. And, uh, in the interest of time, I'm not gonna touch base on the code snippets, uh, that I have for this, uh, three different types.
00:12:35But afterwards, if you guys are interested, uh, I can share the presentation and then you can have a look.
00:12:39All right. Very good. This is, uh, at least on the left side.
00:12:44Let's discuss a little bit on the right side because this is when you give full autonomy to the LLM.
00:12:49If you guys remember, uh, at least the previous attempts from my product, this is exactly what we did.
00:12:54So, we just say, so, hey, LLM, how, uh, how would you compose this experience knowing that you have these components, right?
00:13:02But, uh, that was, uh, there was a little bit, uh, a little bit of our opinion because it's, uh, it has, uh, it had, uh, some of the components available.
00:13:11But it could happen that, uh, you can just delegate fully to the LLM.
00:13:16And then right now I'm here on Claude and I asked Claude, so, hey, create an org chart with three levels.
00:13:22That was it.
00:13:23And then Claude just render this diagram and it works really well, right?
00:13:29But here, uh, if I put myself in the shoes of a company, I'm not sure I would delegate fully to the LLM.
00:13:35Because I cannot control, uh, at least the out, uh, the output, uh, and the outcome.
00:13:40And me personally, me guys, as a UX leader, the UX side of me, will always say, no, you gotta be in control.
00:13:48There's been a couple of talks, uh, here, uh, at least this week on design, on taste and judgment.
00:13:54And this matters a lot.
00:13:56If you guys wanna embark on this journey of leveraging these protocols, you don't wanna delegate too much, uh, of the actual experience to the LLM.
00:14:04You gotta find alternatives.
00:14:06And I'm gonna touch on that in just a little bit.
00:14:08Okay, cool.
00:14:09And then this is how the open-ended, uh, approach works.
00:14:14So essentially there, uh, there's going to be an MCP tool and then this will literally ship, uh, the, uh, HTML.
00:14:21And then in a sandbox, uh, iframe, uh, environment, this will be, uh, rendered, uh, in the host of your choice.
00:14:29But it can be a chat.
00:14:30It can be this, uh, yeah, cloud.
00:14:32It could be chat GPT, it could be perplexity, or it could be, uh, any other chat.
00:14:37If you're willing just, uh, to give full control to the LLM, uh, good luck.
00:14:41But the one that I would like to highlight is this here.
00:14:46And this was our choice that we call the declarative.
00:14:49The declarative is, uh, in the middle.
00:14:51Uh, if you guys, uh, heard some of the protocols, and I don't wanna get into the specifics of each, because they have different characteristics.
00:14:58But H2UI from Google.
00:15:01JSON render from, uh, Vercel.
00:15:04Open UI by thesis.
00:15:06Those are some of the protocols that will give you this, uh, in between here.
00:15:10And let's have a look, right?
00:15:12So, at least for, uh, my product, the one that I just showed, the orchestrator agent will, uh, eventually, if you think of, uh, the whole traversal, uh, the user will enter the query.
00:15:23And then there's gonna be the intent classification based off of the intent classification.
00:15:28Uh, then the tools will be invoked.
00:15:30The data will be retrieved.
00:15:31And, uh, somewhat, uh, somewhat, uh, in between, uh, at least, uh, there will be the mapping of the eligible components from your catalog, uh, to, uh, the, uh, the entities, uh, of, uh, of the tools, right?
00:15:44And then the orchestrator would just broadcast this UI description.
00:15:48It's like a UI spec.
00:15:49This UI spec will be also, we have this component catalog here and we use the ZOD schema.
00:15:55And then you gotta be compliant with those protocols.
00:15:57This is just one of the requirements.
00:15:59And then you just rendered that, right?
00:16:01Uh, and then the final output will be the native UI.
00:16:05In this case, the react components.
00:16:07Uh, this is it.
00:16:08The good thing about, uh, the declarative, uh, approach, uh, is, uh, it will be compliant with your design system everywhere.
00:16:16This matters a lot.
00:16:18So in our case, we did not want to delegate, uh, to, uh, the LLM because you guys saw, uh, over there, you can change the copy, right?
00:16:27So it's not key one.
00:16:28Sometimes it's gonna be March, uh, January to March.
00:16:32It matters a lot.
00:16:33So within, uh, UX, we have different factors, right?
00:16:36We have the actual UX is, uh, if you think of the overall experience, there is UI.
00:16:41There's also copy UX, uh, writing, uh, and so on and so forth.
00:16:45But this approach, uh, gives us this in between.
00:16:48It is less deterministic.
00:16:50And I think this is a really good segue to some of the challenges.
00:16:53Because imagine, I'm gonna use my example once again.
00:16:56The orchestrator, uh, would just, uh, fetch the eligible, uh, components for that query.
00:17:01But then it's up to the LLM, uh, how to place in the UI.
00:17:06And this can get really, really messy, right?
00:17:09And, uh, those are the challenges that I'd like to share with you guys here.
00:17:13I gotta be careful because I've got only 30 minutes, but let's go.
00:17:17So the first challenge is if the agent picks the components, who's in charge of what entity is in charge of arranging them.
00:17:26And this is information architecture, right?
00:17:28This is a critical, uh, aspect of UX.
00:17:31And then once again, if you put yourself in the shoes of the average customer, it matters a lot.
00:17:36So if you're just left alone, the placement can be totally random.
00:17:40So at least in my team, we borrow this concept of atomic, uh, design and atomic design.
00:17:48It goes like this.
00:17:49Let me just change here.
00:17:50Uh, yeah.
00:17:51So as you can see, uh, atomic design.
00:17:54And then I have the definition is, uh, a methodology composed of five distinct stages working together to create interface design systems in a more deliberate and hierarchical manner.
00:18:05This helps a lot, right?
00:18:06Because those are the individual elements.
00:18:09And if you think of the overall structure of the page, it gives me the ability to steer as I see fit.
00:18:16And what we've done in my team, so this UX agent, we harnessed this UX agent.
00:18:22So we eventually taught, uh, this, uh, this UX agent, what good looks like.
00:18:27What, what is, uh, the optimal, uh, layout for a given situation.
00:18:31And we have a catalog of different templates.
00:18:34Uh, yeah.
00:18:35But I would like to show you, uh, at least this because this detail is very important.
00:18:40Here.
00:18:41It is the overall hierarchy.
00:18:43Hierarchy.
00:18:44So if you remember, uh, part of the orchestrator, the orchestrator will eventually just fetch the eligible components to accomplish the query, uh, of the user.
00:18:54But then, uh, the next big question is how do we arrange that?
00:18:57The approach that we use was think of this hierarchy.
00:19:00So you have the overall page, the layout, the layout, uh, will contain different slots.
00:19:07So think of this one here, the header, you can have the main, and then you can have sub, uh, sub slots.
00:19:13Sub slots can have sub slots.
00:19:15And within the sub slots, you can have, uh, eligible component categories.
00:19:19And then this will allow us to steer, uh, eventually the, uh, the, uh, the optimal placement of the components that, uh, have been retrieved by the orchestrator.
00:19:30So this is literally us codifying, uh, uh, our UX knowledge into this agent.
00:19:36So the next time, uh, you know, it, it doesn't matter.
00:19:39We'll just, uh, follow the same approach.
00:19:41And then this is the hierarchy that we're using.
00:19:43So layout, to slots, to sub slots, to components.
00:19:47However, because of the orchestrator, we, we flipped the order.
00:19:51So from components, components will map to sub slots, sub slots, to slots, slots to templates.
00:19:57And then we can, uh, just, uh, arrange, uh, as needed.
00:20:01So this was a hell of a challenge.
00:20:03It is still a challenge, by the way.
00:20:05And it's a really good segue to the second one, uh, the second challenge, which is, uh,
00:20:10is, uh, the actual design, your design system and your catalog.
00:20:15This becomes the heartbeat of the whole thing, right?
00:20:18I cannot stress enough.
00:20:20If you guys see the potential of leveraging this UI protocols for your product, this is going to be a big deal, right?
00:20:27So we're pushing the boundaries and then we're testing and we keep on testing.
00:20:31There's different protocols that I mentioned, H2 UI, JSON render, uh, open UI and so on and so forth.
00:20:37But this has been, uh, quite challenging because the, the catalog is the contract between the agent, uh, and the UI.
00:20:44So every property matters.
00:20:47And not only for the catalog, but for the layout as well.
00:20:50So if you remember, the layout has its own components, the slots and the sub slots, each of those components will have, uh, its own attributes.
00:21:00And all of that, this curation, let me just encapsulate into curation.
00:21:04This curation is absolutely needed so that you can deliver something meaningful.
00:21:09Not some demo that you will see out there for the sake of demo, right?
00:21:13So this, this, this will give you a control, will allow you to steer from a UX perspective.
00:21:19Very good.
00:21:20And, uh, last but not least, uh, one significant challenge that we had is that my teams do not design the pixel anymore.
00:21:28I don't know if you could see that, right?
00:21:30So we're not here designing the entire flow.
00:21:32Uh, the entire flow now, uh, AI can dictate that to a significant extent, but the nature of the work shifted quite a bit.
00:21:40And it's been an interesting journey to say the least a really good one.
00:21:43But even for the non-technical, uh, PMs and UX designers, it w it was a big hit because right now we talk about, uh, the schema.
00:21:53Let's talk about this curation of the catalog.
00:21:56Let's talk about the rules.
00:21:58Let's talk about the synthetic data that we can generate.
00:22:01How can we generate the queries that will map to a given component as part of this mapping logic?
00:22:07Let's talk about interaction patterns.
00:22:09So this, uh, if, once again, if you guys are going to embark on this journey, be aware that the people element is very important.
00:22:17When I talk to, uh, other leaders, I, I talk about the three P's.
00:22:21People, product, and process, right?
00:22:23And then a very lightweight process.
00:22:25But this matters a lot.
00:22:27And, uh, with that, um, I'm going to leave.
00:22:30I'm going to leave a couple of resources here.
00:22:32So, and by the way, those are talks from AIE.
00:22:36For what it's worth.
00:22:37So, uh, yeah.
00:22:38People that have been talking about these protocols over and over and over.
00:22:42So, uh, please just take advantage.
00:22:44Take a screenshot.
00:22:45And, uh, if you guys want to connect with me here.
00:22:48Um, yeah.
00:22:49My LinkedIn or just take a screenshot.
00:22:51I would love to, uh, talk more, uh, about the topic.
00:22:55I can tell you this.
00:22:56It's just a matter of time, right?
00:22:57So, this is coming.
00:22:58So, thank you so much.
00:22:59Thank you so much.
00:23:00Thank you so much.