Event-Driven Architecture, Webhook Chaos, and the Rise of AI Agents | Better Stack Podcast Ep. 17
BBetter Stack
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Welcome to the Better Stack podcast, where we have conversations about software development,
00:00:04AI, and all sorts of new technology. I'm one of your hosts, Andrus, and today I am joined by
00:00:10James and Alex. Hey, Alex, welcome to the show. Thanks for having me, guys. So let's start off
00:00:16with the company that you run. It is called Hookdeck. So for those who are not familiar,
00:00:22what is Hookdeck? What does it do? Tell us more about it. Yeah, I like to think of it as the
00:00:27webhook company that's trying to kill Webhooks. That's probably a story that we can get into.
00:00:32But we build kind of two main products. One is the Event Gateway. So the Event Gateway is serve as
00:00:37kind of like a specialized event bus for all events that are coming from outside your infrastructure.
00:00:41Webhooks is a prime suspect, but we see a lot of kind of like IoT and SDK-based use cases and so on.
00:00:47So basically, we provide kind of like untrusted endpoint. You can send any events to you.
00:00:51And then all the capabilities to manage those events. So that goes from like filtering,
00:00:55transformation, routing, queuing, alert, and issue management replay is kind of like the whole thing.
00:01:01So really, it's covering both kind of like the interoperability in the sense that I need to receive
00:01:05events and webhooks from all the vendors I work with, right? That might be Stripe, Shopify, Twilio,
00:01:10WhatsApp, TikTok, whatever, you name it. They each have their own kind of like criterias and quirks
00:01:16and specific requirements that they have. And so we can kind of like standardize all that and bring
00:01:20that into a single contract. And then everything from the kind of queuing standpoint. And so for
00:01:27instance, like Shopify, there's like a flash sale on your store and you get like just absolutely
00:01:31swarmed, like stampede with events. You normalize that so you can control the throughput and the event
00:01:38gateway and all that. So that's a big part of it. It's kind of like the consumer side. And that's
00:01:42where really the deck story kind of began. And more recently, we also released Outpost. Outpost is fully
00:01:49open source, Apache 2.0 project. And we now have a managed service for it as well. And it's to send
00:01:55events. And so it's kind of like the other side of the equation, right? It's you as a publisher, as a
00:01:59platform, a dev tool. We're really at this point, kind of like everyone is also sending events. I'm surprised
00:02:05every day. And so Outpost is this managed service or self-hosted service that you can use to like
00:02:12declare who your tenants are, what the destinations, the endpoints that they have registered, configure
00:02:17topics and kind of publish events to them. And handles everything from visibility, metrics, delivery
00:02:23guarantees, retries, and kind of like all the usual suspects around like signatures, signature rotation,
00:02:28and all that. The little kind of like joke I was making about trying to kill Webhooks is that Outpost,
00:02:35yes, allows you to send Webhooks, but also allows you to publish events directly to your user's message
00:02:39bus. And so Outpost natively supports what is kind of now known as event destinations. And so Webhook is
00:02:46thought as the transport router. And so you're basically like sending events over Webhooks and
00:02:53Webhooks is the standard transport mechanism, right? And you can choose new transport protocols, things like MQ
00:02:59for RabbitMQ or Kafka. And we support like publishing directly also to all the usual suspects like Pub/Sub,
00:03:05SQS, AWS EventBridge, and then obviously the YokeDeck Event Gateway, right? So I like to think of it as the
00:03:12better destination, but we'll see if people believe it is. So you said you want to kill Webhooks. So I
00:03:18imagine this idea and its inception came from the fact that you were all tired of Webhooks or it was
00:03:25too hard to manage. Like, tell us about how you came up with the idea. Yeah, I was just telling the
00:03:31story actually to someone the other day. Like five or so years ago, I published this article in Medium.
00:03:37I'm probably coming in on six years now. And the article is titled and playing right off to what you're
00:03:42saying, "Webhooks sucks, but there's something you can do about it." And that was just like me in my
00:03:48basement, you know, like messing around and building kind of like a V1 kind of like prototype of a deck.
00:03:54But it very much like came from that frustration. I was personally working in eCommerce and we built
00:03:59like a lot of custom kind of software to power the eCommerce from like custom subscription management to
00:04:05warehousing and fulfillment and all that sort of stuff. And just like so many of your issues
00:04:10boiled down to Webhooks. And it was almost kind of like a recurring joke because like on one part,
00:04:15I was trying to like sell was women fashion business, right? I was trying to sell like
00:04:18oisree and like nylons and stuff. And then on the other end, it's like, where the hell is the
00:04:23sword or Webhook? What happened to it? It just like felt like very mismatched in terms of of concerns.
00:04:28And so it very much like came from that, just like I'm frustrated as a consumer of those Webhooks
00:04:33that there's no real kind of like batteries included, you know, way of dealing with this
00:04:38problem. And if you went online and kind of like search for recommendation, like there's like,
00:04:43I mean, like Webhooks are not new. Like at the end of the day, it's HTTP requests. There's
00:04:46obviously like patterns established around how you're supposed to deal with this. But then like you
00:04:50very quickly go down the rabbit hole. Okay. Okay. You need like your ingestion consumers that can like
00:04:55autoscale and then you need to queue into a queue like SQS. And then you need to deploy a set of
00:05:00consumers that are going to consume from like SQS. And then if something goes wrong, it's going to end
00:05:05up in the data queue. And then you need like some script to be able to recover from the data queue and
00:05:09understand why things ended up in there in the first place. And there's just like all those concerns.
00:05:14And as complexity grows up, you also want to be able to like replay historical events that you've
00:05:18received. You want to be able to like see what specifically the payload of the Webhook was.
00:05:22And then you start dealing with the different vendors because like maybe Stripe is going to give
00:05:26you a UI and then Intercom is not going to give you a UI. And then you like have all those like
00:05:31different quirks that you have to deal with. And so it very much like stemmed from that frustration and
00:05:35just like why was there no solution to this, right? And at first I was looking at it very much from like
00:05:41an Obvisability standpoint and it didn't really work. The part that I kind of miss is that Obvisability is
00:05:47only really one part of it. Ultimately I have this saying I call kind of like Webhook the gateway drug
00:05:51to event-driven architecture. And the reason for this is, and it's really also why I like to insist
00:05:56on the term event rather than Webhook, is that behind every Webhook there is an event and there is
00:06:01now event-driven architecture paradigms that come like packaged with that Webhook, right? And so every
00:06:07time that you receive Webhook for any serious scale or level of kind of like critical use cases, you now have to
00:06:14think about in the potency ordering delivery guarantees and kind of like all the type of
00:06:19challenge that you have once you start dealing with like asynchronous event-driven programming paradigm.
00:06:25And so really like a lot of what we built kind of evolved into like building a full-fledged queue
00:06:29at the end of the day. So there's this interoperability that you're sure that's we're dealing with events,
00:06:33but then all those kinds of like semantics are more around dealing with events and pub subsystems
00:06:38and so on in general. And we came from a place of not trying to reinvent queues. I think we like
00:06:43stumbled on a bunch of like pretty genuine cool ideas. And one thing that we're seeing now is there's
00:06:48actually people are just like moving to a deck to like swap out pub sub or SQS from their stack, which
00:06:53to me is kind of mind-boggling because like we don't run your VPC and all that sort of stuff. I think
00:06:57there's a lot of issues with that maybe the maybe maybe maybe into the road map. But the point is, the point
00:07:05is, I think there's like a lot of semantics around adventure and architecture that people have kind of
00:07:09just like grown accustomed to and no one really would question. And it's like, why are we doing
00:07:13deadlier queues again? Like the fuck is wrong with that? Because I don't know anybody that thinks like
00:07:19deadlier queue is like a convenient semantic for dealing with errors in your queues, right? So anyway, and I'm happy to
00:07:25like maybe touch on some of those things because there's a couple of strong, strong opinions in
00:07:29there. I don't know how much you guys are nerds of pub sub and Avenger in architecture. I don't
00:07:35want to drag you too far into it. I feel like, yeah, my understanding of dead letter queues and all
00:07:40that stuff is very, very minimal. Like some of the things I'm hearing today, I hear for the first time.
00:07:48Yeah, yeah. I was gonna say mine's quite surface level as well, but I've dealt with webhooks
00:07:53a fair bit and dealt with some of the problems you mentioned. And I think like, and that's how
00:07:57you guys are kind of making the point for this webhooks, the gateway drug to Avenger in architecture,
00:08:01because I'm willing to bet that for kind of developers of your background, webhooks is
00:08:06probably the first exposure that you've had to, wait, this is async. How do I deal with this?
00:08:12And I think there's this whole learning curve, right? It's like the classic iceberg where it's like,
00:08:17oh, you have the HTTP request that comes in and then like, you know, the rest of the iceberg
00:08:21behind the kind of like in the in the water, right? And I think like one of the thing that I'm trying
00:08:26to do is kind of like bring a level of DX that hasn't really been achieved in that space so that
00:08:32you don't actually have to figure out the rest of the iceberg, right? There's some things that are a
00:08:37little bit inevitable, like I don't want to misrepresent it. Like I think once you start
00:08:40working with those sorts of problems, like you need to have good understanding of in the potency,
00:08:45for instance, you need to have like good understanding of like ordering guarantees. So
00:08:49like it's not, it's not like you can solve every problem. I think you're still dealing with events
00:08:53at the end of the day. But I think there's a lot of the semantics and complexity that is mainly coming
00:08:58from not like fully thought through tooling more so than, than like, you know, genuine complexity
00:09:05associated with like that problem space. So I think right now like we end up like serving kind of two
00:09:13sorts of customers, we end up serving the customers that like really know the problem and in depth and
00:09:18like will, you know, do solution engineering around like their, their current like Kafka setup and all
00:09:23that sort of stuff like all day long. And then, you know, we pitch them on some of those new semantics,
00:09:28and they're really excited by it. That's like, one category. But the other category is like,
00:09:31I don't, I don't want to actually have to learn any of this. And basically I try to like kind of
00:09:36sidestep this whole thing. Right. So, so, so we see kind of a lot of like those two, two profiles.
00:09:41Is that also a reason why you open source this, this tool Outpost to make it easier for developers
00:09:47to start with, when using that? In some ways. The other way is because like, we don't need Outposts
00:09:53to make money. Okay. And so we're like, I guess we should just open source it then. Yeah. It's just,
00:10:01I know that a lot of our YouTube audience really like knowing about new open source tools. That's
00:10:07why I just, you know, went to ask about that as well. But, but I, so first of all, I'm like,
00:10:13I kind of regret a little bit, like in the early days of like not taking like a bit more of an open
00:10:18source first approach. And I think there's a lot of things that, how you end up like building your
00:10:21software that's very different when you're building for kind of closed source versus open source. But
00:10:26coming to the open source kind of decision, I think one thing you're seeing kind of like with venture
00:10:30back, like startups and that sort of stuff is that you end up like with those like kind of fake open
00:10:34source, right? It's like open source, but either it's like open core or like, yes, it's open source,
00:10:39but like they really don't want you to use the open source or they end up like having to relicense
00:10:44around kind of like not true open source licenses and that sort of stuff. Right. And I think,
00:10:49I think that got me really excited for Outposts and open source is that I felt like we had an
00:10:54opportunity to have the right incentives. Now, maybe we get a little bit into the weeds of how we
00:10:59think about it kind of like the business and that sort of stuff. But in my mind, people that have
00:11:03to receive web books kind of like on the consumer side is at least two to three orders of magnitude more
00:11:09than people that need to send a web book, right? Because like, think about it, if I send web books,
00:11:13I'm sending it to all my customers, and my customers might be, you know, 10, if you're just
00:11:17starting 100, like 1000, 2 million. Right. And so, and so inevitably, like the consumer side is just a
00:11:24lot more people. And so when I'm thinking about it more from like an entrepreneur point of view,
00:11:28I'm like, I'm more excited about the business of pushing on the consumer side. Right. But I think
00:11:33what we realized, though, is obviously, you don't have consumers without good producers. And I think
00:11:37there's more and more demand for web books kind of like in general, across apps that you develop. And so,
00:11:42like also more people want to offer it and don't want to go through like the edX and the overhead and
00:11:46so on. But from my perspective, first, like sending web books is just a simpler problem, because you
00:11:51control the parameters while on the receiving side, you don't the vendor gets addicted dictates what the
00:11:57parameters are. So it tends to be, I don't want to say it's like simple, simple, but like it's,
00:12:02and there's definitely like a bunch of ways you can do this wrong. But I think in terms of a problem
00:12:06space, it's definitely more limited on the producer side and on consumer side. And then second,
00:12:10like our job, I think is to encourage producing of events. And that's the thing that we ultimately
00:12:15care about because we want more people to consume those web books and eventually potentially be
00:12:19customers and that sort of stuff. Right. And so I think that puts us in a position where we can like
00:12:23truly say, look, if you're going to use Outpost, whether you open source it or you, or you deploy it
00:12:29with Oogdeck on the managed version, it generally doesn't matter for us. Like we're accomplishing kind
00:12:35of like the business goal either way. Right. And so like that alignment incentive got me really
00:12:40excited. And I think there's a couple of things that we did first, like we released the open source
00:12:44before even considering building a managed version. Like there was generally no plan to build a
00:12:48managed version when we build the open source. And the open source project has been published about
00:12:52like two years ago at this point. So it's been took us almost like two years before enough people were
00:12:57asking for the managed version that we ended up billing it. So that's one part. And the other part
00:13:03as well as the managed version of Outpost runs the exact same Docker build that's being published on
00:13:08Docker up from the open source, it kind of builds that right in the sense that we don't have a private
00:13:13fork. We're not packaging features and so on outside the open source. We're truly running exactly the same
00:13:19Docker build. And so like the question for you is not like, oh, does it offer feature X or Y,
00:13:23whatever in either version? It's literally just do you want to deploy it and have to take care
00:13:29of the operational burden associated with that? Or do you want to pay someone else to do it for you?
00:13:34Right. And I don't think there's a right or wrong answer. That depends on every business or level of
00:13:38comfort or stock, their compliance requirements, kind of like you name it. Right. But because of that,
00:13:44I feel like we can be good contributors to the open source community with that project. And yeah,
00:13:51I've been, I've been very glad to work on it because it's the first kind of like big open
00:13:55source project that I'm involved in, like, you know, outside of like hearing their contributions
00:14:01and that sort of stuff, but as like a core maintainer. And so, yeah, no, it's been good.
00:14:05How have you found the, uh, the open source world now that clawed code and that lot exist? Have
00:14:09you had a lot of PRs and security vulnerabilities that aren't real or has it been okay to manage?
00:14:14Um, look, I don't think we're at a scale where I can speak to some what, what other people are
00:14:20experiencing. So I definitely don't want to like misrepresent this situation because it definitely
00:14:25sounds bad out there for a lot of folks. I will say that I've been surprised by the qualities and
00:14:30the contribution we've had, but we've definitely also had contributions where there's this like one
00:14:34specific PR, I think it's still open on the repo right now, which is to add Cloudflare queues as a
00:14:40destination. Cause like one of the supportive, supported event destinations all go on the surface,
00:14:44right? Description sounds good on the PR and that sort of stuff. And then when you actually dig through
00:14:48the code, it's like using a bunch of like completely made up Cloudflare APIs.
00:14:52And they didn't double check it. Just, I mean, clearly the person that opened the PR didn't even
00:14:56actually try to run it. Right. Yeah. And so, and so now the burden is kind of on us of like, okay,
00:15:02now that there's this open PR, cause obviously I'm like slightly concerned about being perceived as
00:15:08we're like not trying to like encourage people that we could like be seen as competing with.
00:15:12So, so I do think like we should add Cloudflare queues and that sort of stuff, but at the same
00:15:15time, like it was on the closed roadmap, like no one else other than this person asked for it and
00:15:19that sort of stuff. Right. And now it's like in this afbic state where it's like, okay, if you close it,
00:15:24you look like you don't want it to support people that could be perceived as competitors. Right.
00:15:28But then do you actually dedicate resources and like move up your roadmap and that sort of stuff,
00:15:33like make sure you implement it correctly. Right. So it's been a bit of a catch 22 with those
00:15:37sorts of situations. Um, but I, so far I feel like, uh, it's been a really good experience and
00:15:43especially because we use the same build. Uh, one thing that we've like kind of taken too is whenever
00:15:48customers kind of reach out for feedback or they talk to us about like specific features and so on,
00:15:53we open issues and get up, or we point to like existing PRs, existing issues and that sort of stuff.
00:15:58And like, we kind of make sure to always try to like have people know, by the way, the repo's there
00:16:03and you can go and contribute. And like, actually to that point, we've had a several kind of like
00:16:07managed users make contribution and basically just like go and implement their own features.
00:16:13Right. And I think it did reduce the barrier to entry tremendously for those sorts of kind of like
00:16:18genuine, uh, use cases for contributions. Cause you know, historically the projects in go,
00:16:24most of our users probably aren't go developers in the, in the first place. Right. Like we see a lot of
00:16:28Python type script, obviously like no JS, like, and so there's a barrier entry for go. And then also
00:16:35just like, you know, learning a new project to contribute is like non-trivial. Right. And so
00:16:39I think it did reduce that barrier to entry substantially. And that's, that's really good.
00:16:42Cause now if, as a user you're taking with this, like need of like, oh, this thing like really bothers me.
00:16:48And plus like, as a maintainer, that's such a strong signal that this feature is worth building.
00:16:53Someone like goes out of their way and spends their, you know, wordy tokens on implementing this.
00:16:58Like it's a signal that like, okay, this actually matters. Right. In terms of like stock ranking,
00:17:02like, you know, what's most important. So I think like reducing the barrier to entry is really good.
00:17:07And so I think if you can manage to get out of the floodgates of spam and all that store stuff,
00:17:12there's a lot of positive to it.
00:17:14In the landing page, one thing that really stuck out to me was a testimonial from the CEO of Vercel.
00:17:21So Vercel is actually using a hook deck as well as a product.
00:17:25That if you read that specific quote, like it is recommending it too.
00:17:29Oh, okay.
00:17:29Kind of like Vercel users and that sort of stuff.
00:17:32Um, but there's actually kind of a funny story there.
00:17:35Cause Guillermo kind of tweeted about it on a Saturday night.
00:17:39I had no idea it was coming.
00:17:40I don't really have a prior relationship with, with Guillermo and that sort of stuff.
00:17:45And it can speak to the influence of like some of the people in the community.
00:17:48Cause like the amount of just like sign up and awareness and so on that we got from that was
00:17:53like insane.
00:17:54Um, since then like Guillermo and I have been kind of in touch here and there.
00:17:57And it's like always been like good chats and all that.
00:17:59And it's interesting.
00:18:00Cause, uh, actually if you look at a company like Vercel, one thing that Guillermo was telling me
00:18:04is that he's actually like seeing a lot more people using WebEx now.
00:18:07And his read on it was that it's mainly driven by LLM options.
00:18:12And that's, that's creating new use cases for data.
00:18:15You usually wouldn't have cared about before.
00:18:18Like kind of think of your customer support system.
00:18:20For instance, you have like your agents and like humans that are in there.
00:18:23And like, there's very little reason to like do anything with, you know, new tickets event.
00:18:29And that sort of stuff.
00:18:30Cause like, anyways, it's like the human that's going to go and, and answer that message.
00:18:34But now we're obviously like in a world where that's completely changing.
00:18:37And when you move from humans triggering the agents, like what trigger agents, it's events.
00:18:42Right.
00:18:42And suddenly now you care about events from your, your customer support system and your
00:18:46marketing tool and your blah, blah, blah.
00:18:48Right.
00:18:48And so, um, and so that's definitely something that I think as platforms they're seeing.
00:18:54And like, that's been kind of like interesting discussion points for us, obviously.
00:18:57But yeah, no, it's Vercel specifically just to qualify not directly a user.
00:19:01I mean, I, I'm sure a bunch of developers use like your local CLI and that sort of stuff
00:19:05at Vercel, but, uh, yeah.
00:19:06So what is the typical customer for someone using hook deck?
00:19:10Like if I had, I don't know, a small size t-shirt e-commerce shop or something, how would
00:19:15I use hook deck for it?
00:19:17And would it make sense to use it in this case?
00:19:19Probably not.
00:19:21Okay.
00:19:22You're asking me kind of like the, you know, million dollar question in the sense that I think it's
00:19:26something we've always very struggled in the sense that like the breadth of reasons why you'd use
00:19:32WebEx or depends on WebEx is just like insanely large.
00:19:35Like we get WebEx from, you know, usual suspect like Stripes and Shopify and, but we also have
00:19:40like WebEx coming from like all the Australian telecom providers and the UK clearinghouse.
00:19:47And there's actually a bunch coming from like the EVE online game.
00:19:50There's also like apparently a community of people still using, it's still playing this like Conan,
00:19:55the barbarian game, like from like 15 years ago or something.
00:19:58And like, they're big on WebEx too.
00:20:00Don't ask me why.
00:20:00Right.
00:20:01But the point is like, there's this like breadth that's like very hard to capture and saying like,
00:20:06these, this is our customer.
00:20:08Right.
00:20:09And so like when we look at people that use like that, there's like, there's no single use case
00:20:13or group or whatever.
00:20:14That's like maybe more than 10%.
00:20:16But coming to your e-commerce use case or like example.
00:20:19So first of all, like a small t-shirt store probably hasn't built like custom apps
00:20:25and that sort of stuff that would rely on it.
00:20:27But they almost certainly have installed apps.
00:20:30Like for instance, like an app to collect reviews and an app maybe for back in stock notifications
00:20:35and an app for inventory management and an app with their 3PL.
00:20:39And like all those guys are almost exclusively relying on WebEx.
00:20:43Right.
00:20:43So if I want to send a back in stock notification, what I'm doing, I'm listening to all the stock
00:20:48updates in the Shopify store through their WebEx.
00:20:50And then as soon as a product that was previously out of stock now is stock,
00:20:53I'm like, okay, got to send the email.
00:20:54Right.
00:20:55And so basically every single thing they're going to install or add and so on is going
00:20:59to rely on WebEx behind the scenes.
00:21:00And so that's definitely like one big category.
00:21:02People are building like apps and it's not just Shopify, obviously like Stripe, for instance,
00:21:07as like a big apps marketplace.
00:21:09And a bunch of these are customers, that sort of stuff.
00:21:11The other thing we'll see is big stores.
00:21:14Like think like the gym shark and like the good Americans and the rogue and that sort of stuff
00:21:19where they have their own technical teams and they're building custom apps.
00:21:22They're building custom apps for their system integration, often like very operational in terms
00:21:28of like sequencing and when an order needs to be synced with a 3PL and needs to be adding notes.
00:21:34And maybe there's like customer specific, you know, preferences that needs to be added to that data
00:21:39before it reaches 3PL, that sort of stuff.
00:21:41Right.
00:21:41And some of it's going to empower kind of user experiences, things like, you know, custom
00:21:46subscription management or, you know, specific emails that you want to send when someone buys
00:21:51a gift card and wants to send it to their friend and, you know, like all that sort of stuff.
00:21:55And so that's what we tend to see, like the small Shopify store is probably the one you could have
00:21:59picked that we don't see a lot of people using WebEx for, but you know, almost everywhere else
00:22:04you aim kind of like in that, in that, you know, segment, you're gonna, you're gonna land on a
00:22:08company that uses WebEx.
00:22:09Fair enough.
00:22:10Fair enough.
00:22:11How does it compare to something like AWS EventBridge?
00:22:14Because that's sort of one of the other choices that I've heard of.
00:22:17Is it just better developer experience or?
00:22:19Well, obviously we all know AWS, I mean like better stock is more or less making the same
00:22:24case, right?
00:22:25Yeah.
00:22:26And so whatever you guys say in kind of like your unique selling point versus, you know,
00:22:31cloud watch and Amazon blog services and all that sort of stuff kind of like carry over.
00:22:37So, and that's very much like the developer experience, like argument, right?
00:22:40You know, like proper APIs and MCP and nice UI and like rock solid office ability and all that
00:22:46sort of stuff.
00:22:47Right.
00:22:47And obviously like the integration of it, I think like the AWS problem is always that you
00:22:51end up with like, you know, 20 services, most of, most of which probably forgotten the name by now.
00:22:56And then kind of like the data is like sparsely through this and the configuration is parsing
00:22:59through it, sparsely through it and all that.
00:23:01Right.
00:23:01So, so that's one point.
00:23:02But I think the broader point is AWS EventBridge doesn't actually integrate with anybody,
00:23:07everybody.
00:23:08So like the vendor needs to have like integrated with AWS EventBridge.
00:23:11There's ways that you can work around that with like the API gateway and blah, blah, blah.
00:23:16But, um, so, so the point is like, it's very much a built for the AWS ecosystem where most of the
00:23:23use cases are actually like internal events to AWS, like things like a new S3 document was uploaded and
00:23:31that sort of stuff.
00:23:31And then the interoperability with the third party, while they do support like 40 odd, like whatever
00:23:36vendor, uh, don't quote me on this might've been updated by now, but you know, not thousands.
00:23:40You always end up in this scenario where, okay, well, Shopify supports EventBridge, but then like
00:23:45Twilio doesn't.
00:23:47Right.
00:23:47And so you end up with those, those issues where you can actually like bring an agnostic solution
00:23:51and just like works across the board.
00:23:52And so I think like, that's one thing we're trying to do, right.
00:23:54It's like build us in such a way that like, we don't need the vendor vendor to opt in or be okay
00:23:59with it, or actually do anything on their end.
00:24:02And to that point, the way that we're building a deck, I think is like fairly novel in the sense
00:24:08like it purely works over HTTP.
00:24:10And so the idea is you, we provide you with a URL and you can replace your existing web URL
00:24:15with the URL that we provided you.
00:24:17And then in a deck, your destination is what would have been your deck URL.
00:24:22And so we kind of work as a push base queue in between that.
00:24:25And so we'll get all the web through HTTP, and then we'll push those out at the rate
00:24:28that you specify to your own endpoint.
00:24:31But what that means is that you don't even need to redeploy your code.
00:24:34You can be on any cloud on any stack for this to work kind of basically right away.
00:24:39Right.
00:24:40Cause really the only requirement is to update the URL at the end of the day.
00:24:44And that's like kind of a far cry from AWS EventBridge, which is like ultimate vendor
00:24:48lock-in needs specific compatibility only works in AWS ecosystem.
00:24:53Yeah.
00:24:53I, but I, I usually think of most event bridge as kind of like the other big event gateway.
00:24:58We actually are seeing Azure event grid, which is like a competitor also like I've gained some ground.
00:25:04And so when I think of like our most direct competition or really like our inspiration in some
00:25:10ways it's, it's those two, those two products.
00:25:13But, but I definitely think like the scope of what we're trying to do is like quite a bit larger.
00:25:17And for some, for some people it's a good thing.
00:25:18For some people it's, it's not a good thing, right?
00:25:20Like for some people that's what they want.
00:25:22Like, you know, they're fully ingrained in the AWS ecosystem.
00:25:24They're using cloud formation.
00:25:26They're familiar with all the services.
00:25:27They want to be able to pick and choose the capabilities.
00:25:30And you know, that's fine.
00:25:31Like, I don't, I don't think we're ever going to have 100% market share.
00:25:34So actually really in the US, I think a couple percentage point is probably all you need.
00:25:38So yeah, I think we're trying to make the case that, you know, for aesthetic companies and
00:25:45developers and so on, that's going to be the right solution.
00:25:48And you know, for others, AWS event bridge is going to be in and that's fine.
00:25:51So I think that I've noticed in the past year is that the amount of downtimes for like big
00:25:59companies have been increasing.
00:26:00Like there's all these memes of like GitHub being down.
00:26:02And I remember like two years ago when I was working, like the P99, if you would,
00:26:07if it would drop like to 99.5 or something or lower, like you would have a serious conversation
00:26:13with your team.
00:26:14Like, how can you make this happen?
00:26:15Right, now it's kind of part of the course.
00:26:17Yeah.
00:26:17And now it's like, people have kind of gotten used to like GitHub being down for such a long time.
00:26:23So I'm thinking like, from your perspective on hook that are you seeing that there's also
00:26:28a lot more downtimes this year than previous years and how do you deal with those challenges?
00:26:33Yeah, that's interesting because obviously it's a consideration you have to do if you're going
00:26:37to consume WebEx from actually GitHub WebEx specifically are particularly problematic.
00:26:42You know, there's like a chance out of two, if you log in on like Railway or Versa or whatever,
00:26:46there's going to be a little banner.
00:26:47And it's like, you know, deployments are down because like GitHub WebEx don't work or whatever.
00:26:51And and actually the GitHub CTO made this this like blog post like on their infrastructure stability
00:26:58kind of in the, you know, in the in the midst of all the memes and all that sort of stuff,
00:27:02trying to like temper it down.
00:27:03I don't think it really worked, but there wasn't a time.
00:27:05But like one of the things that was explicitly blaming in this article was like WebEx stability
00:27:09and how it depends on like MySQL and like blah, blah, blah.
00:27:12Right.
00:27:13And to their credit, just because I think the way out maybe even phrasing this is a bit dismissive.
00:27:17Like I really don't doubt that just their infrastructure challenge is completely insane
00:27:21with the rise of agents.
00:27:23Like the same way we're talking about those OSS maintainers getting flooded.
00:27:26That's that's also to get them infrastructure flooded that right.
00:27:30And so so I'm sorry, I'm sure that's like extremely difficult challenge.
00:27:33But the point I'm getting at where like where I'm leading to you is that, yes, we can see it.
00:27:38And not only that, like we do actually monitor it for some vendors.
00:27:41So for instance, we have this thing we call we call it a deck radar.
00:27:44But the idea is that we can look at aggregated statistics from all the customers and all the
00:27:49WebEx that we get through a deck and then calculate things like, you know, what the delivery latency
00:27:53is like what the P99 delivery latency is for a vendor, what their uptime is and that sort of stuff.
00:27:58So for instance, like our radar for Shopify is like quite popular.
00:28:01I have a couple hundred like subscribers for it.
00:28:04And so the idea is we'll we send you alerts basically if the latency goes out like a certain
00:28:09like standard deviation of the baseline latency.
00:28:12Right.
00:28:12I think in the case of Shopify, like the baseline latency is around like four or five seconds for
00:28:17for delivery.
00:28:17I think it's like beyond like 10 seconds.
00:28:19Right.
00:28:20We'll we'll kind of send it alert.
00:28:22But the idea is you can like monitor this like a kind of like time series data and like look at
00:28:26what their uptime, not just in terms of like raw uptime, but also like their latency profile is
00:28:31over time.
00:28:32And that's definitely something that you see.
00:28:33And I mean, they're like specifically Shopify team as well aware of it.
00:28:36But you can see that there's a lot of variance and those delivery times.
00:28:41So there's going to be, you know, there's maybe once every two months or something like that,
00:28:46like a fairly like significant incident on latency.
00:28:49And I think that's something that usually you don't see.
00:28:51Right.
00:28:52Because obviously, like downtime is much easier.
00:28:54You go in your data dog or in your better stack.
00:28:57You guys call it again like the metric product.
00:29:01Observability.
00:29:02Observability, yeah.
00:29:03So you go in your better stack, obviously building that sort of stuff.
00:29:05Right.
00:29:06And like you look at like the the HTTP calls to the web again point and just like stop dead.
00:29:11All right.
00:29:11Like, you know, problems pretty obvious and that's straightforward.
00:29:14But like if the latency creeps up to 60 seconds, now that's something that's going to be very difficult
00:29:19to tell from your from your actual like metrics.
00:29:22Right.
00:29:23And so that's because that's on the latency that you take to process it, which is probably something you
00:29:27you'd be, you know, tracing and so on.
00:29:28But it's the delay on the vendor side.
00:29:31But now if you have a new assumption in your code, your business operation and so on,
00:29:34that assumes like, you know, some real timeness to those events, if they're now coming 60 seconds
00:29:39later, that can start introducing like a bunch of kind of like nuanced issues and bugs and bad
00:29:44assumptions and that sort of stuff in the code.
00:29:46So I think that's a place like where maybe like a bit more uniquely positioned to be able to
00:29:50like kind of provide like a good data.
00:29:52And so you can know like, oh, is it me or is it my vendor?
00:29:55Right.
00:29:56That's that's having issues.
00:29:57I don't know if I can comment like empirically on, you know, how much has gone up or down.
00:30:03And there are they like the usual suspects that they get out and so on.
00:30:06That's kind of like easy to point finger at.
00:30:08I don't know if I would say it's a genuinely distributed problem that we see all the time.
00:30:13I think it tends to be like more concentrated in specific vendors.
00:30:16And I think the part where people tend to get bit a bit more is on those delivery latencies.
00:30:23The other thing, too, is that I think people assume that delivery latency is lower than it actually is in
00:30:27most vendors, because we're definitely talking a couple of seconds in most cases, which depending
00:30:31on how you look at it, it might be a long time, might not be a long time.
00:30:34But but but I think when I show this like latency chart, for instance, Shopify to a bunch of Shopify
00:30:40developers, they have like a first like I did not expect that.
00:30:43Right.
00:30:43Like there's there's kind of like breaking from expectation that happens.
00:30:46I used to work for a fairly big e-commerce company and we had the same problem.
00:30:53Like the latency is bad, but like everyone was like the Spider-Man meme pointing fingers at
00:30:58whose team is responsible for it.
00:30:59Because usually it's basically a third party vendor, like you said.
00:31:05And sometimes it's just nothing you can do.
00:31:07You just have to work around it.
00:31:08So it's cool.
00:31:09You want to be careful, like blaming the vendor, too.
00:31:11Right.
00:31:11Like I think it's something as and obviously like in the context of what you guys do,
00:31:16you know, it's kind of always a common problem.
00:31:18Whenever we have an incident, it's like, OK, how much blame do you put on the vendor?
00:31:22But at some point, you know, there's also like what's being reasonable versus not reasonable.
00:31:27And what's a fluke versus not a fluke accident.
00:31:29Like, for instance, the other day, railway, we so some of the outpost management deployments
00:31:34are running on railway because they're pricing structure.
00:31:36Since we have to run the same version as the open source, we haven't built kind of
00:31:40like a multi multi tenanted that wouldn't be relevant and like open source.
00:31:44So basically we do like actual deployment for every single customers.
00:31:48Right.
00:31:49And so for free plan users, we'll kind of deploy an instance for them on railway,
00:31:52which generally speaking, like works pretty fine.
00:31:54But the other day, their GCP account got shot down and then they were down for like six
00:31:59hours or something.
00:31:59Right.
00:31:59And so then it's like, OK, you know, can you not not talk about it in your in your
00:32:06there's like one thing about, you know, like deflecting the blame.
00:32:09And I definitely like you have responsibility for your vendors.
00:32:11But the point is, I think the sign is pretty hard to walk.
00:32:14And I definitely see that there's like this natural tendency of people to like want to
00:32:18like point fingers because it kind of like de-responsibilize you.
00:32:20And so like you have to do like some fighting against that.
00:32:23Right.
00:32:24And but in the in the cases where you can be sure that it's the vendor,
00:32:28and I think it's like interesting to be able to like identify that and,
00:32:31you know, have the art data around it, which is often a problem with vendor problems.
00:32:35Right.
00:32:35You don't you don't see their side of the data.
00:32:37And so it's very difficult to like say with any certainty or like empirical evidence that
00:32:42it is clearly them.
00:32:43You have to like derive it from your own data, which could be right, could be wrong.
00:32:46Right.
00:32:47So I was wondering when there is an outage at something like Shopify or Stripe,
00:32:51how do the events play catch up and does that hit your servers really hard?
00:32:57Yeah, basically the Rex ma'am is what happens.
00:33:00And and I because there's those scenarios where like we always say that,
00:33:05like you shouldn't assume like a specific web of volume.
00:33:09Mm hmm.
00:33:09So we've got this characteristic where every vendor is going to enforce like a timeout.
00:33:14All right.
00:33:14They're going to give you like three seconds,
00:33:15five seconds kind of depends on the platform to respond.
00:33:18And so that means like you can't really do any meaningful amount of work,
00:33:21especially when you account for like network latencies and that sort of stuff in that in that delay.
00:33:25Right.
00:33:25And so that's why I mean, not just why,
00:33:28but it's like one of the reasons why the common thing is you need to queue it.
00:33:31You can't like process that synchronously.
00:33:33But I lost my train of thought.
00:33:39Can you repeat your question?
00:33:42I was talking about when there's an outage at a company like Stripe and Shopify.
00:33:45Oh, yeah.
00:33:45And playing catch up.
00:33:47Yeah.
00:33:47Yeah.
00:33:47Yeah.
00:33:47So anyway, all that to say latency, blah, blah, blah, convoluted way of saying you need
00:33:52to be able to auto scale.
00:33:53And so you need to be able to scale on like a pretty short time frame.
00:33:56And those spikes are coming from multiple different reasons.
00:33:58It might just be kind of like naturally you did like a bulk import when your customer did a bulk
00:34:03import or whatever.
00:34:03Right.
00:34:03There's other reason why spike would happen.
00:34:05But downtime is actually one of those.
00:34:07Right.
00:34:07There's this moment where if Shopify went down for half an hour, by the time they recover,
00:34:11they're just going to like plow through the backlog, especially that they're going to try to reduce
00:34:16the back pressure on their crew.
00:34:17So basically like work through everything that accumulated during the downtime.
00:34:22And so during speaking, what you'll see is that they'll even scale higher than their usual
00:34:26capacity because they're trying to like catch up on it.
00:34:28Right.
00:34:28And that results in basically just saying a bunch of requests to to the end stores.
00:34:32And so, yes, you have those irregularities, but some of those irregularities and throughput
00:34:37are not caused by you.
00:34:38It's like 100% caused by the vendor because, you know, vendor is going through his own
00:34:42infrastructure challenges and all that.
00:34:44And obviously like Shopify capacity of sending webhooks is much higher than your capacity at
00:34:49receiving webhooks for the most part.
00:34:52Yeah.
00:34:52Funny enough, actually, one of our investor was the CPO at Twilio.
00:34:56And I think that's part of the reason why I was interested in investing.
00:34:59Because one of the things that you said is that basically Twilio was like routinely
00:35:04just DDoSing their customers, you know, for all purposes.
00:35:07And it's a little bit inevitable and kind of like part of like the problem of doing those
00:35:14like, you know, vendor-driven events.
00:35:17And so it's very much kind of like a well-known kind of like problem with vendors.
00:35:21And they know it too, because that's something you can track in the response
00:35:24latency from the servers too, right?
00:35:26If you DDoS your customer, server response latency goes up.
00:35:29And at some point, they start timing out.
00:35:31That problem compounds because it's harder to have more throughput if there's longer timeouts.
00:35:34Like basically the longer you have to hold up the HP connection, the less requests you
00:35:38can make for any given worker that you have on your end.
00:35:40And so now you have to scale and then you send more because you scaled and you're right.
00:35:45It just kind of like piles up together, right?
00:35:46It tends to be very self-reinforcing because basically the least capacity you have to
00:35:52process those web books, the faster the latency degrades, right?
00:35:57And so as you reach kind of like saturation on your server, then latency goes really bad.
00:36:01But those requests like keep spying, keep piling up.
00:36:04And what a lot of vendors do is that at some point, they'll just like disable your endpoint
00:36:07because otherwise it's kind of keep building up, right?
00:36:10And they don't want to like be holding up, you know, hundreds of thousands of HTTP connections
00:36:14because your server is slow to respond.
00:36:15But then as soon as they disable it, disable it, then you lose the data.
00:36:19So that's the catch point too, right?
00:36:21So really ensuring kind of like a response time through those like variants are often out of your
00:36:25control is a big part of the challenge there.
00:36:29I also wanted to ask, with AI now, how is it changing the whole space of web hooks?
00:36:37I know that you said that like LLMs are now sending more web hooks and maybe processing them.
00:36:43But internally, like how do you use AI for hook deck?
00:36:48And what do you see as the future of AI in this space?
00:36:51Yeah, I think there's a lot of things going on on that front.
00:36:54Like kind of obviously, there's us as just a software development shop.
00:36:57And I feel like you guys are kind of like going through like the same challenges around,
00:37:02you know, workflows and workflows changing every week and token costs
00:37:05and how much is a reasonable amount of token costs and, you know, all that sort of stuff.
00:37:10Yeah, we don't have to do leaderboard.
00:37:15I'm very afraid of the leaderboard idea.
00:37:18It seems like a surefire way to destroy your margins.
00:37:24So a couple of thoughts.
00:37:25First, coming back to what I was saying earlier, you know, there's growth and means for events
00:37:29that are driven by those agentic use cases because agents are moving from, you know, being human
00:37:35triggered to event triggers and you're seeing, you know, all the cloud agent products and so on come
00:37:41out and lots of our stuff.
00:37:42And all of these are essentially triggered either by schedule or by events at the end of the day.
00:37:46And some of those events are kind of like abstracted away from you, but there still are things like
00:37:49GitHub PR or, you know, when you commit or when you comment on GitHub, like all this is web driven.
00:37:56But obviously, it's going to kind of like branch, like way beyond that, like the customer support
00:38:00and the slack and the blah, blah, blah, and even probably real time stuff that happens kind of like
00:38:04in the real world from like sensor data and all that sort of stuff, right?
00:38:08I think also a lot of agents will need to trigger events with other agents, basically,
00:38:12like the output of the agent is going to be an event, which is possibly going to trigger
00:38:17another agent, another company and so on, which, yes, you could kind of like describe
00:38:21as just under WebEx use case, and it kind of is.
00:38:23But I think also like some of the semantics around WebEx right now are kind of flawed
00:38:28around like security and around like the protocol and the efficiency and that sort of stuff.
00:38:33And that's why we've been kind of like pushing event destinations as a better pattern
00:38:36and kind of like a more optimized pattern for this. The other thing, too, is if what you run
00:38:40in response to events is agentic workflows, which are undeterministic, can run for a long time,
00:38:47that sort of stuff, it makes it much harder to actually adjust your capacity and scale kind
00:38:51of like in consequence to the events that you get. So I think throughput management and capacity
00:38:55management and all that becomes much more difficult. Your failure rates is going to be higher because
00:39:00of like just, you know, cloud timing out and just your evals not passing and all that sort of stuff.
00:39:07So I think from a application building standpoint and like the core primitives that you want to use
00:39:12and where that runs and how long that runs for and how you deal with your back pressure and that sort
00:39:16of stuff, there's a lot of like new challenges there. And then in terms of us kind of like internally,
00:39:21I think we're still trying to figure out the right way. There's part of me that's just like
00:39:25completely blown away by capacity. I'm not surprising anyone, like I don't think I'm bringing any
00:39:29like kind of unique insight there. The one thing I'll say though, is that the bar between like
00:39:34going from, you know, your CLI prompt or your codex or whatever to actually shipping kind of like
00:39:41high quality, tasteful product. I think some of that nuance is still being lost and all the memes and
00:39:45all the exposed and that sort of stuff where it's like, I have some doubts that you can actually spend
00:39:50that level of token and produce something that's actually value added to the world at the end of the
00:39:55day. And at least in our experience so far, obviously there's a lot of it around kind of like security,
00:40:01that it's like massive enablers, code reviews and that sort of stuff. But I think when it comes to
00:40:06like, those things are kind of like the table, table stakes. They're not a thing that like genuinely
00:40:11add value at the end of the day, right? Like when it comes to actually like building something very
00:40:14valuable that other derive value from, I feel there's still quite a gap there. And I feel like
00:40:19our team is still very responsible for like bringing that level of taste and insight and customer
00:40:25understanding and empathy and all that sort of stuff that you, that you just don't get straight out of
00:40:31like the chat box. And so maybe we're the one being foolish and we could be moving a lot faster. And
00:40:35if we just kind of like one shot everything and ran with it. But right now, our approach is still to
00:40:42keep that same level of expectations in terms of like the net output and what we deliver to end user
00:40:47at the end of the day. Yes, obviously, it means we can do more. And also, I think more people that
00:40:53have that taste and that empathy and so on can bring value to the user because the barrier was the code,
00:40:57right? And so for instance, like pretty much everyone in the company now quote unquote codes,
00:41:01right? Like our designer is like resident Vipe coder. Now he's the one like fully responsible for the
00:41:06the website, but even there's a bunch of like dashboard work and that stuff, you know, or product
00:41:11marketer or out of dev rel, like all, you know, not token maxing, but you know, if there was a leader
00:41:16to board, they're probably somewhere at the top. And I think that's a really good thing. It's enablers
00:41:20because those people have to have the kind of like critical thinking and all that, that was describing
00:41:25to be able to like ship like kind of tasteful end user things and like the barriers more the code.
00:41:30And so in some ways, I think like some of the largest benefit is going from there more so than
00:41:36like, like kind of pure engineering. Again, not to say that the pure engineering is not getting a ton
00:41:40of value from it, but I think the ball next still remains on on that taste. And plus, I'm like spending
00:41:45so much time reviewing kind of pointless PRs. Now it's like, there's, there's the downside to it too.
00:41:51Right. Like for sure. And the amount of code you have to review.
00:41:56Yeah. And like, let's make this obscure vulnerability that there's basically no chance.
00:42:01And like, I'm not even sure it even qualifies as low, right. But, but obviously once it surfaced,
00:42:06it's like, you feel a responsibility to it. And that's, I think that's normal. But like,
00:42:09at some point it was like, you know, we've never had this many PRs open at any given point in time.
00:42:14And it's just like getting a little insane to really wrap your head around all of this. And,
00:42:18and I think like bringing the judgment as to like, it's not because you can do everything,
00:42:21you should do everything. And I think like, it very much like comes with that LLM kind of like
00:42:26attitude. And I think like bringing kind of judgment to like, what will eventually derive
00:42:31in value or has the potential of like deriving in value is like more so important than ever.
00:42:34Cause there's, um, there's like the checkbox kind of like, uh, kind of to-do list satisfaction to
00:42:39working with agents, you know, like every once in a while, I kind of just brain dead. It's like,
00:42:44I, I just want to go through my list and all that things you are and just like, check, check, check,
00:42:48check. Right. And there's something that comes with that with LLMs where it's like, I start this
00:42:53conversation, start this conversation, start this conversation and maybe five or six agents going
00:42:57on and they're all like doing things, but like, and they all have checklists that they're checking off as
00:43:02well. Right. Right. Right. Right. Yeah. Yeah. So, so, so there's something about like kind of the,
00:43:06the, the satisfaction and the, the quick reward associated with that. And I think like sometimes
00:43:11like get the better of us. Right. Or at least like speaking for myself, how big is the hook tag team?
00:43:15We're a 10 right now. Oh, wow. That's super lean. Yeah. Kudos. It's always been an ambition of mine.
00:43:23I don't think of myself as the best manager. And so I think it's better for everyone that way.
00:43:29No, I'm kind of half getting, I think like from the get go, we were kind of born in the midst of COVID,
00:43:33right? Everyone's remote. And, uh, there was very much like this mindset from the get go of kind of
00:43:38hiring, you know, senior kind of experience, like extremely autonomous type people like to,
00:43:45to give a sense of it. Like we really just do like a single call every two week, uh, to discuss kind of
00:43:50like product and infrastructure. Um, and so like, we tried to like very much have this like asynchronous kind
00:43:56of like workflow. And I think, I think it lends, it lends itself well to AI use cases too. Cause like,
00:44:01we already hired and optimized for people that like add this autonomy. Right. I don't think there's any
00:44:06right or wrong. I'm not gonna like preach or like the way that we do things, but I think it works for us.
00:44:11Nothing works for me and my sanity. So there's that.
00:44:14I wanted to touch up on the thing you said at the very beginning of the conversation that, uh,
00:44:21uh, web hooks are dead. The new way is event gateway. Did you coin that term or was that somewhere in
00:44:27the ether already? Uh, what is an event gateway? I have no understanding of that.
00:44:32Yeah, we coined it and it's been very difficult to build a product and kind of like a product
00:44:37category that's not established. Uh, cause you have the challenges around us like marketing and
00:44:41communication and how you even like describe this thing. Right. And for a while we were talking,
00:44:46calling it like web book management infrastructure and that sort of stuff.
00:44:51Nothing that really rolls off the tongue. So, so you have the challenge on the communication,
00:44:54but you also have the challenge on the product. Cause when you're building a product in an existing
00:44:58category, let's say like better stack, I was ability, you know what you're trying to optimize for,
00:45:02like your specific USPs you're going for, which in your case, I think a lot of it is around like pricing
00:45:07and DX and that sort of stuff. Right. But the point is like the core semantics exist. And so what
00:45:10you're optimizing for is that unique value position. When you're building in something that doesn't
00:45:14really exist already, you have to invent the semantic and then also build like a compelling
00:45:19value proposition on top of those semantics. Right. So that becomes like very difficult.
00:45:24And that's one learning for the last couple of years, very difficult to build a product in the
00:45:28existing category. And so the idea with event gateway and where that came from is
00:45:32trying to like capture as best as we could. And, you know, two or three words that hopefully roll
00:45:37off the tongue kind of like what it does. And so our goal was to like bridge between event bus
00:45:43and basically API, API gateways. Right. And kind of capture this notion that like gateways are there
00:45:49to, as an interface, the interface between like a vendor and your own system and so on. Right. And
00:45:55event bus in the sense of like, this is full blown, like event management and queuing and, and so on.
00:46:00Right. So we're trying to like find a term that kind of like marries those two. And really when
00:46:04you think of event bridge, you know, not too far from event gateway. And so really like, we want
00:46:10people to like perceive us as like competing with it was event bridge, for instance, like we see
00:46:14yourself as like a clear alternative to it. Right. And so I think the idea around event gateway was also
00:46:19like, give a sense of like, look, there are existing products right now. There's no common terminology
00:46:24around those things. Right. They don't like outright call it an event gateway, but we'll give that a
00:46:29label. And then there's, you know, a couple of already competing product kind of like in that category,
00:46:33one of, uh, one of, one of ours, but then there's AWS, there's Azure, there's others like Kong is an event
00:46:39gateway. Now there's gravity. There's a bunch of like API gateway providers are kind of like going into
00:46:45event driven architecture as well. So at this point, there's probably like, you know, half a dozen product,
00:46:49at least that would call themselves event gateway or event gateway like, and I don't know if we had
00:46:54anything to do with this, but I can tell you when Kong released their event gateway, I was like, Oh,
00:46:58shit, like it was like two years in from us, like calling it that way. But the part that's like,
00:47:03actually rewarding is when customers come to you early developers come to me and they're like, I am
00:47:08looking for an event gateway. And it's like, Oh, like, that's, you know, now you're like, okay,
00:47:12like, I think the term is like getting somewhere in terms of like, people can I have a mental
00:47:16model with that is, or like starting to like explicitly look for it, or they might say like,
00:47:20we're trying to replace our own event gateway. That's things I've like come up to in the
00:47:23conversation. But I can tell you for like the first year, that would not not come up. And now like,
00:47:28as time goes on, it's a term that I see other use like more and more. I think that's a good thing.
00:47:34I think it's a good thing for us as a business, but also just like, in the sense of like trying to
00:47:38create a set of expectations of like, okay, this, this is this cloud infrastructure primitive that exists.
00:47:43And yours are the, you're like the basic set of expectations that you can have around it. And
00:47:48hopefully at some time, like at some point, like things will start to align and semantic and that
00:47:52sort of stuff too. Like if you're using AWS event bridge and a deck, like terminology is like very
00:47:57little in common. Now, I don't want to design my product around AWS event bridge because of all
00:48:01the product that it has. So I'm definitely not going to reuse their terminology if I don't think it
00:48:05makes like, you know, rock solid sense. But I do think like over time as you know,
00:48:09there's more condition and more people building and people at around the product, that sort of stuff,
00:48:12there's probably going to be like some amount of convergence.
00:48:15Yeah.
00:48:15Do you find if I asked for an event gateway on an LLM that it would recommend you guys?
00:48:20Oh yeah. Try it now. I think, I think the odds are pretty good.
00:48:24I could try it live.
00:48:25It's something we track, right?
00:48:27Yeah.
00:48:27It's something we track. We get cited in about like 60% of every single,
00:48:31every single prompt that is around like WebEx or event gateway, that sort of stuff.
00:48:35That's cool. And something that we obviously like spent a lot of time on. And as far as our data
00:48:42goes, which might not be at that high quality, we're doing a pretty good job at it. So
00:48:46It was the first one in the list.
00:48:48Oh, nice.
00:48:49There you go.
00:48:49That ChatGPT just gave me. So well done.
00:48:52Yeah. Although I think on event gateway, I think we'd probably do just as good
00:48:56on any kind of like WebEx related queries and that sort of stuff. But event gateway is definitely
00:49:00like kind of playing to our favor in the sense, like we coined the term, like I'd,
00:49:03I'd be mad if we weren't first.
00:49:05I actually now realized I don't actually know what event driven architecture means. How does it differ
00:49:11from a regular architecture? Like if I, if I slap on a Kafka queue in my system,
00:49:17am I automatically an event driven architecture?
00:49:21Yes and no. In the sense that I think when you're talking about event driven architecture is like
00:49:25more of a set of like paradigm and expectations that you have around it. So part of it is going to
00:49:31be, yes, the tools, right? Like you're using Kafka, you're using message queues or event
00:49:35streaming that are kind of meant to decouple systems. So really when it comes down to it,
00:49:39the idea is they have producers and consumers, those systems don't know to,
00:49:43don't need to know about each other, right? So the consumer can consume from a stream of
00:49:47event that can be produced kind of from anyone. And the only real set of expectation that you have
00:49:53is like a contract around what is the event itself. Like what is the payload, what is the shape?
00:49:57Yeah. And oftentimes there's going to be like schemas, like specific schemas that people will use,
00:50:02like kind of Avro based schemas and pro buff and other thing, right? Where there's gonna be like
00:50:06standardized expectations around what those payloads are and so on. And really, so the contract becomes
00:50:11around, okay, these events exist. And at some point I might have an event and as a consumer,
00:50:16my responsibility is to do whatever I'm supposed to do with like a order created or a product update and so on.
00:50:21Right. But now an organization that really embraced EDA is what you'll see is kind of like a
00:50:25systematized kind of pattern around this, right? Where there's going to be like specific
00:50:29guidelines around how you're supposed to do this and what the payload shapes are supposed to be.
00:50:32And it's going to be like an architectural approach where you will intentionally decouple
00:50:36their services and create the communication between those like event driven, right?
00:50:41I feel like a lot of companies take like a hybrid approach, right? Something is event driven,
00:50:45but something is not, right? So is the goal to be 100% event driven?
00:50:50Or is it just like a subset of an architectural paradigm?
00:50:54I don't think it's 100%. I just think that over time, as you grow and complexity increases,
00:51:00and there's more third party dependencies and so on, it's kind of like the natural way that things tend
00:51:04to go. Because at some point, it's just very difficult to have all those systems coupled.
00:51:09And then you have to think about like the scaling and interdependencies and all that sort of stuff
00:51:14in such a way that makes it pretty difficult. But generally speaking, when we think of like
00:51:19adventure in architecture, in the sense like the term, I think people like think,
00:51:23I mean, rightly so, like think like big enterprise, right? And I think for probably like the first
00:51:28decade of adventure in architecture, it was mainly concentrated in big enterprise. But I think now
00:51:32that's changing part because of people getting more comfortable with the patterns, in part because
00:51:37WebEx are a gateway drug to it. And but also like some of the tooling getting better. Like I'm thinking,
00:51:41I'm thinking like, for instance, like the rabbit in queue, or like the the bull MQ, which is built on
00:51:46Redis, kind of like libraries, there's also like a bunch of popular libraries and Python, there's
00:51:51celery and Ruby, there's sidekick. I think there's another one now, like foundation, that's also from
00:51:57the folks that sidekick. So anyway, like there's been like kind of progressive like tooling around
00:52:01that also makes it like easier to work with and less and less intimidating to go into where you don't
00:52:05just like need, you know, big architecture team and big Kafka deployments, that cost a bunch of
00:52:11bunch of money and so on to kind of like start doing adventure in architecture. But I also think the term
00:52:15has lost a little bit of its meaning. I'm sure some people wouldn't be happy with me saying that. But
00:52:19I do think it's like lost a little bit of its meaning over time, or at least like the water is muddied in terms of what we mean.
00:52:25And I think when when I say it, what I'm referring to is mainly this decoupling of systems. And then
00:52:31also the programming concerns that you're not going to have as like an engineer in terms of like,
00:52:37you know, having to think about independency and ordering and all those sorts of concern. And so when
00:52:43I say adventure in architecture, it's more and I meant like you're now entering into the world where
00:52:48when you build your app, you're gonna have those sets of concern that you didn't have before.
00:52:52Right. Got it. And and being able to like learn and adopt like how you build things to be able to
00:52:58accommodate for those for those concerns. I don't know if I initially mean in like kind of like the
00:53:03big enterprise type way. And I think in some sense, like what we're doing is trying to like, you know,
00:53:08bring that bring that to kind of like everyone in some ways. Right. And we're not we're not the only
00:53:14player kind of doing that. Right. There's a bunch of other people that are taking different approach.
00:53:18There's kind of all the workflow engines and step functions there that I think like overlap, like
00:53:22temporal and ingest and trigger.dev and like those folks as well. So so I do think it's actually a
00:53:28space where a lot of things are going on. And there's maybe like no real good fixed definition
00:53:34for it anymore. For those that are really interested in kind of like learning more
00:53:38about event driven architecture and like the paradigms associated with that and so on. There's this guy
00:53:42called David Boyan, who was previously a developer advocate at AWS and actually was working on event
00:53:49bridge. That's those series of kind of like drawing type explainers. But at this point, he probably has
00:53:55like several hundreds that he also runs a open source project now called event catalog might be another
00:54:02good podcast guest for you guys. But all that to say like it goes in like the amount of depth that's
00:54:07just like is almost unreasonable. Right. So it's definitely recommend those for for those that are
00:54:14curious. Cool. We'll add it to the show notes. Yeah, as all of your knowledge on events and web
00:54:19hooks and everything. So see, you have quite a lot of it all come from just sort of building hook deck.
00:54:23Or was there obviously you said before that you had a few issues with web hooks, but was this when you
00:54:27really dived deep into events? Very much so. And I think it's like learning that came very much from
00:54:34the problems, but also like first principles in the sense that like, when I was dealing with those
00:54:38issues, I didn't actually have a ton of knowledge around it, which was like part of why I was dealing
00:54:42with those issues. Yeah. So I think it came very much from a place of like, okay, trying to like work
00:54:47through the problem and the solutions of that problem, rather than like working through solutions and
00:54:51retrofitting them to the problem. Right. But but at the end of the day, like at this point,
00:54:55like we've worked with like hundreds of thousands of whoppers I've like, you know, from all across
00:54:59the spectrum. And so I think I just like soaked up from those conversation. And yeah, it's basically
00:55:05it's been pretty interesting. I actually started as a product designer originally. So so kind of turn,
00:55:11you know, product designer, kind of full stack developer, then like back in the offer,
00:55:15then like infrastructure engineer. And now it's like, now kind of like very, very deep down the
00:55:21rabbit hole. But but but that came through just kind of like working through customers and listening
00:55:26to her concern and reviewing the architecture and all that sort of stuff. But also from the team,
00:55:30right? Like we have people on the team as well that have like a ton of experience working with
00:55:33those systems and also like kind of brought that knowledge to the company. Yeah. When did you,
00:55:38you realize you found a market for this? I assume obviously you started working on it and then
00:55:42posted it somewhere. And was it a good reception almost immediately? Or is it being quite a sort
00:55:47of slow burn to get people to know this is the better option is slow, slow burn hardly defines it
00:55:53in the sense that it started with that medium article is referring to, right? Like this kind of like web
00:55:57sex and there's something you can do about it. I'm like very much like the mindset of like build product
00:56:01for people. And so like the very first version was kind of like a self service, you know, you could
00:56:07like go in and like create, you know, your first like connection that we call it and all that sort of stuff.
00:56:12And, and posted this article and stuff. And there's no ambition to like make a business out of it or
00:56:16whatever. It's just like one out of like the 20 other like failed site projects, right? That I kind
00:56:21of like been working on at the time. And, and I mean, in kind of like retrospective, like now the
00:56:26numbers seems like just ridiculously small, because there's maybe five people that like reached out
00:56:32from that article or something. But I know for like, for everyone that's like worked on site projects,
00:56:37and, and kind of like been through the motions of trying to like get someone to use something that
00:56:42you've built and so on. Like five is fucking insane. It's like five is like better than I probably ever
00:56:48got before. Right. And so, so I had times kind of like super excited by it. I was actually like in my
00:56:55camper van, British Columbia, rock climbing. And I was just like very far from, you know, trying to build a
00:57:00startup or whatever. Just kind of like having crustacean with those folks, like walking,
00:57:05like walking them through it, like I was thinking and the problem they were having and that sort of
00:57:08stuff. And then, and then kind of like kept trickling in like maybe one a week and be like kind of rock
00:57:13climbing. And I like pull up slack and it's like the slack notification. We have this like notification
00:57:18channel for every signups, right? Like the same one that we've had for like six years or something.
00:57:21At this point, it's like pretty difficult to even like keep track because it just kind of like scrolls
00:57:25fast. But, but, but like we'd have like that, I'd have like that notification, that channel. I'd be
00:57:31like, oh shit, you know, like, you know, someone signed up. Are you DDOSing slack?
00:57:38No, definitely not in those days. But right now, you said you still have it right now. So that's why
00:57:44I'm asking. Right, right, right. Well, I mean, like things are going well, but I don't think it's to
00:57:48the point of DDOSing slack. Okay. Anyway, so, so guys, sorry from there, but I think like the one core
00:57:54assumption at the start that was like problematic was, was this idea that it's mainly built for
00:57:58obvious ability. I think it's just like, wasn't far enough. But once you go from saying like,
00:58:02I'm building an obvious ability tool to I'm reinventing a message bus, like suddenly like the scope just
00:58:08like blows open dramatically. Right. So I actually kind of met CTO and co-founder kind of like in the
00:58:15midst of building like the first kind of like queuing engine. And, and so, and so when kind of like,
00:58:19I realized like, okay, like the scope is going to like blow white all open on this. Like that's when
00:58:23we also like went for, for like a small, like pre-seed, but like angel investors and that sort
00:58:28of stuff. Right. It was kind of like pre-evident that, you know, it was going to be pretty expensive
00:58:32to build, which turned out it was pretty expensive to build. And as I understand, you didn't go the
00:58:36typical SF route. You built it in Montreal, right?
00:58:39Uh, yeah, that's maybe a little bit disingenuous for what actually happened. So we did a, we did a
00:58:44pre-seed about like $400,000 from just angel investors, like no institutional VCs. But then
00:58:51a few months in we did our accurate news and that's really when we kind of knew, uh, coming back to
00:58:56your question, James, like, I think like the response from accurate news, I mean, it wasn't
00:59:00like the biggest accurate news, like show HN ever, but it was like much better than we had like ever
00:59:05expected and funny anecdote. But I remember, I remember this one guy that bought like a $300 plan,
00:59:11right? Like after HN. And I remember like my co-founder and I were like, dude, we made it.
00:59:18Like it's a done deal. Like we're like, we for sure making it.
00:59:22Went out for dinner that night, spent it all. It's like more.
00:59:25Yeah, that's exactly it. Right. Like the bowl of champagne. Yeah. But, um, yeah, anyway,
00:59:31so like from that show HN, that sort of stuff. And then at that point, like we had a bunch of
00:59:34interest from investors kind of like proactively reaching out and that sort of stuff. And so we,
00:59:38we did end up like raising around from a Silicon Valley investors, like the firm called matrix
00:59:44partners. So we're still based kind of like the centralized team, like Canadian company. We haven't
00:59:50changed to a Delaware LLC, but, but at the same point, like, I think like our investors have been
00:59:55like tremendous and I'm like very, uh, glad that we did it. And I think like, because of COVID and
01:00:00so on too, there's maybe like been growing acceptation of like companies are not like
01:00:04based on this stuff and like build remote teams, that sort of stuff. I mean, like a bunch of people
01:00:08have been doing, like we didn't invent anything like the, uh, there's still like the Zapier and
01:00:12the GitLab and all those guys that have been like doing this for like way longer than anybody else.
01:00:17Right. So I think it just maybe like normalized it. And, and that's something I've seen like from some,
01:00:21maybe I'm just naive about this, but that's something I've seen from some founders. There's like,
01:00:27you know, as Canadian founders, this bigger station and kind of like Canadian founder Twitter around,
01:00:32you know, incorporating to Delaware and YC stopped accepting kind of like Canadian companies. And then
01:00:38they ended up like reverting that decision and so on. But there was this kind of like understanding
01:00:42like every investor is going to ask you to, to become Delaware LLC. And, and they're right.
01:00:47Every investor is going to ask you the part that misses that story is that you could just say no.
01:00:54Um, so yeah, for sure. They'll ask like, you know, like why not? And it's easier for them,
01:00:58but turns out if you just say, no, it's like also totally fine. So at least in my experience and
01:01:03a lot of people that like, that are kind of like around me and so on. So I think that's the missing
01:01:07part of that story, right? Like at the end of the day, like they're looking, their job is to deploy
01:01:11capital. They're looking for people to invest in or looking for original ideas and that sort of stuff.
01:01:15Like, I don't want to like argue about the pros and cons of being in the stuff. And I'm sure there's like
01:01:19a lot of positive to it, but you know, at the end of the day, like it's your choice as like a
01:01:24founder and a builder, like who you're going to do it with and where are you going to do it with.
01:01:27And I think if you have a good ideas and you put the work behind it, like investors will still
01:01:31respect that, or at least, at least some investors will still respect that. Right. And so maybe your
01:01:36job is to find them, but yeah. So I guess it would be disinjurious to say that we are completely
01:01:41outside of that bubble, but, uh, but like, I know personally, uh, you know, made my life here and
01:01:47you know, I'm pretty glad to still be in Montreal personally. Well, as a fellow Canadian, it's nice
01:01:52to hear like Canadian success stories. So kudos to that, but then you moved to Toronto and then you
01:01:57just like, fuck it all up. Fair, fair. Can I say as a fellow Commonwealth, like great, right? Same
01:02:06king. It's the same thing, you know, right? Yeah. Yeah. We, we do have the queen. I mean,
01:02:11I don't know if they have updated to the king now, but we did have the queen on our bill.
01:02:14Yeah. Was hook tech your first startup as a founder or were there others on the journey that gave you
01:02:19sort of the tools to know how to reach out to investors and find them in a way. I've always
01:02:23been like very entrepreneurial from like the get-go at this kind of like small companies, like going to
01:02:27like retiree zone, like repairing computers at like 14 or 15 and stuff. Right. Um, so I guess in the
01:02:34sense, like more like talking to like the entrepreneurial spirit, but no, I think like
01:02:38most of it was like failed site projects, like releasing like a video game that basically never
01:02:44got anywhere. You know, a bunch of other, like that at some point I was working on some like social
01:02:48network, despite, I'm probably like the, the worst guy at like social gatherings and like organizing
01:02:53things with friends. Anyway, like kind of went through like the motions of things. I will say though,
01:02:58this experience, like at the sea commerce company, it actually was pretty formative in the sense
01:03:03like I joined us the first employee there and like, it was very kind of like involved with like the,
01:03:07the founding team. And we went from, you know, basically like four people to like 40 or something
01:03:13in like three years. And, you know, all the motion of like building that business and all that,
01:03:19all that stuff there. So, so I guess like, I think of exec as like very much kind of like the,
01:03:23the first one, but it'd be like disingenuous to like fully position it as that. I think like there's
01:03:27some like exposure to those things kind of like prior to that. And obviously I had this e-commerce
01:03:31company. Like we, we worked with investors as well and you know, like the board and that sort of stuff
01:03:36and like built relationships there as well. And then the first investor actually in that e-commerce
01:03:39business was also our first initial investor in a deck. So it wasn't like kind of fully starting
01:03:44from scratch in the sense of like, you know, some people might find themselves into, but yeah,
01:03:48I guess I still did not expect to be here, you know, a couple of years ago. And that's great.
01:03:54I saw there's a company called Kiwi Mornings, healthy breakfast startup. What was that all about?
01:04:02And then your homework. So, so that was one of the failed business along the way.
01:04:09I actually started with my wife. We actually like the, the story goes that my wife was, you know,
01:04:14bringing her breakfast at work and all the sales guys at work were kind of jealous of her breakfast
01:04:19and started asking her for breakfast. And I was like, what do you mean your breakfast was sales
01:04:24guys at work? What is wrong? But then one thing I liked another and she's maybe like making a five
01:04:30or six breakfasts like every morning for, for the sales team. And they're like, ah, why don't we,
01:04:36you know, one of those like stupid ideas, like why don't we make a business out of this? And so
01:04:41we built this like service for basically zero waste breakfast at work. And so we deliver kind of like
01:04:47yogurts and like smoothies and like chia puddings and like that sort of stuff and like little glass jars.
01:04:51And we have like a little fridge and the offices and so on. And, and me being me kind of took it
01:04:56way too far. And like the product and engineering side, like it was all orders were like through a
01:05:01slack bot and like, uh, employers could you like have it as an employee benefit where essentially
01:05:06they'd pay like for 50% of the breakfast. And you know, there's this kind of like this whole thing.
01:05:11And so yeah, people would order their breakfast through the slack bot and all that, not necessarily
01:05:16it came to an end. It's just that in that business, like everything goes wrong at four in the morning
01:05:20and plus there's no money in food. And so when you combine those two things, it's like very difficult
01:05:25to build a fulfilling business. And then so at some point we, we ended up kind of like selling the
01:05:29slack bot and all that and like kind of stop stopping, stopping doing it. Um, but thankfully,
01:05:34that was in January, 2021. So like two months before COVID. Um, and then basically every
01:05:42equivalent like companies, like there's a lot of them already doing kind of like lunch and that sort
01:05:46of stuff like office lunch and all that. Basically everybody went out of business. So, uh, we kind of
01:05:52got a little bit lucky on the timing there cause it would have ended no matter what. Right. Like,
01:05:57yeah. But all in, I think we probably sold like 20,000 breakfasts or something, I guess.
01:06:01Oh, pretty cool. Yeah. It's pretty cool.
01:06:06Oh yeah. So we always like to ask, yeah, we always like to ask our guests, what are your hot takes
01:06:12about the industry, AI, whatever, go for it. I feel like I've given a few already in this conversation.
01:06:18I think you have. Yeah. What is your hottest steak? Like burning hot.
01:06:22Burning out of steak. Okay. This is probably where I'm going to lose you because it's very much in the
01:06:29weeds of Avenger and architecture. I'm sure some of our audience members will understand what you're
01:06:35saying. Perfect. So my autistake, uh, autistake is that poll based consumers or like poll based queuing
01:06:42system are just very dumb compared to push based system. And the reason why we haven't adopted this
01:06:49push based system is because no queue is built a good push based system. And the reason why I say
01:06:56this, and now I'll try to give a little bit of context to it is that when you build kind of queues
01:07:02and consumers for every queue that you put events in, you need to have a consumer for it. And that
01:07:07consumer might be like some kind of long running worker that's like pulling off that queue. But the
01:07:12problem is that if you want to dynamically have queues, and so let's say you want to have a queue per
01:07:16customer, right? Because you don't want one customer to like bugle up the whole queue or
01:07:21each one of their located capacity, that sort of stuff. Now you need to have as many consumers as you
01:07:26have queues, which gets insane because you have this like multiplexing problem where each queue needs
01:07:31its consumer. The big advantage with push based system is that all those queues can push to the same
01:07:36consumer. And so that one consumer can be like an API with a low balancer in front of it that you can
01:07:41scale horizontally, or I mean, or vertically, whatever. But the point is, like, you can do
01:07:45that completely decoupled from how many queues you actually have. And I think one thing that we're
01:07:49seeing is like, once you get into like more and more complex use cases, you end up with like more
01:07:54and more granular ways that you want to queue things. So you want to queue per topic and per specific
01:07:58conditions and per customer and that sort of stuff. And it gets like completely insane because then you
01:08:02have like 100 queues and 100 consumers and 100 dollar queues and like all the all the all the mess that
01:08:07comes around with those. But it's very difficult today to find push based message message queues.
01:08:13And the reason for this is because you're shifting where the throughput is being controlled. If the
01:08:17throughput is being controlled in the consumer, each consumer is responsible for saying, hey, I want
01:08:2250 message per second or concurrently or whatever. And then how many messages you're consuming becomes,
01:08:29what's the capacity of a worker? How many workers do you have? And also what's the effective
01:08:33capacity? It's not because you say you want 50 per second. It's actually going at 50 per second,
01:08:37because it depends on all the rest of the code and whether or not it's fast enough and so on.
01:08:40Right. So I think the reason why we haven't moved to bush base queues is that most of them don't
01:08:45actually give you the necessary granularity and controlling throughput. So for instance,
01:08:50like GCP Pub/Sub does have a push mode, but in push mode, they basically like ramp up the rate at
01:08:55which you send your requests until your API starts to slow down, which at this point they basically have
01:09:00degraded it. And then once it starts to slow down, then they ramp back down the delivery rate. And so what you end up
01:09:06with is that it creeps up, creeps up, creeps up, server crash or has serious performance degradation,
01:09:11goes all the way back to zero, and then creeps up, creeps up, creeps up, creeps up, goes all the way
01:09:15back to zero. And it's just like completely done. Like it makes no sense. Right. And I think if you
01:09:20build push base mission skews, and that's like one of the quests I'm in where you have very fine grade
01:09:24control over the throughput and the exact behavior of the consumption rate, it simplifies so much of your
01:09:29architecture. So that's all I take for the people that knows and, but, but, but, but ill, I'm willing
01:09:34to die on. I mean, like, I didn't understand all of it, but it sounded reasonable, you know?
01:09:40I think if you did understand that, go check out hookdeck. Yeah, exactly.
01:09:46Appreciate that, yes. Well, thank you, Alex. Thank you for listening to this episode of the
01:09:50Better Stack Podcast. Find us wherever you get your podcasts, Spotify, Apple Music, or anywhere else.
01:09:57But today it's a goodbye from me. Goodbye from me. And a goodbye from me.