스크립트
00:00:00Google just released a new runtime library that lets you run machine learning and AI models
00:00:05inside the browser at near native speeds. It's called LiteRT.js and it's super impressive. So
00:00:12in this video we'll take a look at LiteRT.js, see how it works, and we're going to test it out by
00:00:18building a real-time 3D motion capture application running entirely in the browser. It's going to be
00:00:24a lot of fun, so let's dive into it. So what exactly is LiteRT.js? Well, it's a JavaScript binding for
00:00:36LiteRT, their on-device inference runtime, that's been running models on Android, iOS, and embedded
00:00:42hardware for years now. But LiteRT.js brings the same runtime into the browser via WebAssembly using
00:00:50the LiteRT.js core npm package. And here's what makes LiteRT.js different. So Google already has a
00:00:58browser machine learning library, it's called TensorFlow.js, and it's been around for years. But
00:01:04TensorFlow.js runs on JavaScript-based kernels, and that is a big bottleneck. You see, JavaScript kernels
00:01:11can't fully exploit CPU or GPU the way a native runtime can. So TensorFlow.js has always lagged behind
00:01:19native app performance. But LiteRT.js uses WebAssembly, and it ships with its own optimized kernel.
00:01:27So the same native runtime that powers Android and iOS inference is compiled to WASM and then exposed to
00:01:34LiteRT.js. So you're not getting a web-flavored abstraction layer anymore, you're actually getting
00:01:40a truly optimized runtime engine. And this also shows up in their back-end architecture. LiteRT.js has
00:01:47three different tiers that allows it to perform well on CPUs, GPUs, and even NPUs. So on the CPU side,
00:01:55it uses XNNPack, Google's optimized CPU kernel library, which is multi-threaded with relaxed SIMD support.
00:02:04And this is your universal fallback that will run anywhere, no GPU required. But for GPUs,
00:02:11it uses ML drift over web GPU, and this is where the real performance lives. It utilizes native GPU
00:02:18kernels, so things like shaders are no longer rendered using some sort of JavaScript orchestration. And for
00:02:24MPUs, it has WebNN, which is still experimental in Chrome and Edge, and it targets dedicated neural
00:02:31processing hardware for power-efficient inference. And according to Google's own benchmarks, which they
00:02:37conducted on a 2024 MacBook Pro with M4 Silicon, it appears to have a three times faster inference than
00:02:45other web runtimes on CPU and GPU across vision and audio models. And when running workloads like
00:02:53object tracking, audio transcription, or image manipulation on the GPU or MPU, rather than on the CPU,
00:03:00we get performance boosts that jump from five times all the way to 60 times faster results,
00:03:07depending on the task. And it's worth mentioning that that is the best case scenario. And Google also
00:03:13acknowledges this. So your mileage may vary, taking into consideration your specific GPU,
00:03:18thermal throttling, and driver quality. And there's one more thing that really matters for this runtime.
00:03:24If you already have models that you use to run on PyTorch or TensorFlow, if you've got an existing
00:03:30TF Lite file, LightRT.js can actually run it directly. And if you're on PyTorch, there's also LightRT
00:03:38Torch, a conversion path that takes your model straight to .TF Lite. Plus, there's also an AI Edge
00:03:44Quantizer for shrinking model sizes without a full rewrite. So legacy workflows you've been using on
00:03:50TensorFlow.js can easily be ported to the new LightRT.js, which is super cool. But what can you
00:03:58actually do with it? Well, Google has some pretty cool demos out there, which give you a sense of the
00:04:03range. So you can do real-time YOLO object detection, monocular depth estimation with the
00:04:09depth anything library that turns your webcam feed into a live 3D point cloud. And it can also do image
00:04:16upscaling with the real ESR GAN library. And all these demos are running completely client side with
00:04:23no backends, no APIs directly inside your browser. And they've already announced what's coming next,
00:04:29LightRT.LM.js. And this is going to be for running full language models locally in the browser.
00:04:36So this isn't just for computer vision. Soon it will also offer local LLM inference too.
00:04:42All right, that all sounds great, but I wanted to test it out on my own to see how powerful it actually
00:04:48is. So for this demo, I decided to create a real motion capture application that uses your webcam for
00:04:55real-time pose estimation. And it also runs entirely on your browser, client side, and offline with no
00:05:03strings attached. So I vibe coded this app on cloud code using the new Fable 5 model. And here's the end result.
00:05:10And by the way, if you want to download this repo and play around with it yourself, I've also added a link to
00:05:15the project in the description below. So here LightRT.JS is running through the browser using a blaze pose model
00:05:23with 33 body landmarks, driving a 3D character in real-time using 3.js. There is no backend server and
00:05:31everything you see is happening on this machine right now completely offline. So we can see here that on the
00:05:38CPU, we are averaging around 38 FPS with an inference of 23.3 milliseconds. And we can also see that the pose
00:05:47estimation is lagging behind a bit. But now if we switch to the web GPU version, we can see that we are now
00:05:54getting a solid 120 frames per second with an 8.4 millisecond inference. And the pose estimation is also a bit
00:06:02faster. And that's roughly a three times jump in frame rate and a 2.8 times drop in inference time, which
00:06:10lines up with Google's own numbers, though it's on the modest end of what they published. But we have to
00:06:16take into consideration that blaze pose is genuinely a small model. So there's less raw compute for the GPU
00:06:23to chew through. But even with that, moving the tensor math off the CPU and on the web GPU cuts the latency
00:06:30enough that the lag is significantly lower. And I also added a feature where you can record your pose
00:06:36capture animation and export it as a JSON or as a bio vision motion capture file, and then open that in
00:06:44something like blender and retarget that animation to your own custom character. And as you can see here,
00:06:49it's not perfect. The movements are not very detailed or refined. So this is still very much a work in
00:06:56progress. But it's super cool that I was able to get this initial version working in just 10 minutes
00:07:01using the new LightRT.js. So there you have it, folks. That is LightRT.js in a nutshell. It's
00:07:08honestly amazing to see how far WebAssembly has pushed the capabilities of running complex
00:07:14applications on the browser. And this new LightRT.js library truly proves that sometimes JavaScript can
00:07:21be a real bottleneck for getting to those near native performance speeds. So I'm super impressed with this
00:07:27library and I can't wait to try out LightRT.js when it finally ships later this year. But what do you think
00:07:35about LightRT.js? Have you tried it? Will you use it? Let us know in the comments down below. And folks,
00:07:40if you like these types of technical breakdowns, please let me know by smashing that like button
00:07:45underneath the video. And also don't forget to subscribe to our channel. This has been Andres from
00:07:50BetterStack and I will see you in the next videos.