Written with AI assistance from the source video. The video is the authority.
Log in to leave a comment
No posts yet
Running AI models in a web browser is exciting, but you quickly hit limits. If you try to process high-resolution images or start heavy computations, the screen starts to stutter. TensorFlow.js (TF.js) uses the WebGL backend and JavaScript kernel bindings, which create significant overhead in large matrix operations.
On the other hand, LiteRT.js compiles the C++ native runtime into WebAssembly (Wasm) and ports it to the browser. This architecture fundamentally solves the bottleneck. Furthermore, the process of bringing PyTorch models to the web becomes much simpler. Previously, you had to go from PyTorch to ONNX, then through TensorFlow, and finally into the TF.js format, which often led to broken operator compatibility and loss of precision. Now, you can just use a single library (ai-edge-torch) to export a standard .tflite file and you're done.
If you don't want to discard your existing assets, you can use the @litertjs/tfjs-interop package provided by Google. This allows you to keep your data preprocessing and post-processing in the TF.js pipeline while swapping out only the core model prediction execution part for LiteRT.js.
TFLite models converted from PyTorch usually require an (channels, height, width) channel-first structure. However, the browser's Canvas ImageData array is in (height, width, channels) format, where pixels are laid out sequentially. You need a preprocessing function that synchronizes this gap without loading the main thread.
First, create a Float32Array by multiplying the total number of pixels in the ImageData by 3. Then, designate the flattened offset positions for the red, green, and blue channels as 0, the total pixel count, and twice the total pixel count, respectively. Finally, normalize the pixel values (which are between 0 and 255) by dividing by 255.0 and assign them to each channel offset. This process rearranges the input data into an flattened tensor buffer.
`javascript
/**
const rChannelOffset = 0;
const gChannelOffset = totalPixels;
const bChannelOffset = totalPixels * 2;
for (let i = 0; i < totalPixels; i++) {
const srcIndex = i * 4;
nchwBuffer[rChannelOffset + i] = data[srcIndex] / 255.0;
nchwBuffer[gChannelOffset + i] = data[srcIndex + 1] / 255.0;
nchwBuffer[bChannelOffset + i] = data[srcIndex + 2] / 255.0;
}
return nchwBuffer;
}
`
The most terrible scenario when running deep learning on the frontend is the UI freezing. For the browser to display smooth animations at 60 frames per second, the event loop must process synchronous tasks within 16.6ms. However, tensor operations easily block the single-threaded main renderer loop.
LiteRT.js is about 3 times faster in base execution speed than existing JavaScript-based tools. When adding WebGPU or WebNN acceleration hardware, it becomes 5 to 60 times faster than CPU mode. The WebNN backend, which uses a dedicated NPU, requires JavaScript Promise Integration (JSPI) to be enabled to connect the synchronous WebAssembly kernel scheduler with the browser's asynchronous hardware control loop. To utilize these acceleration resources while keeping the main thread alive, you should isolate the entire library initialization and inference pipeline within a Web Worker.
If you simply pass buffers during inter-thread data communication, internal memory copying occurs, piling up overhead in the CPU and heap memory. You must use Transferable Objects to pass the ownership of the physical memory address region itself to eliminate latency. Since the transferred buffer is immediately invalidated in the sender's context, you can also maintain thread safety.
`javascript
// litert-worker.js - Web Worker module dedicated to background inference operations
import { loadLiteRt, loadAndCompile, Tensor } from '@litertjs/core';
let compiledModel = null;
let isLoaded = false;
self.onmessage = async (event) => {
const { type, payload } = event.data;
switch (type) {
case 'LOAD_MODEL':
try {
await loadLiteRt(payload.wasmDirectory, { jspi: payload.enableJspi || false });
compiledModel = await loadAndCompile(payload.modelUrl, {
accelerator: payload.accelerator || 'webgpu'
});
isLoaded = true;
self.postMessage({ type: 'MODEL_READY' });
} catch (err) {
self.postMessage({ type: 'ERROR', error: Initialization failed: ${err.message} });
}
break;
case 'RUN_INFERENCE':
if (!isLoaded || !compiledModel) {
self.postMessage({ type: 'ERROR', error: 'Model has not been loaded' });
return;
}
try {
const rawInputData = payload.bufferData;
const inputShape = payload.shape;
const inputTensor = new Tensor(rawInputData, inputShape);
const results = await compiledModel.run(inputTensor);
const cpuOutputTensor = await results[0].moveTo('wasm');
const outputBuffer = cpuOutputTensor.toTypedArray();
inputTensor.delete();
cpuOutputTensor.delete();
results[0].delete();
self.postMessage(
{
type: 'INFERENCE_COMPLETE',
payload: {
data: outputBuffer,
shape: results[0].shape
}
},
[outputBuffer.buffer]
);
} catch (err) {
self.postMessage({ type: 'ERROR', error: `Inference failed: ${err.message}` });
}
break;
default:
self.postMessage({ type: 'UNKNOWN_OP' });
}
};
`
`javascript
// litert-bridge.js - AI orchestrator class for the main thread
export class LiteRtBridge {
constructor(workerPath) {
this.worker = new Worker(workerPath);
this.promiseMap = new Map();
this.tokenCounter = 0;
this.worker.onmessage = (event) => {
const { type, payload, error } = event.data;
if (type === 'MODEL_READY') {
if (this.initResolve) this.initResolve();
} else if (type === 'INFERENCE_COMPLETE') {
const currentToken = this.tokenCounter;
const promiseHandler = this.promiseMap.get(currentToken);
if (promiseHandler) {
promiseHandler.resolve(payload);
this.promiseMap.delete(currentToken);
}
} else if (type === 'ERROR') {
const currentToken = this.tokenCounter;
const promiseHandler = this.promiseMap.get(currentToken);
if (promiseHandler) {
promiseHandler.reject(new Error(error));
this.promiseMap.delete(currentToken);
} else if (this.initReject) {
this.initReject(new Error(error));
}
}
};
}
bootstrap(wasmDirectory, modelUrl, accelerator = 'webgpu') {
return new Promise((resolve, reject) => {
this.initResolve = resolve;
this.initReject = reject;
this.worker.postMessage({
type: 'LOAD_MODEL',
payload: { wasmDirectory, modelUrl, accelerator, enableJspi: true }
});
});
}
execute(inputFloat32Array, inputShape) {
return new Promise((resolve, reject) => {
this.tokenCounter++;
this.promiseMap.set(this.tokenCounter, { resolve, reject });
this.worker.postMessage(
{
type: 'RUN_INFERENCE',
payload: {
bufferData: inputFloat32Array,
shape: inputShape
}
},
[inputFloat32Array.buffer]
);
});
}
}
`
When using TF.js, the pattern of using tf.tidy() to clean up tensors within a synchronous call scope was standard. However, when asynchronous code or promises were involved, it often led to bugs where tensors were destroyed or collection was skipped before the async task even finished.
LiteRT.js is even more ruthless. It is not subject to the browser engine's garbage collection (GC). JavaScript engines like V8 cannot track the heap state of WebAssembly's linear virtual memory space and WebGPU buffers. If you do not explicitly call .delete() on tensor instances when you are finished with them, the browser's memory will grow infinitely. If you are running a service that streams high-definition video frames dozens of times per second, the tab will crash within minutes.
You should feel more at ease if you manually manage the lifecycle by creating a scope tracker class that logs the lifetimes of tensors created across the asynchronous pipeline and guarantees batch destruction.
`javascript
/**
/**
/**
/**
`
`javascript
// Example of implementing safe and robust multi-async AI inference processing using a memory scope tracker
export async function runRobustVisionInference(rawPixelArray, compiledModel) {
const scopeTracker = new LiteRtScopeTracker();
try {
return await scopeTracker.enforceScope(async (scope) => {
const inputTensor = scope.register(new Tensor(rawPixelArray, [1, 3, 224, 224]));
const predictionResults = await compiledModel.run(inputTensor);
predictionResults.forEach((tensor) => scope.register(tensor));
const firstOutputTensor = predictionResults[0];
const wasmTransferTensor = scope.register(await firstOutputTensor.moveTo('wasm'));
const targetJsArray = wasmTransferTensor.toTypedArray();
return targetJsArray;
});
} catch (err) {
console.error('Fatal crash occurred during the model pipeline execution:', err);
throw err;
}
}
`
You also cannot ignore the issue of increasing resource download sizes. To ensure tree-shaking works in your build bundler, you should remove CommonJS-style static references and write your source code based on ES6 module syntax (import/export). You also need to keep an eye on the sideEffects: false setting in your build tool to ensure the bundle remains lightweight.
The LiteRT.js core runtime, @litertjs/core, conditionally loads one of three WebAssembly kernel builds based on device performance. For modern browsers like Chrome or Edge, it selects a module supporting multi-threading and SIMD (litert_wasm_simd.wasm), while for legacy environments like Safari, it imports the default fallback module (litert_wasm.wasm). If GPU compilation fails, the XNNPACK runtime acts as a backup, pushing the entire set of hardware operators into the CPU Wasm sandbox.
To prevent initial loading delays, you must check the accelerator according to the device specifications and import the module dynamically.
`javascript
// litert-loader.js - Runtime device detection and dynamic accelerator integration engine
export async function bootstrapHighPerformanceInferenceEngine() {
const supportsWebGpu = 'gpu' in navigator;
let chosenAccelerator = 'wasm';
if (supportsWebGpu) {
try {
const gpuAdapter = await navigator.gpu.requestAdapter();
if (gpuAdapter) {
const info = await gpuAdapter.requestDevice();
if (info) {
chosenAccelerator = 'webgpu';
}
}
} catch (e) {
console.warn("GPU profile probe failed, resolving execution chain to fallback WASM.");
}
}
const { loadLiteRt, loadAndCompile } = await import('@litertjs/core');
const cdnWasmHostPath = 'https://cdn.jsdelivr.net/npm/@litertjs/core/wasm/';
await loadLiteRt(cdnWasmHostPath, {
jspi: chosenAccelerator === 'webnn'
});
return {
loadAndCompile,
chosenAccelerator
};
}
`
Make the ownership transfer of memory via Web Workers and the explicit destruction of objects in the Wasm region the fundamental basis of your design. Once you start directly controlling the data flow, you can deploy on-device AI services to production without worrying about browser crashes.