Forge

A second major feature group is the profound performance improvements Forge delivers, such as dramatically increased inference speed and throughput. For instance, when optimizing the Qwen3 235B mixture-of-experts model, Forge cut the time to first token from 320 milliseconds down to just 42 milliseconds, a 7.6x reduction in token generation latency that enables real-time responses at scale. Furthermore, throughput can increase by an order of magnitude, as demonstrated by a jump from 312 tokens per second to 3,180 tokens per second on the same hardware, allowing services to handle ten times more concurrent users without procuring additional GPUs. These speed enhancements are critical for applications requiring low-latency interactions, such as chatbots, real-time translation, and interactive AI assistants, where user experience is directly tied to response time.

Additional capabilities include significant cost reduction and power efficiency gains achieved by maximizing GPU utilization from low levels like 24% up to 95%. The content highlights a 90% cost reduction per million tokens, from $4.30 down to $0.43, by ensuring that nearly every GPU cycle is used for productive computation rather than being idle or stalled. This also leads to a 67% reduction in power consumption, as optimized kernels complete work faster and allow hardware to enter lower power states more frequently, contributing to both operational savings and environmental sustainability goals. Forge also provides enterprise-grade infrastructure options, including on-premise deployment, dedicated GPU clusters, and custom hardware support, ensuring that organizations with strict data sovereignty or security requirements can still benefit from its optimizations.

Technically, Forge works by taking an existing AI model—from frameworks like PyTorch or repositories like HuggingFace—and processing it through its automated optimization pipeline without requiring any modifications to the user's source code. The system analyzes the model's operations and data flow, then generates and compiles optimized GPU kernels using technologies like CUDA and Triton, which are specifically tuned for the target NVIDIA GPU architecture, such as B200, H200, H100, L40S, or A100. This approach ensures that the optimized model is a drop-in replacement, maintaining the exact same API and numerical correctness while executing far more efficiently on the hardware. The entire process is managed via a command-line interface, making it integrable into existing CI/CD pipelines and MLOps workflows.

The benefits and measurable outcomes for users are substantial and directly quantifiable, including saving thousands of dollars on monthly GPU costs, with an example citing $18,000 saved per month. Users achieve up to 5x faster inference speeds compared to using torch.compile with max_autotune, alongside a 97.6% correctness guarantee. The optimization leads to higher throughput, enabling the same hardware to serve more requests, and reduces power consumption by 67%, lowering both the financial and environmental footprint of AI operations. These improvements translate into better scalability for applications, reduced infrastructure spending, and the ability to deploy more complex models or handle larger user bases without proportional increases in compute budget.

Concrete use cases include optimizing large language models like the 235B parameter Qwen3 for real-time inference, where Forge drastically reduces latency for applications like interactive chatbots or coding assistants. Another example is enhancing vision models such as YOLO for object detection, or speech models for Arabic text-to-speech and speech-to-text, ensuring they run efficiently on dedicated GPU servers. Workflow examples involve ML engineers simply running the Forge CLI on their model checkpoint, receiving an optimized version within an hour, and deploying it to their production environment with no code changes, immediately observing higher throughput and lower latency during inference serving. Enterprises can integrate Forge into their model deployment pipeline to continuously optimize new model versions before they go live.

The target users are ML teams, infrastructure engineers, and enterprises that deploy AI models at scale and need to maximize GPU performance. Integrations are seamless with PyTorch and HuggingFace model formats, and the tech stack leverages CUDA and Triton for kernel generation on NVIDIA datacenter GPUs. Pricing is custom and tailored to enterprise needs, with options for a free demo to optimize one model without a credit card, and enterprise plans that include dedicated infrastructure, on-premise deployment, custom SLA and support, and NDA and IP protection. Support is provided by a dedicated team, ensuring users receive assistance for deployment and optimization challenges.

In summary, Forge provides a critical solution to the pervasive problem of GPU underutilization in AI inference, automatically generating optimized kernels that deliver dramatically faster speeds, higher throughput, and major cost savings. By offering a drop-in replacement that requires no code changes and guarantees numerical correctness, it enables teams to instantly improve their model's performance on existing hardware. The primary value proposition is maximizing return on GPU investments through automated, reliable optimization that transforms inefficient compute into highly productive work, making advanced AI deployment more scalable and affordable for organizations of all sizes.

Updated 2026-02-28