Cloudflare recently announced that AI Gateway is now generally available. Described as a unified interface for managing and scaling generative AI workloads, AI Gateway allows developers to gain visibility and control over AI applications.
AI Gateway is an AI ops platform that offers a unified interface for managing and scaling creative AI workloads. It acts as a proxy between services and estimation providers, regardless of where the models run.
Source: Cloudflare Blog
Kathy Liao, Product Manager at Cloudflare, Michelle Chen, Senior Product Manager at Cloudflare, and Phil Wittig, Director of Product at Cloudflare, write:
We've spoken to many developers and organizations building AI applications, and one thing is clear: they want more visibility, control, and tooling around their AI operations. This is something that many AI providers lack because they focus heavily on model development and less on platform features.
Connecting an application to an AI gateway enables it to monitor user interactions with analytics and logging, and provides scaling features such as caching, rate limiting, request retries and model fallback. Add Liao, Chen, and Witig:
With a single line of code, you can unlock a set of powerful features focused on performance, security, reliability, and observability – think of it as your control plane for your AI operations.
In addition to Cloudflare Workers AI, the new AI Gateway supports a number of third-party providers, including OpenAI, Google Vertex AI, Azure OpenAI, HuggingFace, Amazon Bedrock, and Anthropic. Amog Sarda, co-founder of Ezel, comments:
I'm interested in seeing it in action. I'm sure there will be some fun ways to test its sensitive data detection capabilities.
The AI Gateway dashboard displays metrics such as number of requests, tokens, and cost associated with running the request. It also tracks individual requests, providing information about the prompt, response, provider, timestamp, and whether the request was successful.
Source: Cloudflare Blog
AI Gateway is not the latest announcement from Cloudflare in the AI space. The company also previewed Firewall for AI and made Worker AI generally available, along with several other capabilities aimed at how developers build and deploy AI applications. Janakiram MSV, analyst and consultant writes:
Cloudflare is challenging Amazon Web Services (AWS) by continuously improving its edge network capabilities. Amazon's serverless platform, AWS Lambda, has yet to support GPU-based model inference, while its load balancers and API gateway have not been updated for AI inference endpoints.
Credexim founder Brendan Skousen commented:
The latest AI tools I'm building include Cloudflare's services. I would replace platform-specific API endpoints with Cloudflare's, whether it's my own API through a worker or using something like AI Gateway or Web3 Gateway. Why? Because it's free after all, and includes features like secure analytics out of the box. Real-time logging, caching, and rate limiting are important when building LLM apps.
AI Gateway's current core features are free on all Cloudflare plans, but future premium features, such as continuous logging and secret management, will be available for a fee.