How SiteGPT built an enterprise-grade AI support platform on Portkey

How SiteGPT built an enterprise-grade AI support platform

From a weekend project to serving enterprise customers at scale, SiteGPT grew into a robust customer support platform handling billions of tokens across multiple LLMs, all while maintaining uptime, reliability, and observability with Portkey.

About
SiteGPT is a personalized chatbot that answers your visitors’ questions using a model trained specifically on your website content.

Industry
AI-powered Customer Support

Company Size
11-50 employees

Headquarters
India

Founded
2023

Why Portkey:

Multi-provider model switching, enterprise-grade observability, and failover support

4M+

Requests Processed

10+

Models In Prod

6B+

Tokens Processed

Overcoming infrastructure complexities for enterprise AI

SiteGPT started in early 2023 with a simple goal: to bring generative AI to customer support. Within months, the platform evolved into a full-fledged solution used by thousands of businesses, from startups to large enterprises.

As their customer base expanded, the engineering team faced increasing infrastructure complexity. They needed to support multiple LLMs across providers, monitor usage at a granular level, and meet strict performance and compliance standards.

“When you're handling millions of requests, even a 0.1% failure rate is unacceptable. We needed infrastructure that could scale with our enterprise customers while maintaining perfect reliability.”

— Bhanu Teja Pachipulusu, Founder, SiteGPT

Scale your AI apps, without the risk of failovers

Building reliable infrastructure for AI support at scale

To maintain performance at scale, SiteGPT turned to Portkey as their centralized AI infrastructure layer.

Key reasons why Portkey stood out:

• Seamless switching between 13+ models in production
• Built-in rate limit handling for each provider
• Enterprise-grade logging and observability
• Metadata tagging for easy segmentation and debugging
• End-to-end performance tracking and failover support

Portkey gave SiteGPT the control, visibility, and reliability required to serve enterprise-grade SLAs, while continuing to experiment with new models.

Managing complex operations with complete visibility

Portkey handles all LLM interactions across SiteGPT’s platform. Every call is logged with detailed metadata, token usage, and response data — helping the team debug, optimize, and iterate fast.

With automated failover and real-time performance tracking, SiteGPT can route requests dynamically and maintain service even when individual providers experience latency or rate limits.

That’s where Portkey plays a central role.

• Prompt management: 100+ prompts are versioned in Portkey’s library. Changing tone or format for a region is as simple as updating one prompt, no need to alter n8n logic.
• Full observability: Every call is logged with latency, token usage, model metadata, and output quality. The team can track underperforming calls and compare across providers.
• Provider flexibility: With Portkey’s forward-compatible architecture, switching or trying out a new model takes minutes, no flow changes needed.

The impact: more reliability, less overhead

Portkey's logging capabilities have been invaluable for debugging complex tool calls and understanding user patterns. The metadata features allow us to quickly segment and analyze user behavior in ways we couldn't before.

— Bhanu Teja Pachipulusu, Founder, SiteGPT

The results have validated SiteGPT's decision to build on Portkey:

• Successfully processed 6 billion tokens with high reliability
• Achieved 5% reduction in support tickets for enterprise clients
• Maintained enterprise-grade uptime across all deployments
• Enabled complex workflow automation capabilities

SiteGPT continues to expand its AI capabilities with Portkey, focusing on testing new models while maintaining its high standards for reliability and performance. Their roadmap includes enhanced guardrails, improved CSAT collection, and deeper support workflow integration.

Lessons for AI teams scaling their operations

For teams building AI-powered customer support, SiteGPT's experience offers valuable insights:

• Start with robust infrastructure - SiteGPT's ability to scale rapidly while maintaining reliability was crucial for enterprise adoption
• Prioritize observability - Comprehensive logging and metadata management proved essential for debugging and optimization
• Plan for scale - The ability to support multiple models and handle billions of requests requires sophisticated infrastructure from day one
• Focus on reliability - Enterprise customers require perfect uptime, making automated failover and rate limit handling essential

SiteGPT’s journey proves that with the right infrastructure, AI products can scale from idea to enterprise without rewriting the backend.