I built a platform to make it easier to experiment and test different LLMs and figure out what works for your users and specific use case

[jamisonbirdsong]

I've been building a platform to make managing and optimizing your LLM applications more streamlined: https://optimix.app/. We make it easy to automatically route your API requests to the best LLM for your task and preferences, and provide useful analytics on how your LLM's outputs are performing in real-time for your users and specific use case.

Here are some of the main features:

• Automatic, context and data-driven LLM switching.
• Playground to test and compare prompts and models (including new models like GPT-4o, Gemini 1.5 Flash, and Llama 3).
• A/B test prompt or model changes to see if they are helpful to the user, and backtest on historical data for safe experimentation.
• Metrics on latency, cost, error recovery, user satisfaction, and more.

I'd love any feedback, thoughts, and suggestions. Hope this can be a helpful tool for anyone building AI products!