How I Reduced Our Startup's LLM Costs by Almost 90%

[chardionne1226]

@redeemed2000 Amazing read! Working with GPT nowadays in my daily job we encounter a lot of the challenges you’ve mentioned. Thank you for sharing!

 

[claireu1111]

@redeemed2000 Wow, that's some savvy cost-cutting! Been there, done that.

 

[luke2017]

@redeemed2000 Very insightful. Thanks for sharing.

 

[jaliha]

@redeemed2000 Thanks, this is insightful!

To clarify my understanding: you recorded 50000 calls made by real users, fine-tuned Mistral on those inputs and outputs, and then did an A/B test to compare its performance to GPT - is that correct?

 

[redeemed2000]

@jaliha pretty much yeah!

 

[discipleofchrist1]

@redeemed2000 I love this idea and will definitely be using it

 

[kenm]

@redeemed2000 Was cost the main reason that you decided to use an open source LLM as opposed to doing fine tuning on OpenAI? Do you remember how much that difference was?

 

[redeemed2000]

@kenm Both for cost and performance. Way cheaper and just better performing. Also gives me the flexibility to not be locked into OpenAI

 

[anagrace]

@redeemed2000 Hi @redeemed2000 I really enjoyed your post. I would like to learn how to do this stuff. I have a fairly good programming background. What resource do you recommend for someone like me to get up to speed with building an app that uses an open source model like you did? Thank you.

 

[redeemed2000]

@anagrace Feel free to reach out on twitter - glad to chat more piersonmarks

 

[nowhereman]

@redeemed2000 This is exactly why the open model is going to win out. There are too many competing models it's clear that they're all racing in the same direction which makes it a commodity.

 

[cerri]

@redeemed2000 Thanks for the info... Didn't even know something for this existed... But I should know there is always an app for that!

 

[redeemed2000]

@cerri There's always an app for it! Except there wasn't for Jellypod... so I built it lol

 

[stmitche74]

@redeemed2000 Great case study. Cost concerns related to LLMops is going to grow over time amongst companies.

Did you experience significant deviations in model accuracy?

 

[redeemed2000]

@stmitche74 100%, and no it performed equally well. The usage of the model at each step was very narrow and I had response validation with regex (where possible). So essentially there are no failure because if the model responds with something unexpected, I retry.

For the actual summarization task you can't really test except with evals, but that's okay. I use customer feedback in the app to ensure it's summaries are accurate and high quality

 

[stmitche74]

@redeemed2000 Excellent, I appreciate your perspective.

 

[keeperoftheflame]

@redeemed2000 This is great! I checked out your website. It looks like you've made amazing progress! Is the IOS App only available in limited countries?

 

[redeemed2000]

@keeperoftheflame Thanks! Some big ndw features being released this week.

Yes - right now the iOS app is only in a few countries so far

 

[keeperoftheflame]

@redeemed2000 That's a bummer! I signed up for your newsletter. Looking forward to when you expand Jellypod worldwide.

The information you shared was really helpful by the way! Had a meeting with our dev team today and we're looking into this now for for our platform.

 

[fav54con]

@redeemed2000 My question is (its a stupid question but i really need to know that) where can i get the data to fine tune my model??