How I Reduced Our Startup's LLM Costs by Almost 90%

[williamjo10]

@redeemed2000 If you’re open to sharing your twitter profile I’d follow you.

 

[redeemed2000]

@williamjo10 Totally - since my name is already on my website, it's piersonmarks

 

[kinggeorge]

@redeemed2000 I faced similar issues; OpenPipe's a game-changer indeed.

 

[loveyallb]

@redeemed2000 ! Thank you

 

[ambasea]

@redeemed2000 Thanks for sharing!!

 

[mundy]

@redeemed2000 Banger of a post! Thanks for the insight.

 

[moriarty777]

@redeemed2000 Microsoft for startup is giving $5000 credits for startup, you could check it out.

 

[strings51415]

@moriarty777 it goes upto 125k

 

[redeemed2000]

@moriarty777 This is super helpful - thanks!

 

[gfalaro]

@moriarty777 this is for gpt 3.5 right? not for gpt 4 unless they changed it?

 

[asnyder]

@redeemed2000 How does the pricing work? Are you only paying for the API calls or do you also pay for deployment of your finetuned model (i.e. can you scale to zero)?

 

[redeemed2000]

@asnyder Yeah just inference + training costs. No hosting costs

 

[asnyder]

@redeemed2000 I take it they don't keep your endpoints warm all the time then. Do you have any insights in the cold start times? You likely don't care about cold start times since I figure you do batch inference once a day, but i am trying to find out how viable this is for real time inference (lets say, is sub 5 seconds latency (startup+inference) for 1k tokens achievable?).

 

[redeemed2000]

@asnyder Hmm I'm not sure. Once warm its super fast but I don't know what the cold start time is. It's not advertised (and I haven't measured)

 

[faaraah]

@redeemed2000 Yes, I always wondered about the running costs and OpenAI bills. Seems that you made the right decision in only sending API requests when necessary! Great job. I'd love to read more about that side of "AI startups"

 

[redeemed2000]

@faaraah I'm on Twitter!

 

[faaraah]

@redeemed2000 What’s your handle?

 

[isaac32]

@redeemed2000 Thanks for sharing. Was it expensive to fine-tune on OpenPipe?

 

[redeemed2000]

@isaac32 No - cost about $100

 

[jbron423]

@redeemed2000 Nice advice