How’re you guys doing after GPT-4O and Google I/O

[chrisroary]

It has solved for a lot of usecases that a bunch of startups were aiming to solve.

I’m trying to ask a deeper question- Are you still looking to build better foundational models? How does that look like?

What do these updates mean for your companies?

 

[solokwa]

@chrisroary I’ve been told by partners at Sequoia, First Round, and the VCs alike are not looking to invest in any companies working on foundational models. The large players in the game have risen and they aren’t looking to invest money in an attempt to compete with them

 

[dharmmy]

@solokwa How are any startups training foundational models anyway? The cost is surely prohibitive.

I read somewhere that to train Llama 2, it cost Meta $20m of compute/electricity. And that's with them having their own hardware. If they'd had to rent from GCP or AWS it'd have been even more. And that's one version of the model - obviously the implication is you iterate many times.

Seems to me like all startups can hope to do (at best) is fine tune base models.

 

[pwc1970]

@dharmmy Llama is freaking huge investment though there’s probably a place somewhere for more mid models, stable diffusion 1.5 relatively speaking wasn’t that bad cost wise, and if stability had actually known what they were doing, they could have done a lot better.

 

[soyeong]

@pwc1970 Why would you make a "mid" model if Llama is available?