Suggestions
Kirat Pandya
Co-Founder & CEO @ Osmos
Kirat Pandya is the Co-Founder and CEO of Osmos, a company focused on building an external data platform that helps organizations manage and scale their data relationships. Prior to founding Osmos, he served as the Machine Learning Technical Lead at Google, where he was involved in driving strategic partnerships related to Google Cloud Platform's machine learning initiatives.12
Kirat holds a degree from the University of Illinois Urbana-Champaign and has a background that includes experience at Microsoft, further enhancing his expertise in technology and data management.34 At Osmos, he emphasizes the importance of effectively handling external data, addressing challenges companies face in integrating diverse data solutions.2
Highlights
Microsoft's latest paper "Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers" is a very real measure of training vs inference FLOPs trade offs.
Llama 3.1 8B performs at (ballpark) GPT-4o performance if you’re willing to spend 100x the tokens on the same reasoning/math problem.
In other words, let the AI prompt engineer itself to success!
What’s really interesting with this kind of approach is that startups can trade off spending more on burstty batch inference jobs as a trade off for requiring lots of idle TPUs/GPUs just to host huge models.
Optimizing your business in this accelerator-based future will require way more understanding of the stack than just “set the Kubernetes Horizontal Pod Autoscaler to auto scale at 80% utilization” that we’ve been doing in the CPU era.
Screenshot is tokens and inference calls to solve the GSM8K dataset.
Eating soup with a fork.
Spend enough time making traditional code interface with an LLM and that's what it all feels like.
We don't have tools to easily cross the (structured, deterministic, testable) <-> (anything, anything, shrug) interface.
JSON mode and grammars are not enough because they limit you in all sorts of ways (eg: Chain of thought becomes way more complicated and you need a multi-turn chat trained model to do it.).
In fact, every bit of tooling I have seen that tries to help here is all built around muzzling the LLM in some form (Eg: Microsoft Guidance, Semantic Kernel etc.)
I expect that LLMs will develop faster than our ability to interface with them programmatically. It's going to be easier to let the LLM commandeer our programming tools than it is for us to plug in LLMs to specific integration points in our code.
Today:
- user clicks something in the app
- App pulls together a prompt (RAG, other context etc.) and calls an LLM
- try to make use of its response via JSON etc.
Pretty soon:
- User clicks something in the app*
- LLM is sent the activity
- LLM queries the system for any state and information it needs, and then interacts with endpoints to mutate state as needed.**
This is not just going to change how we write software, but literally everything around it.
In this new world, how would you:
- Design an interface? (UI and UX)
- Demo an interface to customers? Does Figma even make sense anymore?
- Build the app? Does it even make sense to have developer (with an LLM copilot) write code? If the LLM can write code, can't it just do the job directly? ... ... ...
*really though, it should be "App has already acted in anticipation of user activity"
** LLM, LMM etc. -> insert name for next week's large matrix multiplication model.
