Sign In

Jason Kelly

Chief Executive Officer at Fremont Robotics

Constantly amazed at how stupid I was two weeks ago.

Highlights

Jun 28 · linkedin

What to know which tasks LLMs like Claude, Gemini, ChatGPT are going to get better at? Look at the popular benchmarks.

Text-based: MMLU: Knowledge across 57 subjects across STEM, humanities, social sciences, and more BIG-Bench-Hard: Extra-challenging logic and reasoning tasks

The latest models are >90% accurate on these and are fighting for single percentage points.

Vision-based benchmarks are showing really interesting progress: MathVista: Visual math reasoning MMMU: College-level knowledge and reasoning on images

Newer models (Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4o) are below 70% on these vision benchmarks but I'd bet on seeing substantial progress quickly.

This is a good overview image of the MMMU evaluation dataset (from https://lnkd.in/gUkbBFcC)

Jun 4 · linkedin

Researchers at UC Berkeley, Stanford, CMU and Google just released this open-source model for robotic manipulation. The exciting part? All of the code, models, and training data are available on GitHub and HuggingFace. I'm going to try it out now!

Related Questions

What is Jason Kelly's current role?
Which universities did he attend and what did he study?
What prior engineering and technology leadership roles has he held?
Which companies in cloud, aerospace, and venture sectors has he worked for?
Jason Kelly
Add to my network

Experience

Chief Executive Officer at Fremont Robotics
Vice President Of Software Engineering at Ageagle Aerial Systems

Education

Information Systems, University Of Washington
Bachelor Of Science In Communication, Computer Science, Journalism, Iowa State University

Location

Seattle, Washington, United States