Suggestions
Andleep Farooqui
Machine Learning Engineer
Andleep Farooqui is a Software Engineer currently working at Axyom, a company focused on Web3 and AI technologies, since April 2023. Prior to this role, he served as a Software Engineer at TRC Healthcare starting in July 2021, and he also completed an internship as a Software Engineer at SymSoft Solutions.1
His professional background highlights a progression in software engineering roles, showcasing his involvement in innovative fields such as artificial intelligence and blockchain technology within the Web3 domain.
Highlights
RLHF (reinforcement learning from human feedback) and DPO (direct preference optimization) are great techniques to add on top of supervised finetuning to help align LLM generation with the sorts of generations that humans would want.
That said.. Both RLHF and DPO use a model of human preferences called the “Bradley-Terry” model. It has a lot of positives but it’s pretty old (orginated in 1952) and there have been subsequent decades of research in computational cognitive science.
Somewhat recently, cogsci researchers have introduced a new paradigm - KTO. The primary difference between KTO and DPO is the form of the human value function - KTO is informed by the work of Kahneman and Tversky in 1992 and uses an experimentally determined power law distribution - with specific parameters gotten by experiment!
In experiments, researchers found that KTO performs as well or better than DPO at all scales 😮 best of all, we do not need preference data which opens up a lot of possibilities..
The Archangel repository contains a number of KTO and DPO finetuned models to try and i’m wondering if anyone has played with this technique and what they think.
Posting so more people hear about this and know! It really is a cool development.
Contextual AI put out a blogpost here with all the links you’d need: https://lnkd.in/e-_wKrTD
Lmk if you have tried it or what you think of the archangel models, I haven’t had much experience with dpo myself yet (hoping to change that soon 🙂)
A huge piece of value for GPT4 is to use it as a tool to build miniature functions - specifically in terms of text processing. Named entity extraction, splitting text into pieces, etc.
It feels really underrated. I keep building out minor LLM apps to just deal with everyday stuff.
