Suggestions
Agostino Calamia
Data Scientist at N26 - - M.Sc. Big Data & Business Analytics
Agostino Calamia is a data science professional with experience at N26 and Shopify, specializing in machine learning lifecycle management and MLOps practices. Here's a breakdown of his career and contributions:
Role at N26
- Position: Junior to Middle-Level Data Scientist (2019–2022)
- Key Projects:
- Developed credit risk models and installment product analytics for N26’s banking operations.13
- Focused on MLOps practices to ensure data quality and model reliability in regulated environments.1
- Collaborated cross-functionally with teams like customer service and product development to implement data-driven solutions.3
Career Transition and Expertise
- Shift to Shopify: Moved to Shopify as a Senior Data Scientist, working on sales and post-sales analytics, including merchant acquisition and product recommendations.1
- Technical Skills: Proficient in Python, SQL, TensorFlow, PyTorch, and tools like BigQuery and Apache Kafka.1
- Community Contributions: Authored The Playbook for Machine Learning Lifecycles to streamline ML deployment and authored free courses on AI/ML strategies.12
Current Work and Advocacy
- Ethical AI and MLOps: Actively advocates for ethical AI practices and robust MLOps frameworks.1
- HP Data Science Ambassador: Promotes data science education and best practices.1
His work at N26 laid the foundation for his expertise in balancing regulatory compliance with innovative ML solutions, later expanded at Shopify.13
Highlights
I found my ultimate RAG cheatsheet 🚀
(save it, so you can use it too)
I've been following Eduardo Ordax for a while now.
Last week I came across his RAG cheatsheet.
And it is all I was looking for.
It captures every detail of RAGs
Initially, RAGs seem simple. Good performing ones are difficult to build though.
The cheatsheet covers:
- RAG Foundations
- Chunking Strategies
- Vector DBs: Milvus, Chroma, etc.
- Frameworks: LlamaIndex, LangChain, etc.
- Advanced Retrieval Techniques
- Generation
- RAG Evaluation & Frameworks
Check it out. You don't need it daily. But when you do, it helps a lot.
#LLM #Data #Analytics #MachineLearning #ArtificialIntelligence
This is how I validate LLMs in 4 steps
(using an open-source tool)
Building an LLM application is easy. However, creating a good one is challenging.
A great LLM application needs testing.
Here are 3 key things to test:
- Which prompt performs best?
- Which LLM should I use?
- Is the output what I expect?
Promptfoo is the simplest way to achieve this. Plus, it is open source.
Let me give you an example:
We'll compare the performance of GPT-4o and Claude 3.5.
Here’s what I do:
-
Set Up Your Evaluation:
- Define your models in a "promptfooconfig.yaml" file.
- Set parameters like temperature, etc.
- Define your prompts (e.g., "Solve the riddle:").
- Prompts can be passed as strings or directories.
-
Create Test Cases:
- Identify relevant tests your LLM needs to solve.
- Pass your test content (e.g., a riddle) and the expected outcome.
- Promptfoo lets you check if the answer contains specific logic or words.
- All of this is done in a YAML file.
-
Run the Evaluation:
- Execute the evaluation from the terminal.
- View results in a user-friendly UI.
- Save the results to a file (e.g., JSON).
-
Analyze the Results:
- Understand where the models differ semantically.
- Examine latency, tokens, and costs.
- Make a decision based on your use case.
ADD-ON: Promptfoo also allows you to set assertions with thresholds. This can make a model fail if it exceeds a specified latency.
In my opinion, this is important for every team.
Links in the comments!
P.S.: Do you know of a better tool than Promptfoo?
#LLM #Data #Analytics #Performance #ArtificialIntelligence
