Sign In

Agostino Calamia

Data Scientist at N26 - - M.Sc. Big Data & Business Analytics

Agostino Calamia is a data science professional with experience at N26 and Shopify, specializing in machine learning lifecycle management and MLOps practices. Here's a breakdown of his career and contributions:

Role at N26

  • Position: Junior to Middle-Level Data Scientist (2019–2022)
  • Key Projects:
    • Developed credit risk models and installment product analytics for N26’s banking operations.13
    • Focused on MLOps practices to ensure data quality and model reliability in regulated environments.1
    • Collaborated cross-functionally with teams like customer service and product development to implement data-driven solutions.3

Career Transition and Expertise

  • Shift to Shopify: Moved to Shopify as a Senior Data Scientist, working on sales and post-sales analytics, including merchant acquisition and product recommendations.1
  • Technical Skills: Proficient in Python, SQL, TensorFlow, PyTorch, and tools like BigQuery and Apache Kafka.1
  • Community Contributions: Authored The Playbook for Machine Learning Lifecycles to streamline ML deployment and authored free courses on AI/ML strategies.12

Current Work and Advocacy

  • Ethical AI and MLOps: Actively advocates for ethical AI practices and robust MLOps frameworks.1
  • HP Data Science Ambassador: Promotes data science education and best practices.1

His work at N26 laid the foundation for his expertise in balancing regulatory compliance with innovative ML solutions, later expanded at Shopify.13

Highlights

Aug 12 · linkedin

I found my ultimate RAG cheatsheet 🚀

(save it, so you can use it too)

I've been following Eduardo Ordax for a while now.

Last week I came across his RAG cheatsheet.

And it is all I was looking for.

It captures every detail of RAGs

Initially, RAGs seem simple. Good performing ones are difficult to build though.

The cheatsheet covers:

  • RAG Foundations
  • Chunking Strategies
  • Vector DBs: Milvus, Chroma, etc.
  • Frameworks: LlamaIndex, LangChain, etc.
  • Advanced Retrieval Techniques
  • Generation
  • RAG Evaluation & Frameworks

Check it out. You don't need it daily. But when you do, it helps a lot.

#LLM #Data #Analytics #MachineLearning #ArtificialIntelligence

Jul 29 · linkedin

This is how I validate LLMs in 4 steps

(using an open-source tool)

Building an LLM application is easy. However, creating a good one is challenging.

A great LLM application needs testing.

Here are 3 key things to test:

  1. Which prompt performs best?
  2. Which LLM should I use?
  3. Is the output what I expect?

Promptfoo is the simplest way to achieve this. Plus, it is open source.

Let me give you an example:

We'll compare the performance of GPT-4o and Claude 3.5.

Here’s what I do:

  1. Set Up Your Evaluation:

    • Define your models in a "promptfooconfig.yaml" file.
    • Set parameters like temperature, etc.
    • Define your prompts (e.g., "Solve the riddle:").
    • Prompts can be passed as strings or directories.
  2. Create Test Cases:

    • Identify relevant tests your LLM needs to solve.
    • Pass your test content (e.g., a riddle) and the expected outcome.
    • Promptfoo lets you check if the answer contains specific logic or words.
    • All of this is done in a YAML file.
  3. Run the Evaluation:

    • Execute the evaluation from the terminal.
    • View results in a user-friendly UI.
    • Save the results to a file (e.g., JSON).
  4. Analyze the Results:

    • Understand where the models differ semantically.
    • Examine latency, tokens, and costs.
    • Make a decision based on your use case.

ADD-ON: Promptfoo also allows you to set assertions with thresholds. This can make a model fail if it exceeds a specified latency.

In my opinion, this is important for every team.

Links in the comments!

P.S.: Do you know of a better tool than Promptfoo?

#LLM #Data #Analytics #Performance #ArtificialIntelligence

Related Questions

What are some of the key projects Agostino Calamia has worked on at Shopify?
How did Agostino transition from a project manager to a data scientist?
What technologies does Agostino Calamia specialize in?
Can you tell me more about Agostino's book on machine learning lifecycles?
What are some of the challenges Agostino faced while working at N26?
Agostino Calamia
Agostino Calamia, photo 1
Agostino Calamia, photo 2
Add to my network

Location

Berlin Kreuzberg, Berlin, Germany