Sign In

Tobias Norlund

Industrial PhD Graduate Student - Natural Language Processing

Tobias Norlund is an AI/NLP Researcher and Engineer based in Stockholm, Sweden.1 He has a background in data science and has held several positions at Schibsted Media Group, including Lead Data Scientist and Data Scientist.4

Norlund's professional experience includes:

  1. Working as a Data Scientist at Schibsted Media Group from September 2017 to September 2020.4
  2. Contributing to research on the use of AI technologies in journalism, including co-authoring a paper on evaluating the perceived "human-likeness" and "informativeness" of text produced by an NLP model trained on Swedish online forum data.2
  3. Involvement in projects related to workflow automation and AI agents, as evidenced by his participation in AI Sweden's Startup Community events.3
  4. Development of AI model architectures, such as UI-Act, designed for autonomous interaction with user interfaces.5

Norlund has an academic background that includes studies at Chalmers University of Technology.1 He is active in the field of artificial intelligence, particularly in areas such as natural language processing (NLP) and intelligent automation.5

On GitHub, Norlund maintains a profile with 55 repositories, indicating his involvement in various coding projects.6

Highlights

Oct 12 · linkedin

Happy to share a project I've been working on together with John Pertoft - UI-Act.  It's an AI model architecture designed to autonomously navigate and interact with computers using the graphical user interface. Think of it as a co-pilot that "sees" your screen and acts on it, just as a human would.

🔍 What's UI-Act? UI-Act is a custom Transformer-based ML model that takes raw-pixel screenshots of the full desktop as input, and predicts low-level actions (mouse clicks) in sequence. It can be trained to perform tasks from expert demonstrations using behavioral cloning, optionally also with a textual/token-based prompt. In a provided example demo, the model has been trained to calculate a simple arithmetic expressions in a GNOME Calculator window, based on a prompt. It is also (at this stage) very small and can easily be run locally on a laptop.

💡 Why is it interesting? General AI agents are set to drive automation and increase productivity in many different ways. LLMs equipped with e.g. function-calling is one approach, such as e.g. Microsoft Copilot and HyperWrite, that rely on text-based state representations and APIs to take action. In cases where this is unfeasible, UI-based agents might provide a more general alternative. As the agent's interface to the computer is shared with humans, it can be easily taught using expert demonstrations, and require little or no technical expertice.

🤝 Collaboration and Feedback It's still very early stages, so any thoughts, feedback, and ideas are warmly welcome. For example, from a technical perspective, I'm excited to explore different pre-training strategies to allow the model to scale similar to #LLMs. From a use-case perspective, I'm keen to discuss with people working in e.g. #intelligentautomation and #rpa about your thoughts of this space.

AI for automation is only getting started, and I'm super thrilled for what the future holds! If you find this project interesting and want to chat about it, please comment below or DM me directly :)

https://lnkd.in/g7bxNwtt

Related Questions

What are some key projects Tobias Norlund has led at Schibsted Media Group?
How has Tobias Norlund contributed to the field of computational journalism?
What is the significance of the experiment at Aftonbladet mentioned in the article?
How does communicative AI impact the work of journalists according to Tobias Norlund?
What are the main challenges Tobias Norlund has faced in implementing AI solutions in newsrooms?
Tobias Norlund
Tobias Norlund, photo 1
Tobias Norlund, photo 2
Add to my network

Location

Stockholm, Sweden