Suggestions
Øyvind Strøm
AI implementation, Prompt engineering, Prompting courses, Strategic AI, Innovative solutions, EDW/BI, DB modelling, SW design, Architecture, Business Processes/Terms/Rules, CEO, COO, CTO, Project Manager, Agile.
Øyvind Strøm is a multi-talented person with over 25 years of experience in designing databases and systems, and over 18 years of experience in Business Intelligence. He has worked as CEO, COO, CTO, Board Member, Project Leader, Business Analyst, IT-Architect, IT-Designer, and IT-Developer for various organizations, including Nykredit, BI.Nordjylland, Pock Mobile, Platon A/S, and ncg Sweden ab. He is passionate about helping weaker members of society, environmental issues, and ruralism. Øyvind is a former professional photographer and a writer, with a few books in the pipeline. He loves traveling and has been to 50 countries. He is open to invitations and has a network of over 12,500+ 1st level contacts on LinkedIn.
Highlights
I am invited as a speaker at a Datawarehouse conference in Hannover in September. I will present how AI can assist in building a company’s Enterprise Datawarehouse (EDW). The conference is called Data Dreamland, and is held by Scalefree, a company that works entirely with implementation of Data Vault. Data Vault is a modelling technique and a methodology for implementing EDW’s, and in Nykredit we use this modelling both in the existing EDW and in the upcoming EDW. The process we use to add elements to our EDW involves a lot of steps, and for several of these steps, use of AI can drastically improve the process. - Ensuring that requirements meets standards. An AI can assess requirements – OR – and AI can assist in preparing requirements. I have made an AI assistant that interview someone with a report need, and collects all required information. Based on this, the AI presents an example report to the user. - Connect requirements and Data Governance. An AI can categorize data elements into the correct data domain, examine our business glossary, suggest proper element names and definitions. - AI can assist in identifying suitable sources. - AI can assist in data profiling, identify possible values and assist in creating data quality controls. - AI can assist in Data Vault modelling and dimensional modelling. - AI can integrate with our ETL automation set-up for even faster time to market. - AI can assist with creating Power BI reports. - AI can create documentation. As I only have a slot of 30 minutes, I will briefly list the above, and dive into how AI can assist with Data Vault modelling, as Data Vault is the main theme of the conference.
Can you use AI to automate (most of) your EDW? Adding stuff to an Enterprise Data Warehouse can be a tedious task that costs time and money. What if AI could interview a manager, and next day give him/her a report, that is fully based on the EDW. With no human assistance? The sole purpose of an EDW is to provide users with data they need, so the users do not bog down the productions systems with inquires. In larger organizations, it is a tedious process to add stuff to the EDW. It involves architects, designers, business analysts, data modellers, data engineers, data governance, testers and data security, and the user can expect a delivery time of weeks or even months. And most likely more, because the EDW department is understaffed, and the backlog is long. In such cases, the manager might even have “to fight” to get the report prioritized. If so, a lot of time is spent in meetings and meeting preparations. And the cost for the organization are in the tens of thousand dollars range, even for simple reports. Based on my experience with EDW/BI and AI, I will claim that it is possible to automate most of the tasks involved. I have worked in the EDW/BI domain for more than 25 years, doing everything from assisting users to identify requirements, identify sources, profiling, designing data models for the various layers, writing ETL programs, writing specifications for ETL developers, testing, making multidimensional models, tabular models, creating end-user reports and designing automations. And add MDM and BPM to the mix.
I can set up an AI assistants that interview the users, and make sure everything is covered And I can make a multitude of AI agents that identify sources, profile data, onboard data to DSA/Data Lake, model data, move data, communicate with business glossary, make test cases/test plan, performs testing and even deploying everything. You need to see the AI as a co-worker - that you need to on-board. And you need: - A formalized Standard Operating Model (SOP)/Business Process that describes each step, handovers and deliveries. - Access to a business glossary - Documented naming standards for tables and columns. - Access to Data Linage - Overview of sources - System documentation for tables in your different layers. Initially, you would want human specialists to verify the output between each step. Even so, it should be obvious that the throughput of such a setup would allow a team to deploy solutions at a MUCH faster speed. The AI make data models and write ETL code in seconds. When the setup has proven itself, more and more steps can be chained and automated, by using AI agents. And the Agent flow may or may not include human approval at certain steps.
The only challenging task here is the data models. Out of the box, ChatGPT 4's is capable of making DataVault 2.0 and Dimensional models, but some additional instructions are needed for better results.
