Analytics Engineering (NOT DBT!) 101
Analytics Engineering Isn’t About Dbt; Synth Video Presenters Are Becoming A Thing; Cyber Security for Vikings
This is the Three Data Point Thursday, making your business smarter with data & AI.
Want to share anything with me? Hit me up on Twitter @sbalnojan or Linkedin.
Let’s dive in!
Cyber Security for Vikings
Viking shields were round, flat, and lightweight. They were adapted to the weaponry used then and the way of combat.
Now imagine a new enemy with new weapons, except each weapon is unique and customized to the particular shield the Viking uses.
Even worse, the enemy weapon can adapt and change! A very unfair game for the Vikings.
That’s what is happening right now in cyber security. The new generation of genAI has also brought a new generation of cyber threats to the average company.
GenAI makes cyber threats that previously were mass-oriented now personalizable and much more potent.
The business opportunity: Data-driven cyber security will be a booming sector, in our opinion, thanks to the unique combination of a growing threat and a growing mountain of data to combat it.
What should a company do? You need to step up your cyber security game; phishing attacks will be personalized on all channels, so watch out and tighten your security everywhere.
Synth Video Presenters Are Becoming A Thing
Synthesia’s round C got us excited; it turns out this trend is only accelerating.
This is Lisa, introduced back in July on an Indian TV channel.
And she has a ton of friends in Southeast Asia. Read up on some details and some worrisome developments in “The Batch by Andrew Ng.”
The gist of it:
AI is already good enough to let amateurs create convincing videos.
Companies like Synthesia are forming to make this very simple.
We’re not seeing much from that right now because we’re more AI-phobic than Southeast Asia.
So what? First, we’re betting on that space as a huge growth market; the AI phobia will be overcome and open up a big market.
But there are already two niche market segments where everyone should consider using synthetic video presenters:
Corporate training videos (the niche targeted by Synthesia)
Broadcasting of video messages (that could be considered a broader niche)
What should we do? Play around with it. See how people react to AI presenters inside your videos. And we’d love to see more companies created in that sector!
Analytics Engineering Isn’t About Dbt
While Tristan Handy, the Fishtown Analytics gang, and DbtLabs might’ve invented the term analytics engineering, analytics engineering today, per se, has nothing to do with this specific tool.
We think it’s most helpful to think about analytics engineering as an organizational change.
To be clear: Just because you have dbt doesn’t mean you’re doing analytics engineering.
Why? Because analytics engineering is about bridging a gap in communication between analytics and data engineering. That’s a people problem, not a tech problem.
You only solve a people problem by working with people first and tech second.
Enough of that; here’s our take on this organizational change.
Analytics Engineering Organizational Change Summary
This is Dan, the data engineer. he just finished a hot new dagster pipeline, pulling order and account data out of the production system and pushing it into a data warehouse. he tops it off with a simple star schema to make the data workable for the analysts.
Adam, the analyst, created a new fancy dashboard showing the monthly rolling revenues by customer segments compared to last year.
However, a week later, Adam is confused. The revenues went up until two days ago, when things started to plummet.
After a lot of discussions with Dan, they figured out the problem. Dan didn’t know orders had an “order status” for shipped/canceled orders. Until recently, canceled orders were included in the order data, but now they are not anymore. So, the revenue numbers were overreported…
This is the data analytics - data engineering gap; Missing knowledge on one side, missing communication on the other.
The analytics engineering solution? Hire Eve, the analytics engineer, who knows enough business and collaborates with Adam to have the necessary knowledge, but also (just) enough tech skills to pull off the harder parts of this job Adam shouldn’t have to do.
Twitter description: Bring engineering into analytics or analytics into engineering.
Define: Introducing Analytics Engineering means introducing analytics engineers into your company to bridge the gap between analysts and data engineers.
You do so by (1) either upskilling analysts or (2) hiring people to sit in-between.
Introducing Analytics Engineering thus is almost entirely a process change! It affects the data teams as well as the business analysts/decision makers, both sides of this human interface.
Explanation: The idea is simple, data flows from production to consumption, through the data engineer (or his pipelines). The analyst sits at the consumption (in manual analytics use cases like business intelligence). And there is a gap between data engineer and analyst.
Analytics Engineering (get it?) bridges this gap.
How? Usually, the analytics engineering approach is to make the analyst more engineery. The underlying assumption is, that the missing business knowledge cases a ton of trouble in the data engineering department, and thus in the flow of data from its production through the pipelines to the consumption.
Is this a change you should consider right now for some urgent reason? In our opinion, no. It’s one possible solution to the problem described above.
But there is now outside urgency; there is no reason to believe this problem is becoming more apparent, and there are other solutions.
How do you evaluate whether this is a change for you? Is your data engineering team limited because of missing business knowledge? You don’t see an obvious different way of resolving this problem? And you’re hitting scaling limitations because of it? Then this is an option for you.
Definitely pass on this idea if: You don’t have complex business contexts, and yes, in our opinion, that means no start-up should do analytics engineering. You’ll scale way faster without it.
Further resources: There’s surprisingly little written about analytics engineering in itself. Here’s one good article.
Goodie Time! Exclusive Gifts
Here are some special goodies for my readers:
👉 The Data-Heavy Product Idea Checklist - Got a new product idea for a data-heavy product? Then, use this 26-point checklist to see whether it’s good!
The Practical Alternative Data Brief - The exec. Summary on (our perspective on) alternative data - filled with exercises to play around with.
The Alt Data Inspiration List - Dozens of ideas on alternative data sources to get you inspired!