Thoughtful Friday #22: Gartner is Wrong. Big Data will be Bigger Than Ever - and You Too, if You Embrace it.
Just because big data is hard to do, doesn't mean you shouldn't do it. It's the reason why you should, or you will sooner or later fade into irrelevance behind companies that know how to handle it.
I’m Sven, and this is Thoughtful Friday. We’re talking about how to build data companies, how to build great data-heavy products & tactics of high-performance data teams. I’ve also co-authored a book about the data mesh part of that.
Let’s dive in!
🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮🔮
The research firm Gartner "delivers actionable, objective insight to executives and their teams.", with its famous "magic quadrants" it helps to identify and separate "Niche Players" from "Leaders". Their recent advice is quite actionable "focus on small data instead of big data". It's so actionable, it will push you from being on track to becoming a market leader down to being a niche player, or no player at all.
Do yourself a favor and focus on big data, ignore Gartner, and help your company to understand the value of integrating big data into the company strategy. If you ignore big data and focus on "small data" a competitor will be able to leverage big data to build up data network effects. While that sounds harmless, it will effectively establish a monopoly centered around your competitor and push you out of the market for good, just like Yahoo.
Because, what most people seem to fail to see is, that most leaders in a category are already deeply embracing big data.
Data network effects create monopolies.
(no driver inside, precision to the inch, better than any human driver after 20 years of training, and affordable)
Just take a look at sectors around you, John Deere is all-in on AI & data and market leader with 44% in agricultural machinery building AI-powered autonomous tractors. Facebook is basically a huge data-crunching engine, with a market share of close to 80% in social, Google, the mother of all data companies is almost the only search engine out there. And Airbnb is leveraging data to match stays and stayers on a massive scale, even growing the whole market segment.
Data network effects always work the same way. All companies deal with some kind of data. But a lucky few are able to turn this into a reinforcing engine where each new user of the product or participant in a multi-sided platform brings more data and this data makes the product/platform even better for everyone else!
Like Google: The more web pages there are, the better the algorithms get. The more images, the better the image search. The more stays that happen on Airbnb, the better AirBnB matches. More autonomous tractors sold mean better autonomous tractors for everyone (they are software powered and this software can be updated). Google, Airbnb, and John Deers tractors get better for everyone when someone new joins, a new transaction happens and the data is fed back into the data-crunching engine.
With all kinds of network effects, this means the strong only get stronger, and everyone flocks to the strong players, creating markets of one.
Key Point: There are just two key skills you have to master:
Grow your ability to extract value from any kind of data (best tied to users/participants, anything that will speed up network growth)
grow the amount of data you're gathering on a permanent basis
and that's it. As every company wants to develop data network effects sooner or later, you simply cannot dismiss this, you must become a leader in this regard.
Why Gartner would tell Yahoo to optimize their manual efforts.
When Yahoo started out as a search engine in 1994, they were in the market before Google. They hand-curated lists of web pages because that was how they ensured the quality of the pages they recommended.
When Google came along in 1998, a small competitor at that time, it would probably seem like sound advice to recommend Yahoo to streamline and automate their manual efforts by using automated qualification tools on the web pages they intend to list.
And that is what Yahoo did for 7 years after Google entered the market. You might remember, Google entered with a web crawler, a data collection system, and the famous PR algorithm on day one.
One of these business strategies is powered by data network effects, and the other isn't. Want to know how this played out?
(adapted from The Guardian - Yahoo Results)
Now the problem with Gartner is, they basically telling Yahoo to automate more of their manual tasks, to become efficient, and more optimized. That is indeed basically the idea of "small data", extract more value from the few selected things you already have (or want to acquire).
Google simply shifted the playing field and went all in on big data (in 1998).
Key Point: You need to understand the forces behind what is happening. If your business is aligned with them, then optimize the hell out of it. If you are not, you need to seriously change your game. The two forces behind the trend of "small data" are:
The growing amount of data makes it hard to take "everything" into account, so companies "prioritize", and are efficient.
Data is becoming more and more important, so there is a growing need for a product management perspective, one that simply tries to derive the maximum value for a company from the given data.
This leads many companies to adopt a "small data" approach, but for the wrong reasons. The playing fields are shifting all around them, and while they do need a "product management" perspective, and they do need to prioritize, they need to prioritize on a different playing field, one of the big data.
It's not their fault, so what do you do?
It's not Gartners' fault. Small data is a trend, and as researchers, they are focused on just that, trends.
But trends are mostly misleading, mostly "wrong" if you're looking to become a leader. Trends are trends because other people pick up something other people do. The true questions are what are the fundamental forces behind trends?
So what do you do? That's simple, and yet hard to do.
"But John Deere’s status as a leader in AI innovation did not come out of nowhere. In fact, the agricultural machinery company has been planting and growing data seeds for over two decades" (VentureBeat)
Key point 3: If you're a data engineer, do learn about big data. If you're a data leader, help your company understand the possibilities of integrating big data into the company strategy. If you're a CEO/ founder, this is your game. Change the playing field.
You need to unlearn the ideas that are so common. Understand and study the forces behind the data trends, question them, and pick up what is aligned with the underlying story.
So yes, do embrace data product management, but in a way that aligns your company strategy with data.
And yes, stop hoarding data just to have it, make it part of your business strategy!
Or try to keep on building & optimizing manned tractors until John Deere can literally "roll over you".
Ending Note: I've literally just read the headlines on this topic from Gartner, so I have no idea what "advice" they are handing out. I'm not making any claims on what Gartner is actually advising you to do, but I think the above says enough about what I think you should do.
What did you think of this edition?
-🐰🐰🐰🐰🐰 I love it, and will forward it!
Want to recommend this or have this post public?
This newsletter isn’t a secret society, it’s just in private mode… You may still recommend & forward it to others. Just send me their email, ping me on Twitter/LinkedIn and I’ll add them to the list.
If you really want to share one post in the open, again, just poke me and I’ll likely publish it on medium as well so you can share it with the world.