What You Must Know About Real-Time Data
This is the Three Data Point Thursday, making your business smarter with data & AI.
Let’s dive in!
… it’s not about the real-time data.
In 2012, a father in Minnesota got super angry at Target because they started to send his teenage daughter maternity goods in the mail. He felt they were harassing his daughter - to the embarrassment of the Target store.
However, a couple of days later, it was the father who suddenly felt embarrassed, realizing Target knew his daughter was pregnant before he did.
While Target has now changed its policy to a less aggressive one, it is clear that 13 years ago, purchasing behavior was enough to figure out such intimate information about consumers.
To me, however, the real magic in this story doesn’t lie in the use of big data (this is truly big data) but rather in the use of it. What made the father so upset and later embarrassed? It wasn’t the collection of all this data, it wasn’t the fancy algorithm that Target used.
Rather, it was two things: The actions Target took, and the time, the almost real-time; for the father, it was more like pre-time, actions.
And this is exactly what everyone gets wrong about real-time data: There is next to NO value in having real-time data, data that is collected in real-time and available for analysis in real-time.
Almost all of the value is in fast actions. Real-time actions can be completely removed from real-time data. For Target fast actions mean:
Acting before anyone else does (the father)
Acting at just the right moment for this particular person (time personalization)
Fast actions are really “before competitor actions.”
If you’re an Uber driver and miss a potential drive - you just lost that money. But it’s not because you didn’t have real-time data. There could be minutes between the data collected in real-time and your acting on it.
The key isn’t you at all; it is the competing drivers. That’s a weird thing, right? The time for competitors to act dictates most of the decay of information. If you’re the only Uber in town, well, you can be an hour late. But if there is only one more Uber, you’re in trouble if you don’t act quickly.
But it’s not just competitors; for the girl that bought at Target, it is also the alternatives. Once she knows she is pregnant and starts to explore products independently, she’ll pick stuff, and Target’s advantage will be diminished.
Just as for Uber, it’s not really just about getting data faster than the competition. The Uber driver needs to build himself a small alerting interface and make sure he is awake and not too busy with other duties. There’s a lot to optimize besides getting the data before the competition.
Fast actions don’t need real-time data
Let me repeat: real-time data and fast actions only collide in one world: The financial world of quantitative trading and hedges.
This is the only world I know of where the pipelines are so standardized that real-time data and real-time actions become the same thing. For everyone else, they aren’t, and they shouldn’t be considered to be the same thing.
In almost every other world, the bottleneck isn’t getting the data fast; it is turning it into actions. Even the Netflix algorithm takes a couple of minutes to update to recent updates in movie-watching behavior.
And that’s not at all bad because having a 5-minute accurate recommendation of TV shows is 100% faster than any television channel.
Fast actions aren’t really “real-time”. They just are faster than the other guy - and that might be measured in days, hours, minutes, or seconds. I am leaning to the former most of the time.
That, in turn, means you might need data quickly but never as quickly as you think you do. Having it on a 1-minute, 5-minute batch interval will be enough to be at the forefront of the business world for now.
Prices on Amazon are like the weather in Iceland. If you don’t like them, just wait 5 minutes—prices on Amazon change 2.5 million times a day.
Why? To optimize, of course. Optimize in two directions: price common articles lower than the competition and uncommon products higher in general.
Indeed, for uncommon products, Amazon could check your viewing history and start changing the prices on these if you keep looking at them - or if you wait.
There is power in acting just at the right moment for a single individual - the power to extract an enormous amount of extra margin or an extra sale.
But this concept is greater than that; personalization usually means adapting the context of what you’re selling to a person, but you can just as well adapt the context over time, what I like to call time personalization.
It’s a powerful tool most companies miss, except, of course, companies like Amazon.
Another example of a coarser approach is Netflix, which loves to recommend “Shows popular in your area right now” - recommendations that have to be based on mostly real-time data, delivered as real-time actions to you (personalized to your location).
A final but similar example is a common feature in many e-mailing suites. Since people tend to open most emails in the morning when they get to work, but email audiences are often distributed across the globe (as this one is), a lot of e-mail tools offer the option to send emails based on the target time zone (personalizing the actual time point to you).
Fast actions mean you choose to personalize the time and/or the context time of your thing over collecting data in real-time.
Fast actions cover the whole chain
The fashion giant Zara still manufactures most of its clothes in Spain, which is quite uncommon for clothing companies of their scale.
But you’ll see why this matters so much at a second look. The Zara founder Ortega decided one day, stopping at a red light, that a motorcyclist was wearing a jeans jacket he loved and wanted to add to his production line & shops - within 1-2 days.
That’s the speed at which Zara is able to respond to changes in the ecosystem; as Oretega says in Blitzscaling by Reid Hoffman, “What seems great today, in two weeks, is the worst idea ever.”
Ortega doesn’t need real-time data to pull off astonishing speed; he does, however, need a quick, complete chain of decisions & actions implemented. He built highly automated manufacturing sights throughout Spain, paired with three hundred small shops that process the greige goods to finished product - a process designed for flexibility and getting new designs into shops within 1-2 days.
The key to fast actions, thus, is to optimize the whole chain of decisions and actions, not so much the data.
Data value has a shelf life
In my mind, the best picture for data and its value is simple:
Data value has a shelf life. It is at 100% with the moment it is created (not collected!). With a physical, real-world event taking place.
The first time a competitor OR a customer moves based on this data, the value is cut in half.
The second time a competitor OR customer takes another action based on it, it halves again, going to 1/4th of its original value.
Eventually, you’re left with a little value; that’s the data you’re using for historical analyses; it doesn’t really matter how much value is in one individual data piece because you have a ton of aggregate. Nevertheless, the value YOU get from data only depends on how quickly you can act on a physical event taking place vs. the market/ the competitors.
The Target and the Amazon cases are taken from the book “Swipe to Unlock - The Primer on Technology and Business Strategy,” which is worth a quick read.
Here are some special goodies for my readers:
👉 The Data-Heavy Product Idea Checklist - Got a new product idea for a data-heavy product? Then, use this 26-point checklist to see whether it’s good!
The Practical Alternative Data Brief - The exec. Summary on (our perspective on) alternative data - filled with exercises to play around with.
The Alt Data Inspiration List - Dozens of ideas on alternative data sources to get you inspired!