From Data to Knowledge to Intelligence | PART ONE: The Journey Map

First, I want to Say Thank You for sending comments and sharing the first article of this series. Personally I found your response both humbling and invigorating at the same time. Thank You again!

Today, I’d like to take a closer look at the overall Journey. It’s not that the individual stages are not worth discussing (yet!), but I think we need to spend some time understanding the bigger picture before diving into a lot of details (and some code).

Essentially, what we are saying is that data goes through stages, as represented in the next figure. If starts in its inception (i.e. a telemetry data generated by a sensor), then morphs into a series of increasingly complex data structures and eventually arrives to some decision-making point.

We are calling these three stages:

  1. The Data Stage: the Data has been acquired, its initial format is nothing more than a collection of raw values
    • Example: A data packet generated by a weather station may look something like “98.91,overcast,1.25,1539725035”
  2. The Knowledge Stage: the Data is morphed into something that helps a human (or a system) gain Knowledge about a particular condition, situation, or event.
    • Example: that raw data packet can be morphed into something like “temperature = 98.91, weather = overcast, rainfall_total = 1.25, local_time = Oct 16, 2018 9:23pm
  3.  The Intelligence Stage is when Data-turned-into-Knowledge is being used for decision making
    • Example: avoid that particular area because of potentially bad road conditions.

This is all good, however there is one CRITICAL piece of data that is missing. Any guesses??

Yes, you are right — we don’t know WHERE that particular weather station is! We reached our first decision point only to realize that we can make a better decision, should we get one more piece of data, namely the location of the weather station.

Just to recap … initially the Journey seemed relatively straight forward … then we realized that we may need to backtrack few steps and relive the Journey from there (again!). In other words, what initially appeared to be a predictable progression from Data to Intelligence turned out to be just another vantage point on a bigger Map. What’s even more intriguing is that our first “intelligent” decision had a very short life and, even worse, it became a dead end as soon as we realized that there is a better decision to be made. In order to make this kind of rapid adjustments, our initial Journey MAP has to become a lot more iterative to the point that we are now start creating dependencies between stages (aka feedback loops).

As shown in the figure above, there may be times when we need to go from the Intelligence Stage back to the Knowledge Stage and see if we can collect additional pieces of Knowledge that we might had missed the first time. If that attempt yields no results, then we need to go all the way back to the Data Stage and see if we can gather more Data. In our example, this would amount to inquiring the weather station (i.e. via an API call) to “reveal” its geo-location, namely its latitude and longitude.

This all may sound like boilerplate, however I hope we can all agree that this thought process helps us establish a structure that we can then use to map out our Data Journey. Knowing exactly where you are in your Data Journey, what lies ahead of you and what places you may need to revisit are all highly acclaimed skills that we all need to master.

One last point before closing … from a more practical point of view it should be noted that there is a cost associated with all these iterations. Having to go back and forth to gather more data, train the model(s) a little more, etc. cost time, money and resources. It also adds complexity. All these incremental costs add up quickly to the point that incurred costs exceed the actual value gained from this exercise. Sometimes Perfect DOES get in the way of Good! Don’t let that happen to you!

That’s all for today! In the next articles we will dive deeper into each of the stages and look at the Data from different perspectives: Consumer, Producer, Insider, Observer, and so on.

That said, I want to leave you today with one question that hopefully you will find it just as intriguing as I do: let’s say YOU ARE that one text message that your cellphone just sent out. What the World would look like to You?

Until next time, all the best! Talk to you soon!