A few weeks ago I visited Malaga, a coastal mediterranean port city in the south of Spain. The city is one of the oldest in the world. Streets are lined with lemon trees and plenty of archeological remains displaying a rich history that dates back three thousand years.
One of the most impressive things to see in Malaga is the cathedral, a gigantic Roman Catholic church soaring in the middle of town. The cathedral is surrounded by street vendors, artisans, and musicians.
As I was coming out of the cathedral one of the vendors lured me to his stand where, among all the trinkets and souvenirs, one particular item on his table caught my eye. It was a hand-made “historiogram” – a timeline of events going back a few thousand years, showing correlations and causations of significant events in history, including elections of emperors, kings and presidents, wars, notable births and deaths, elected religious figures, persecutions, thinkers, discoveries, scientific events, arts, literature, philosophy, and sports. This timeline had correlation and causation for events such as the creation of universities, civil wars, and plagues, it mapped these events to inventions such as the train, telephone, and the phonograph. It represented two thousand years of events, all mapped into a data chart.
“History has a tendency to repeat itself, and if we can understand the past we may be able to predict the future” said the street vendor as he handed the chart to me.
It is mind-boggling to me how someone, in a small city in the south of Spain, can create such intricate data mapping based on his readings, teachings, and knowledge that was passed down to him. I wondered how many more charts are in the world capturing infinite amounts of knowledge.
It turns out that the world is a very noisy place. Data such as this exists in droves and is generated in exabytes every single day. Ninety percent of all the data in the world has been generated in the last two years alone. To make sense of things in the presence of so much information, we humans have to be very selective with our attention to which type of data is important to us, and which is not. Over the course of millions of years we have become fairly good at filtering out bad data from good, and along the way we learned how to associate good data with certain events. We need data, but what is the data really telling us?
Something else that is exceptionally interesting is that as much data as there is in the world we are only exposed to a microscopic portion of the overall scope at any point in time, so the decisions and mappings that we make with the data that we see are limited and constrained to that to which we are exposed. This is the case in most organizations, where we tend to create taxonomy for data that is only relevant to that particular organization. While well governed by lineage, ontology, and data governance, the definition of a particular entity is generally only applicable within that enterprise. More than ever before we are awash in data about virtually everything.
One of the most exciting trends for 2019 is the continued movement of predictive decision-making. And this is not only for corporations, but also for the data-underserved, that being individuals who can finally begin to innovate and use algorithms for personal use. The idea that individuals will be able to leverage their own as well as external data to create rich and accessible information that is viable for better decision-making in an automated way is fascinating. Integrating this capability with extended services of connected devices in a secure and efficient way will also give us the potential to extend decision-making to the physical space. There is a lot that needs to be done to make this real. A common ontology to understand data in a common way across individuals and industries would be ideal. A more secure way to handle, transfer, operate on this data would also be ideal. Defining how and when to trust certain data and algorithms is also key. A lot of effort is going into this space, and significant progress is being made to move the ball forward.
With technology pushing the world to move so fast, this is about to rapidly change. Democratization of infrastructure via public clouds, as well as democratization of generic algorithms in these clouds have fueled this change. If we could learn to find a common way to understand each other, as well as learn how and when to trust algorithms that use our common data, the transformation would be unimaginable.
Dr. Julieta Ross 