The big impact of big data

In my last post I discussed the history of Big Data and two of its three phases being structured data and relational databases and the web revolution and unstructured data. The third phase is the modern world of Big Data with the technolgical leap leading to this new phase being the wide scale adoption of mobile technologies and the rapidly expanding internet-of-things. Mobile devices such as mobile phones and tablets surpassed the number of laptops and PCs for the first time in 2011. These devices are basically constantly creating data whenever they have power! As well as the take over of mobile devices, the IoT has created a world in which many more devices in a home connect to the network and generate data through sensors. This means each houshold is becoming and endless stream of constantly generated data. When people use mobile devices they are constanly connecting to the network through apps and standard webpages and producing data to be analysed through behavioural data such as ...

Human beings have been generating data for thousands of years. Ancient civilisations such as Ancient Egypt and Rome understood the value of recording information and the uses this information could have. Though humans have been generating data for millennia, the true advent of Big Data as we understand it today really came about with the invention of modern computing. The history of Big Data can be viewed in three distinct phases. These phases each began with significant technological leaps. Structured Data and Relational Databases In the beginning, Big Data analyses focused on structured data. As discussed in a previous blog post, this kind of data can be organised neatly into rows and columns for example. The rise of Realtional Database Management Systems in the 1970's, and techniques such as structured query language (SQL), as the dominant approach for data management allowed people to gather and analyse large amounts of data such as sales figures, inventory records, or customer...

Due to the nature of Big Data, it is often hard to get your head around. Becuase of the size of Big Data, the variety of its contents, and its general complexity, its is often discussed using four key charactersitcis. These charactersictics are Volume , Variety , Velocity , and Variabilit y. Volume This charactersitic refers to the vast quantitites and scale of data being produced and stored. When discussing general data, it is often discussed in terms of megabytes, gigabytes, and terabytes. When discussing Big Data, the size of the data is much larger with the smallest amounts being in terabytes and more often involves petabytes and even exabytes! 1TB could roughly store the Lord of the Rings trilogy extended editions in 8K resolution. 1PB could roughly store the Lord of the Rings trilogy extended editions in 8K resolution 1000 times. 1TB could roughly store the Lord of the Rings trilogy extended editions in 8K resolution 1,000,000 times! (https://www.overcasthq.com/blog/how-big-are-v...

 "Data is information that can be interpreted and used by computers. It is a collection of facts, such as numbers, words, measurements, observations or even just descriptions of things. In computing, data is typically stored electronically in the form of files or databases. Data can come from many sources including user input (typed words or images), sensors (temperature readings.) or algorithms (calculations)." (1) In today's modern world this data is being generated at such a scale and rate that it can no longer be processed by traditional data management tools. This is what is known as Big Data. It is estimated that 90% of the worlds data was created in the last two years alone with current trends estimating the amount of data created in 2025 will be 181 zettabytes. (2) These massive amounts of data being created can be categorised in three ways: Structured - This kind of data is highly organised, easily searchable, and is easily processed by Big Data technology in dat...