The Development of Big Data

Human beings have been generating data for thousands of years. Ancient civilisations such as Ancient Egypt and Rome understood the value of recording information and the uses this information could have. Though humans have been generating data for millennia, the true advent of Big Data as we understand it today really came about with the invention of modern computing. The history of Big Data can be viewed in three distinct phases. These phases each began with significant technological leaps.


Structured Data and Relational Databases

In the beginning, Big Data analyses focused on structured data. As discussed in a previous blog post, this kind of data can be organised neatly into rows and columns for example. The rise of Realtional Database Management Systems in the 1970's, and techniques such as structured query language (SQL), as the dominant approach for data management allowed people to gather and analyse large amounts of data such as sales figures, inventory records, or customer demographics. These systems provided people with the tools to efficiently store, organise, and query this structured infromation. This benefited businesses and organisations and allowed them to gain better insights into their operations allowing them to make better decisions and work more efficiently. These early developments still sit at the core of many Big Data Technologies provided by companies such as Microsoft, Google and, Amazon today.(1)

The Web Revolution and Unstructured Data

The new era of the internet came will a fundemental shift in data. The sudden explosion of websites, emails, social media, and online transaction led to an unprecedented shift from structured data to unstructured data. As discussed in a previous blog post, unstructured data is not categorised easily such as text documents, images, videos, social media posts, and web server logs. This explosion of data was extremely valuable but required new approaches and technoligies within the field of Big Data. Businesses and organisations now had access to a wealth of data about customer and client behaviour. Using Big Data technology to analyse click-through rates, search queries, and website navigation patterns were crucial in understanding user engagement and optimising how these businesses interacted with customers in the new digtial realm. These kinds of data and the way business have collected and analysed them have shaped the modern web as we know it today. This era super charged the demand for data storage and analysis.(1)


The next era is the era leads to the world we are currently in with the wide spread adoption of mobile technologies and the rapidly expanding internet-of-things right up to the advent of AI and large language models and will be discussed in my next post.

1.https://www.bigdataframework.org/knowledge/a-short-history-of-big-data/


Comments

  1. Josh McilvinneyMay 14, 2025 at 11:05 AM

    I was very fascinated by the history of big data. Learning that data was measured and used in ancient times is incredible, it makes sense, but I never thought about it in this context.

    ReplyDelete

Post a Comment

Popular posts from this blog

Value of Data

 In the modern world, data is a valuable resource. The value of data though can be harder to quantify compared to other more tangible resources such as equipment, real estate, or employees. Though these assets can be more easily digested in terms of cost vs. value, the impact data has had on the modern world in undeniable through cost reduction, revenue increase, and income generation. With data being infinitely produced at an ever-faster pace, the value of data is only increasing. Though data is being constantly produces in the modern, the type and quality of this data varies massively. Data alone doesn’t necessarily have the value that business and organisations are looking for and that is why Big Data technologies must be applied to extract value from the endless sea of data and gets that data to the right people at the right time. Using Big Data to extract value from data comes at a cost and how the end results balance out this cost can be understood through the 1-10-100...
Read more

What is Big Data?

 "Data is information that can be interpreted and used by computers. It is a collection of facts, such as numbers, words, measurements, observations or even just descriptions of things. In computing, data is typically stored electronically in the form of files or databases. Data can come from many sources including user input (typed words or images), sensors (temperature readings.) or algorithms (calculations)." (1) In today's modern world this data is being generated at such a scale and rate that it can no longer be processed by traditional data management tools. This is what is known as Big Data. It is estimated that 90% of the worlds data was created in the last two years alone with current trends estimating the amount of data created in 2025 will be 181 zettabytes. (2) These massive amounts of data being created can be categorised in three ways: Structured - This kind of data is highly organised, easily searchable, and is easily processed by Big Data technology in dat...
Read more

Traditional Statistics

  In my last post I discussed the value of data. In that post I discussed how value is extracted from data by processing raw data and packaging it but what does this mean? Data in its purest form is just points of information. Without context or analysis applied it doesn’t necessarily serve much use. One way in which human beings derive value from is data is through statistics. Traditional statistics is a tool for understanding and interpreting data. There are many ways in which this tool can be applied to data, with two main branches: descriptive and inferential . Both are ways to extract meaning from data but are used in different ways for different purposes. Descriptive Statistics Descriptive statistics are used to describe or summarise data in a meaningful way. They help us visualise what the data is showing and make it easier to recognise patterns that may emerge from the data. Descriptive statistics are applied to a sample and measures chosen properties of that sample t...
Read more