Build Your Own Cloud Storage System Using OSS

The real threats to stored data are breaches which, of late, have been affecting many cloud service providers. Security vulnerabilities that enable breaches result in a loss of millions of user credentials. In this article, we explore the prospects of setting up a personal data store or even a private cloud.

- Swapneel Mehta

The European Organisation for Nuclear Research (CERN), a research collaboration of over 20 countries, has a unique problem—it has way more data than it is possible to store! We’re talking about petabytes of data per year, where one petabyte equals a million gigabytes. There are entire departments of scientists working on a subject termed DAQ (Data Acquisition and Filtering), simply to filter out 95 percent of the experiment generated data and store only the useful 5 percent. In fact, it has been estimated that data in the digital universe will amount to 40 zettabytes by 2020, which is about 5,000 gigabytes of data per person.

With the recent spate of breaches affecting cloud service providers, setting up a personal data store or even a private cloud becomes an attractive prospect.

Data storage infrastructure is broadly classified as object-based, block storage and file systems, each with its own set of features.

Object-based storage This construct manages data as objects instead of treating it as a hierarchy of files or blocks. Each object is associated with a unique identifier and comprises not only the data but also, in some cases, the metadata. This storage pattern seeks to enable capabilities such as application programmable interfaces, data management such as replication at object scale, etc. It is often used to allow for the retention of massive amounts of data. Examples include the storage of photos, songs and files on a massive scale by Facebook, Spotify and Dropbox, respectively.

Block storage