Innovate More Rapidly

How do you undertake a journey to the data cloud? Digital transformation is going on around us. It is happening across all aspects of the society. We are now learning how to integrate new technologies

- Pradeep Chakraborty

Change has accelerated in the past decade. Earlier, systems were deployed with the expectation that they would last forever. They were not designed to look at each other’s data, and were fairly limiting. Open source was a new idea in the early 2000s. People began to adopt Lucene, a software I had written. There was no institutional backing or publicity. Open source emerged as a tool for development.

A Distributing Computing Platform

Nutch started in 2003. Around 2005, Google published a paper on how they build search engines. They had a paper talking about how they had automated things. We started working on reworking Nutch in 2004. The tale of debugging is much longer. In 2006, I joined yahoo! I developed Hadoop. Hadoop was named after my son’s toy elephant. It was a distributing computing platform, based on Google’s ideas.

A group of people believed that Hadoop could be used much further. Together, they formed Cloudera. I joined Cloudera in 2009. Stepping back from my lesson in Hadoop, if you can increase the scale and focus on flexibility, you can permit them to store more data in raw form and experiment. They can innovate more quickly. The waterfall method inhibited process through data. This gave us a much more appropriate platform. Most of the past data was relational.

New sources of data are events, things recorded from sensors, etc. We need a different class of tools. Companies can run petabytes of data easily today. Software is also eating the world. In every industry, everywhere, the advances being made are predominantly using software. A company’s growth is fuelled more by data, today. The use of data is no longer isolated. It has emerged everywhere.

Challenges