Insight into Big Data

Organizations produce an exploding amount of data while capturing bushels of bytes of information pertaining to their operations, customers and suppliers and this data is continuously increasing. To give an idea, it has been forecasted y International Data Corporation (IDC) that between 2009 and 2020, data will mammoth 44 times amounting to 35.3 zettabytes which is equivalent to 1.8 trillion gigabytes. Thus, a burgeoning volume of digital ‘exhaust data’ is generated by companies as they manage their businesses and interact with stakeholders.

Introduction to Big Data:

Big Data refers to an amalgam of various kinds of data. It comprises of traditional company produced data such as customer relationship management (CRM), administrative and financial particulars, management information systems, supply chain operations , etc that can be stored in the company’s database. It also comprises of content such as social media, video, sensor data (produced by clicking links on social networks) and email. But the term Big Data is coined for the above kinds of data when it is too big for conventional systems to control it. Size alone is not an indicator for bigness. Data is big because of:

• Volume: Abundance of information.
• Velocity: Changing rapidly.
• Variety: Unstructured and not usable in current form.

This data can be made very useful by slicing and dicing it with analytical tools to identify trends and patterns, and assist in decision making by performing business intelligence. In today’s world, information can easily be shared across the organization by deploying SAP, Microsoft Dynamics and Oracle Enterprise Resource Planning (ERP) systems. This merging and sharing of data not only increases the complexity, but also the analytical potential. To illustrate, around a decade ago, company historical data was the only way to estimate future sales. But today, many more resources such as competitive analysis, market economics and website clicks can be used as an input to predict revenue. All these are nothing, but avenues of Big Data.

Challenges:

Data is truly big when companies have to design innovative ways to collect it and make sense out of it which is a challenging task. Companies are being bombarded with data and may not be proficient in handling and analyzing trillions of bytes of data. This can be attributed to:

• Company’s paucity of infrastructure to manage the sheer volume of data: There is often limited storage or server size to accommodate all the data being generated.
• Lack of appropriate talent, i.e. Big Data Analysts or Business Intelligence professionals: As per McKinsey and company, there is a shortage of workforce appropriately trained on experienced in Big Data analysis. Also, management in most small and midsized firms and to an extent large corporate lack management strategy to capitalize on the data.
• Threat of data integrity: As previously mentioned, owing to ERP systems, unstructured data from various sources often gets mixed making it difficult to validate its accuracy leading to garbage data.

Approach for utilizing Big Data:

IT departments are struggling in dealing with Big Data. There are several measures available to ease this problem. The cloud has the bandwidth to manage monster data and hence provides a suitable environment without the need to invest in expensive huge capacity servers. XBRL is extensively used as a structured data language to format data and make it computer readable. A popular technology for analyzing Big Data is Apache Hadoop(High-availability distributed object-oriented platform)