With the growing number of blogs, websites, social networking sites and general online search queries – enterprises, as well as individuals, are dealing with a huge amount of sensitive data on a regular basis. Therefore, big data management, though tricky, has become an integral function within an organisation; and the skills required to handle this are also looked upon carefully.
Besides the huge amount of data generation, there’s a lot of economic value that’s getting generated as well. As a result, organisations are keen to build effective skills to handle this operation. Now the question is, are these skills readily available for the enterprises?
According to experts, it’s often a challenge to locate specific skilled professionals to manage big data. “Considering that data comes in structured, semi-structured and unstructured form and in different volumes and velocity, it is important to have specific set of skills for each of these segments,” says Srinivasan Govindarajan, senior director, practice head – Enterprise Information Management System, Virtusa Corporation.
In order to identify the skills required for big data management and fight this ongoing challenge, the organisation first needs to understand how exactly to deal with big data. Some of the ways by which enterprises deal with big data are: structured data is still prominent, semi-structured data is widely used, social media data is managed actively, unstructured data is gaining importance, and web data.
While talking about the skill identification and how organisations deal with big data, Govindarajan adds, “An enterprise should be able to store, process and restore data that is required without making any value judgment.”
Big data can mean real time information or non-traditional form of media or large volume of data or social media data, but for each of these definitions, organisations still need to identify skilled resources, opine experts. Without having required skilled workforce in place, an enterprise will fail to predict customer behavior, recognise sales opportunities, segment customer base or detect fraud.
According to industry reports, looking at this huge pool of functions, thousands of jobs are going to get created over the next couple of years. The report further states that the combined market share of big data services and big data software is estimated to be around USD 10 million. The compound growth rate is also looking positive in the next 2-3 years.
Looking at the growing trend in big data and analytics, there’s an estimated requirement of 200,000 professionals with deep analytics expertise in this area; and around 1.5 million data savvy managers.
With the merger of cloud, mobility, social and analytics – a lot of opportunities will be thrown at predictive modeling and quantitative statistics.
Some of the roles which will be in demand are: chief analytics officer, data scientist, big data architect, data visualizer, data steward, quantitative analyst and machine learning expert. While some roles are purely data oriented, others are statistics oriented.
In order to simplify these roles, a typical big data team in an organisation can also be divided into six roles - architects, data project managers, domain experts, analysts, application specialist and technical administrator.
According to Govindarajan, professionals in each of these roles need to be in a position to answer the following: how to handle increasing volume of data; how to predict behaviours and derive sentiments; how to identify new patterns from the untapped data; how to correlate different sets of data and how to process unstructured data. One has to master some unique techniques in order to analyse the above concerns, such as data mining, machine learning, pattern recognition, sentimental analysis, spatial analysis, visualisation, classification, segmentation, signal processing, regression and time series among others.
Some of the technologies required to master the art of big data analysis are Hadoop, NoSQL Database, Analytic RDBMS. To conclude, Govindarajan classifies the competency groups in big data into – analytical, businesses and technical. For analytical, professionals need to know advanced analytics and predictive modeling; for businesses one must have domain knowledge and critical thinking ability; and for technical it’s crucial to have knowledge around data acquisition and management.
