Big Data drives a recruitment spike

They are some of the fastest-growing teams in commodities firms the world over – more and more utilities and merchants are following the footsteps of the major investment banks and hedge funds and have woken up to the potential of Big Data, and they are anxious to hire the people to build their capabilities.

Whether referred to as Big Data, Artificial Intelligence, or Machine Learning or Data Science, all are terms for the latest disruptive technologies to hit the commodities industry. We are talking about the application of the newest purely statistical models, which can be used to crunch vast quantities of data, which can then feed through and significantly enhance the value of analytics.

The applications in the commodities space are many. Renewables are entirely weather-dependent, for example, and while pure statistical modelling is not the primary solution available to enhance weather forecasting, that didn’t stop IBM from launching a self-learning weather model and renewable forecasting technology to improve solar forecasts. Other uses include improved solar tracking, which could ultimately significantly enhance solar production.

Then there is the potential for the combination of machine learning and companies’ extensive data systems to simply generate more useful information, by taking the vast array of data points, reviewing it all and identifying patterns and similarities that can be learned from, in order to improve decision making.

It is in this way that Big Data companies like AutoGrid can create software that, when assimilated with large amounts of energy data, can generate automated predictions, and thereby automate the performance of grid-connected devices and monitor energy usage trends.

Earlier this year, Google’s DeepMind discussed with the UK’s National Grid the possibility of using artificial intelligence to help balance the energy supply and demand in Britain, for instance through the use of renewables after machine learning has been used to predict peaks in demand. DeepMind already has a history of helping improve efficiency – its systems have decreased Google’s data centre cooling invoice by 40%

In the oil industry, too, there are countless opportunities to capitalise of the use of Big Data to maximise drilling results and to optimise value.

In sum, the enhanced use of analytics does not have all the answers, but it does allow firms across the commodities market to improve their production techniques, enhance their trading activities, and maximise efficiencies across their businesses. Essentially, in today’s world, the more data you can access the better, because there is no such thing as being too well informed.

From a recruitment perspective, the skills that are needed to populate these new Big Data or Machine Learning teams are capabilities that did not exist in commodities five years ago, and yet businesses are committing serious effort to hiring dedicated resource. Often the talent is coming out of tier-one investment banks, or hedge funds, where highly-skilled analysts and developers have been operating for years. Others are coming out of business intelligence functions in the very largest corporations, and setting their skills to work with a whole new set of criteria.

But even those businesses are talking about a war for talent. Ken Griffin, founder and CEO of hedge fund Citadel, recently spoke to CNBC about the problems his business is facing finding talented individuals with the problem-solving skills necessary to maximise the use of Big Data, while competing with the likes of Facebook and Amazon.

When we speak to Heads of Technology from energy and commodities trading clients, they say they are being encouraged by their business leaders to focus on either creating these new data-driven teams, or on growing out their existing capabilities. Early signs in Q1 2018 have shown that hiring is down in areas supporting E/CTRM, while there has been a sharp incline across hiring in Machine Learning and Big Data compared to previous years.