How the Met Office is preparing for an explosive growth in data
Huge increases in the amount of data produced, stored and processed means public sector organisations must think now about how they will work with external providers to create benefits for the citizen and other partner organisations.
That is the view of Charles Ewen, CIO at the Met Office, who is helping the UK’s weather service manage an exponential growth in information. Ewen joined the Met Office eight years ago and moved into the CIO position five years ago. He has spent the past few years pushing a digital transformation initiative at the organisation, which has included the procurement of three Cray XC40 supercomputers.
These systems, which are three of the world’s 50 largest supercomputers, are capable of processing more than 14,000 trillion arithmetic operations per second. The supercomputing project is expected to enable £2bn of socio-economic benefits across the UK through enhanced resilience to severe weather and related hazards. Potential benefits include better forecasting at airports, more sophisticated modelling related to flooding, improved information for energy markets, and new climate impacts research.
The Met Office, therefore, is at the leading-edge when it comes to using big data. The organisation draws on its array of human and supercomputing resources to create 3,000 tailored forecasts and briefings a day for a range of customers, including government bodies, armed forces, private sector enterprises and the public. Ewen recognises the Met Office is blazing a trail when it comes to data use:
Ewen refers to the exponential growth in data, which leads researcher IDC to predict the amount of data around the globe will grow to 163 zettabytes, each of which is a trillion gigabytes, by 2025. That total is ten times the 16.1ZB of data generated in 2016. While access to this data creates a host of opportunities, Ewen warns IT leaders in all sectors that the inexorable rise in information creates significant challenges:
In five to ten years, the amount of data starts to become so big, you can’t move anywhere – you can’t even write it to a file system because it takes too long. Then, value must be extracted in a new way. We’re already doing research and development in the science domain, which involves largely physicists and mathematicians in orchestrated programmes. But with industry, it’s really centred on server-less technologies, so things like Google Cloud Functions and AWS Lambda, where you can move problems to data as opposed to data to problems.
Ewen says this shift in approach represents a new paradigm in data use. Public sector organisations, like their private enterprise counterparts, tend to generate value by moving data from one place to another. Ewen expects a new methodology to emerge, where a range of technologies and processes help IT leaders and their organisations to process information in real time.
