Met Office supercomputer upgrade powers billions more daily weather observations
The world’s foremost meteorological organisation processes 14,000 trillion algorithmic operations per second. But that’s no longer enough. We step inside a major overhaul of groundbreaking IT infrastructure to learn how you improve one of the most powerful computer systems on the planet.
The Met Office, synonymous with providing the UK public with weather forecasts over the years, is in fact, so much more than that. Founded in 1854, the organisation has pioneered the science of meteorology and its application, earning a reputation for continually pushing the boundaries of technology and innovation to meet demands of the today and future.
In addition to being the nation’s weather and climate agency, it provides aviation meteorology forecasting for no less than two thirds of all flights across the globe daily. If a flight needs to travel from New York to London, it can’t take off without a forecast from the Met Office.
Furthermore, it collects and analyses global climate and scientific data for energy providers, renewables operators for wind farm placements, utilities, and helps government, businesses, emergency responders and the public make informed decisions connected with weather.
The challenge
None of the work which the Met Office does can be achieved without High Performance Computing [HPC]. The organisation’s supercomputer is one of the largest in the world, capable of 14,000 trillion arithmetic operations a second – which equates to 215 billion global weather observations each day.
OBS data [observations] from aircraft, ships, weather stations and outposts from far across the globe flows into the Met Office every 24 hours, amounting to billions of observations. Then there are the numerical weather predictions [NWP] running different simulations across the globe, at international, European regional, and UK-specific levels, right down to 100m stretches of road. Data harvesting at such scale is an enormous task, and had been placing a huge strain on the existing HPC. The infrastructure was coming to its end of life.
To upgrade, the Met Office needed a clearer view of HPC market trends, a greater understanding of its data archive and, crucially, how they were using it. Moreover, the Met Office was gradually running out of power, and storage space – all of which was becoming a big issue.
Effective stakeholder engagement was essential in securing the necessary buy-in from across the Met Office’s decision base, from the top down, helping with evaluations and running projects. In this way, Red Oak Consulting was able to help the Met Office get everyone on the same page, understood the requirements, and share a common, singular goal across the whole organisation.
Understanding the scale of the Met Office, the types of data, workflows and applications, Red Oak applied its intimate knowledge of the HPC market to identify the right investment. Two options were available; stay with an on-premises cluster, and have it located off-site in a data centre which could be scaled up to meet demand over the years, or think outside the box and look at the cloud. This is where Red Oak Consulting came in and advised on how to look at things differently.
‘We’ve been involved across the full spectrum with the Met Office with horizon scanning, and helping them pick the technologies that were going to be coming through for their next procurement. Likewise, engaging with vendors and helping with procurement activity,’ says Owen Thomas of Red Oak Consulting. ‘And it’s also about data management, how you archive data and manage its growth, given they produce maybe hundreds of terabytes every day.
‘So, it’s about a lot more than just predicting the weather, and we worked with them to help liaise with the Government on socio-economic benefits, food security, energy security and predicting the impact of climate change over the next 10, 20 or 50 years,’ he continues. ‘The Met Office is one of these truly competent organisations around HPC, and has a very long pedigree of competency.’
The Solution
Red Oak’s senior consultancy team worked alongside the Met Office, including principal fellow for technology, Richard Lawrence, to help them understand what transformation would mean for the business. It was vital they understood how a significant shift in direction would affect them and what potential advantages and pitfalls might exist in taking a different approach. In essence, to think differently about what they were wanting to achieve and to realise that the Met Office couldn’t just do what they’d always done, rather than just making continued financial investment in the existing paradigm.
At this point in the project, Red Oak conducted a detailed analysis of the data archive to understand its actual usage versus perceived usage. Additionally, Red Oak offered specialised resources and insights for the data archive, which stores vast amounts of weather and climate information. This strategic consultation enabled the Met Office to articulate its vision more effectively, attracting more bids and fostering innovation from potential suppliers, including cloud providers like Microsoft.
Furthermore, Red Oak’s consultants have provided specific guidance on optimising use of the Met Office’s supercomputer and efficiently delivering products to customers, leveraging niche skills and tools. Finally, to enable the Met Office to scale up to six times the computational power without having to build and provide operational support for a much bigger data centre, Red Oak enabled a smooth transition to the cloud.
Results
A ground-breaking first, with the Met Office making the strategic decision to transition to the cloud with the aim of fully implementing cloud-native HPC on the Microsoft platform. Red Oak Consulting’s efforts led to a transformative shift in strategy for the Met Office. As a result, the agency awarded the cloud contract to Microsoft, making it the first UK Government organisation, and one of the pioneers globally, to adopt a cloud-based HPC system for weather and climate modelling and forecasting.
The new system, which is due to go live later this year, will provide the Met Office with far greater flexibility, scalability, resilience and security, as well as access to Microsoft’s cloud resources and innovation. HPC is a critical component for the Met Office. Without it the ability to draw down and analyse the data could not function. For more than 15 years Red Oak Consulting has supported the Met Office with its HPC stability.
As Richard Lawrence, principal fellow for platforms at the Met Office, explains: ‘Red Oak has been responsible for the single biggest change in our strategic direction. Over the past five years, they spotted a trend in the industry, that wasn’t very widespread. At that point they saw a shift in HPC from being traditional on-premises provision to being more in the in the cloud. Red Oak is Probably the only company really that has the specialisation in this space.’
More features:
Opinion: Solving the school attendance crisis with a supportive culture
Riot gear: English unrest looks to tech while blaming digital
Image: NOAA
Leave a Reply