🌦 Coming up with weather puns is a breeze
Late to the Party 🎉 is about insights into real-world AI without the hype.
Hello internet,
the last days of summer are here. I hope you’re reading this at a pool! Let’s enjoy some machine learning as a treat!
The Latest Fashion
- Machine learning models are transforming weather forecasting
- Ever wanted to integrate AI into your pandas dataframe?
- They built a GPT Research Agent, and I have to say I’m pretty intrigued!
Got this from a friend? Subscribe here!
My Current Obsession
All the hard work in the last months has finally paid off!
You can now run four major machine learning-based weather prediction models from the comfort of your own GPU. Two Nvidia FourCastNet models, Huawei’s PanguWeather and Google Deepmind’s Graphcast! We made an announcement blog post on the ECMWF website. What’s more, is that you can view the forecasts on our website, they run operationally alongside our physical models. These models are lower resolution than the physical model, which changes how they look a bit. But you can see how these models that are started from the ECMWF initial conditions can predict the hurricane forming over the Atlantic off the coast of North America and see it evolving. The program to run this is called ai-models, and if you run it with data from the Climate Data Store, it’s free and easy for you to set up an account.
I am incredibly proud of our work. Check it out and share it with folks who always wanted to run their own weather forecast!
Also, let me know if you would like a video about how to use it!
Earlier this week, I was also at the “Large-scale deep learning for the Earth system” workshop, and I genuinely enjoyed it! The big labs that published data-driven weather forecasting models all gave talks. We had a strong presence from Google, who sent Stephan Rasp to present WeatherBench 2, the second iteration of a machine learning benchmark for weather prediction. The website is really nice to browse, actually! Und Stephan Hoyer presented their work on coupling a differentiable dynamic core (physical model) with learnt physics to predict the weather and, more importantly, climate. That talk was also fascinating! My old boss got to round off the two days by basically saying, “All the cool models you just saw? Yeah, you can run those in AI models and view them on our website”—very satisfying and full circle.
Unfortunately, I did not feel comfortable attending social events, since I suspect a new wave of Covid is making its rounds. Currently, five people in my surroundings caught the virus despite being vaccinated, which had me worrying.
Also, I’m starting to set up a homeserver. It’s sitting right next to me in a box right now. Very curious how this goes. Happy about recommendations there as well for some self-hosting goodness.
Thing I Like
Here’s the thing. I ordered a bunch of IKEA, and I bought two rolling carts. One for my filming gear so it doesn’t fly around and one for the closet for my cleaning supplies. If you’re neurodivergent and struggle with cleaning, the rolling cart is a game changer. Imagine having your sprays, sponges, and even a bucket and gloves ready right next to you while scrubbing away. Easy to re-adjust and move around. Also, since it has three levels, it has SO much space. I also put the laundry detergent on there because it fits and is out of the way that way. I didn’t even know I had two dust swiffers. That’s how good it was to sort them on the shelves. Highly recommend it.
Hot off the Press
I wrote about the struggle of training deep learning models and being deeply impatient and optimising your ML training. Then I wrote a long-form piece about making machine learning more reproducible to avoid the catastrophe of having your paper redacted (or worse).
In Case You Missed It
My post about Python cached properties is still going strong!
On Socials
I also shared my speed-up tips for ML training and a little spice about looking for the right kind of machine learning jobs.
Mastodon really liked my announcement of ai-models. I had a nice little conversation on LLM evaluation as well.
I reshared Quantus on Linkedin, which I shared with you 9 months ago, and they liked it.
Python Deadlines
I haven’t found any new CfPs this week, but the PyCon Chile deadline is coming up!
Machine Learning Insights
Last week I asked, What is a vector database, and why did they rise in popularity recently?, and here’s the gist of it:
Vector databases have recently blown up as a technology due to “AI”.
When we look at chatGPT, for example, the interface is text.
But computers are terrible with text. This newsletter gets dissolved into zeros and ones to be sent down a bunch of tubes to land in your inbox.
That’s a lot of zeros and ones to get across this information.
When we deal with billion-parameter large-language models, these use embeddings of sentences that are a little bit more efficient than storing every single character.
That means we can store these sentences efficiently as a vector in high-dimensional space.
So it depends on how we embed our data (doesn’t it always come down to the loss we use in the end?)
And then the cool thing is that we can search these databases with standard similarity measures to find texts that are close.
One example I read about recently is the Viberary projects by Vicki Boykis.
This is the idea that you can train embeddings to get book recommendations based on vibes!
Data Stories
When we train machine learning models for weather, currently, we use normal MSE as a loss.
But on a map, this means the if you get extreme values “right” but just one pixel off its actual location, you get a penalty for the value where it is, as it’s too high and the missed value where it is too low.
This is also known as the “double penalty problem”.
This can lead to the machine learning forecasting system to learn that it “blurs out” the map to hedge its bets, especially over longer forecast horizons.
This blurring is terrible for extreme events in weather, those that disproportionally impact extreme events such as floods, heat waves, and tropical cyclones.
Obviously, everyone in the field is still working on this problem, but in the meantime, we can still evaluate these extreme events from data-driven forecast models.
In fact, we’ve done so in our paper on evaluating PanguWeather.
Here we can see Hurricane Lee over the Atlantic as predicted by Graphcast.
Terrifying. But it’s good we are getting increasingly good results on these models.
Source: ECMWF Charts CC-BY-NC-SA 3.0
Question of the Week
- What are the big problems applying machine learning to weather forecasts?
Post them on Mastodon and Tag me. I’d love to see what you come up with. Then I can include them in the next issue!
Job Corner
The ECMWF is hiring 5 people who touch machine learning right now!
Four positions in the core team to develop a data-driven machine learning weather forecasting:
- Machine Learning Engineer: Focus on model optimisation and parallel implementations to train large machine learning models on vast datasets. Prior experience with deep learning frameworks, model optimisation, and memory footprint improvements is essential. Background in earth-system modelling is welcomed.
- Observations and Data Assimilation Expert: Interface observations with machine learning algorithms and play a vital role in data assimilation. Exceptional interpersonal skills and expertise in using earth-system observations are highly valuable for this role.
- Machine Learning Scientist for Learning from Observations: Contribute to making future earth system predictions from observation data using deep learning frameworks. Experience in earth-system observation data and data assimilation approaches is desirable.
- Machine Learning Scientist for Precipitation: Specialise in accurate precipitation predictions with generative machine learning models. Experience with GANs, VAEs, or Diffusion approaches is advantageous, along with expertise in using neural networks for precipitation prediction.
And one on the EU project Destination Earth
You will leverage cutting-edge machine learning techniques and statistical methods to support uncertainty quantification for weather-induced extremes in the revolutionary Destination Earth (DestinE) Digital Twin. Your work will contribute to more accurate and reliable predictions, shaping the future of weather forecasting and its impact on climate understanding and resilience. If you’re a proactive and talented individual with a passion for Earth System Science and a flair for machine learning, apply now and make a meaningful difference in tackling climate challenges.
(I like to stress that these positions are, as always, written by a committee, so if identify as part of an under-represented minority, please consider applying, even if you don’t hit every single bullet point.)
Tidbits from the Web
- This “debate” about min-max-ers in D&D is very funny (and starts out unexpected)
- I watched Hank Green’s new office being made on Tiktok, Here’s a tour!
- Watch this cute video about a dog on its own dog rug
Jesper Dramsch is the creator of PythonDeadlin.es, ML.recipes, data-science-gui.de and the Latent Space Community.
I laid out my ethics including my stance on sponsorships, in case you're interested!