✈ Home is Germany, but even abroad there are Gerfew
In this edition, I cover a new hybrid weather AI model, a generalized graph RAG system, the limitations of generative AI, and my journey with real-time weather forecasting using AI, along with updates on my Latent Space event and some cute TikToks!
Late to the Party 🎉 is about insights into real-world AI without the hype.
Hello internet,
it’s Saturday! I had a really long workday yesterday finishing things up, but I really wanted to get this one out this week.
I think I just made the decision to a biweekly schedule on this newsletter. As all your favourite creators, we have started new projects and pushed out content like nothing during the pandemic, but now life is catching up. I feel like bi-weekly is still a nice compromise and gives me the opportunity to get back to working on writing and videos.
In this issue we have a hybrid weather AI model, generalised graph RAG, and a reminder that genAI will not build your engineering system. I talk about a cool new event we have started on the Latent Space. And I write a fairly in-depth piece about real-time weather forecasting with AI! We’ll close it out three super cute Tiktoks!
But first, fancy machine learning. Let’s dive right in!
The Latest Fashion
- Google is firing up the PR engines for their fancy hybrid AI weather model NeuralGCM.
- Generative AI is not going to build your engineering team for you was a great read.
- Microsoft open-sourced their generalised Graph RAG system. Pretty neat development.
Worried these links might be sponsored? Fret no more. They’re all organic, as per my ethics.
My Current Obsession
We started a little show & tell in the Latent Space to showcase data visualizations and ML projects. I thought this was a great idea, honestly. I think we’ll make this a regular occurrence! If you’re a maker or creator, feel free to join in, it’s free after all!
I shared about my current work at ECMWF, where we’re working on open-sourcing Anemoi, an open-source system to build graph-based data-driven weather forecasting models. Pretty intensive journey so far, and learning about Python packaging is honestly something I didn’t need that urgently in my life, but here we are.
I gave a talk at the Tübingen AI centre, which was a very nice visit to one of the famous German university towns.
Thing I Like
I upgraded my cutlery to a set from WMF. I’d consider these quite high-quality from Germany. And they have a nice variety of styles as well. Very nice to eat with and just feels very adult-y.
Hot off the Press
After a long hiatus, I wrote a blog post about AI in sustainable energy! It’s based off of the newsletter answer, but I wanted to expand that thought, so maybe it’s still for you!
In Case You Missed It
I didn’t think writing about cached properties would be one of my most visited articles, but here we are…
On Socials
I ranted about AI video models claiming they’re “physics simulators”, and it resonated with the crowd on Linkedin!
I was in Tübingen, presenting at the the AI centre of the university. People were celebrating my interview on the ECMWF website.
And I’ve been meme-ing on threads!
Python Deadlines
I found three new conferences: PyCon Wroclaw, Python Norte, and PyCon Chile
Soon, we have these two conferences closing: PyCon South Africa and Python Brasil
Machine Learning Insights
Last week I asked, What are the key challenges in applying deep learning to real-time weather prediction?, and here’s the gist of it:
Applying deep learning to real-time weather prediction presents several technical, computational, and data-related issues. Here are some of the key challenges:
Data Quality and Quantity
Volume and Variety: Weather prediction relies on vast amounts of data from various sources, including satellite images, radar data, and sensor networks. Ensuring the data is comprehensive, up-to-date, and accurately labelled is critical.
Data Preprocessing: Raw weather data is often noisy, with missing values or inconsistencies. Preprocessing these data sets to make them suitable for deep learning models is time-consuming and complex. In NWP, we call this data assimilation, which includes running the physical model several times to perform a Bayesian optimisation of the most likely state of the weather.
Data Availability: Different parts of the world have varying sensor coverage. Wealthier countries have deployed many stations and sensors or regularly send out weather balloons. At the same time, developing nations have to prioritise funds in other parts of their growing infrastructure. The fact that these countries have fewer sensors available usually increases the uncertainty of our data.
Model Complexity
Handling Spatial and Temporal Dependencies: Weather data has both spatial (geographical) and temporal (time-based) dependencies. Designing models that can effectively capture these relationships is difficult. While techniques like modern neural networks are useful, they require careful configuration to model the complex interactions in the atmosphere. These modern architectures could include masking in self-supervised learning on Transformers or building a Convolutional Neural Network with a recurrent component.
Balancing Model Accuracy and Computational Efficiency: Deep learning models can be computationally intensive, especially when they involve large datasets and complex architectures. Balancing the need for high accuracy with the constraints of real-time processing is a significant challenge. Model distillation could be used to reduce the size of a previously trained model.
Real-time Processing Requirements
Latency: Real-time weather prediction requires models to process and produce forecasts within a short time window. This means models must be not only accurate but also efficient in terms of computation time (as mentioned before).
Scalability: The ability to scale the model to handle vast amounts of data in real-time, across multiple locations, and under different weather conditions is crucial. Scaling this problem is a data engineering problem that requires building and handling this amount of data.
Integration with Traditional Models
Hybrid Models: Combining deep learning models with traditional numerical weather prediction (NWP) models may be necessary to leverage both strengths. However, integrating these models, which may have different structures and input requirements, adds to the complexity. The NeuralGCM comes to mind as a basis for this type of development.
Interpretability: Traditional models offer a degree of interpretability often lacking in deep learning models. Understanding how a deep learning model arrives at its predictions can be challenging, making it difficult to trust and validate these predictions, especially in critical scenarios.
Uncertainty and Reliability
Uncertainty Quantification: Weather prediction inherently involves uncertainty. Deep learning models must quantify and communicate this uncertainty effectively, which is not straightforward. In traditional NWP, we often use ensembles of predictions to quantify uncertainty, but this could be prohibitive with the real-time requirements.
Reliability: Ensuring that deep learning models are robust and reliable under diverse and potentially unseen weather conditions is essential. The models must generalise well from past data to new, unseen scenarios, which is particularly challenging in a field as dynamic and chaotic as meteorology. Medicanes, and generally, the increasing frequency of extreme weather events come to mind.
Resource Requirements
Computational Power: Training deep learning models for weather prediction requires pretty substantial computational resources, including high-performance GPUs and a distributed computing infrastructure. This can be costly and is usually logistically prohibitive for most institutions. Running the models then needs the preprocessing to run with 100% reliability.
Ongoing Maintenance: The models need continuous updating and retraining as new data, such as new satellite types, becomes available. Maintaining the infrastructure and expertise to keep models up-to-date is an ongoing challenge. I'm not just saying that as a scientist for machine learning in weather, but ya know, we all want job security.
Ethical and Societal Implications
Decision-Making Impact: Weather predictions can have significant societal impacts, especially during extreme weather events. Ensuring that deep learning models provide trustworthy and actionable predictions is crucial for public safety.
Bias and Fairness: Ensuring that the model's predictions are fair and unbiased across different regions and populations is also a concern, as any inaccuracies could lead to unequal impacts. This is particularly relevant with the aforementioned uncertainties introduced through the uneven sensor distribution.
I wrote about an example of nowcasting developed by Deepmind here a while ago: https://dramsch.net/articles/deepmind-nowcasting/
Addressing these challenges requires ongoing research, collaboration between meteorologists and data scientists, and the development of innovative methodologies that can bridge the gap between data-driven and traditional approaches to weather prediction.
Got this from a friend? Subscribe here!
Question of the Week
- What machine learning techniques are best suited for analyzing ice melt patterns in polar regions?
Post them on Mastodon and Tag me. I'd love to see what you come up with. Then I can include them in the next issue!
Tidbits from the Web
- Just a dog enjoying a car ride!
- Om nom nom nom nom nom nom.
- And a Rottweiler learning to signal to other dogs that he’s a friendly one.
Jesper Dramsch is the creator of PythonDeadlin.es, ML.recipes, data-science-gui.de and the Latent Space Community.
I laid out my ethics including my stance on sponsorships, in case you're interested!