🎃 Hope your Halloween is hex-tra special
In this issue, we have two Nobel prizes for AI, LLMs not reasoning, and a little book of ML metrics. I worked a bunch on multiple of my websites and now have my PhD thesis as an Epub. I’ve been pretty active on socials and we talk about the current limitations of AI in climate modelling.
Late to the Party 🎉 is about insights into real-world AI without the hype.
Hello internet,
hear ye hear ye, there’s more machine learning happening!
In this issue, we have two Nobel prizes for AI, LLMs not reasoning, and a little book of ML metrics. I worked a bunch on multiple of my websites and now have my PhD thesis as an Epub. I’ve been pretty active on socials and we talk about the current limitations of AI in climate modelling.
Let’s dive right in!
The Latest Fashion
- AI got the Nobel prize in physics and chemistry!
- Apple published a paper casting doubt on the whole “LLMs are reasoning now” argument.
- I wish I had written the Little Book of ML Metrics, honestly.
Worried these links might be sponsored? Fret no more. They’re all organic, as per my ethics.
My Current Obsession
I posted my open-source PhD thesis and besides going mildly viral, Nikolay was kind enough to let me know that some equations didn’t render. I had kinda posted and abandoned it, but this triggered my hyperfocus, so I fixed a bunch of issues. But I also made an epub for e-readers, which I think is pretty cool. It has a few errors and doesn’t have everything the PDF has, but I still think it’s pretty great.
My new skillshare course already has 100 students! Pretty stoked about this.
I have worked a bunch on optimising the Pythondeadlin.es build process and reduced the runtime from 20 minutes to two. Very proud of that one honestly.
Thing I Like
I’ve been having the most amazing naps on my bean bag.
Hot off the Press
In Case You Missed It
I think ECMWF is hiring again, because I saw a spike in traffic on my “How I got my job at ECMWF”! article.
On Socials
I got spicy about AI bros again. Apparently, it’s trendy to say: “Do humans even reason?” now…
This made me post about broad and free education being important for a working society.
I made a post about TorchExplorer today, which seems to be going mildly viral on LinkedIn. I also re-shared my open-source PhD thesis, and this one went pretty wild as well. Mastodon quite enjoyed my post about Pythondeadlines, and Threads enjoyed this post about the math behind transformers.
Python Deadlines
We don’t have any upcoming CfPs, time to chill!
I did find some new Calls for Proposals and conferences though! The PyCon Lithuania cfp was announced. PyCamp Espana and PyCon Mini Tokai were announced. It looks like PyCon Balkan never made it beyond the announcement, so I assume it was cancelled, as there are no CfP or speakers yet.
Machine Learning Insights
Last week I asked, What are the limitations of current AI models in predicting long-term climate change impacts?, and here’s the gist of it:
Artificial Intelligence (AI) models are increasingly becoming powerful tools in climate science. However, they face significant challenges when it comes to predicting long-term climate change impacts. Let's outline the fundamental limitations and challenges faced by AI in this domain.
Data-Related Challenges
Data Quality and Availability
Data-driven climate models require large, high-quality datasets for training. Generally, we would want these models to have seen a complete distribution of realistic climate scenarios, but due to the complicated nature of the weather and climate, it's very difficult to balance realism and cover the full distribution of climate scenarios.
Climate data often suffers from incompleteness, as full model runs can be pretty expensive to model. They usually run a few scenarios that could introduce bias, discussed in the next section.
Moreover, low resolution, especially for certain regions (e.g., deep oceans, remote areas), can be challenging when generating accurate models. Low resolution models tend to suffer from having to use proxies for subgrid-scale processes. Think, for example, of tornadoes as extreme events. These are highly localised but very impactful to humans; these would even be difficult to capture in kilometre-scale models. Additionally, low resolution fails to capture highly populated areas. When I talked to the climate centre in Singapore, we talked about how most weather models are essentially missing the entire country.
As climate models usually depend on historical records, inconsistency in historical records can be very difficult to assimilate for an accurate data-driven model. Usually, these implicate distribution shifts, which machine learning models don't deal well with.
There can be significant difficulty in obtaining accurate long-term data for future projections, as these models produce a massive amount of data which cannot be stored completely. Datasets like CMIP6 can address this problem in part, but all these dataset problems are just complex problems after all. https://wcrp-cmip.org/cmip6/
Bias in Model Training
As with all AI models, data-driven climate models can inherit biases from training datasets.
For example, models trained predominantly on data from well-studied regions (e.g., Europe, North America) may perform poorly in underrepresented areas, such as Africa and the Pacific Islands. As a consequence, we can observe unequal accuracy in global predictions and difficulty in anticipating localised climate change impacts.
Modelling Challenges
Uncertainty in Climate Projections
Long-term climate predictions are inherently uncertain due to complex interactions between different Earth system components (atmosphere, oceans, biosphere) and, of course, the chaotic nature of the climate system.
Additionally, minor errors in modelling can compound over time, leading to significant uncertainties in long-term projections. This is especially true for data-driven climate models, as we lack fine-grained control over the processes and potential dampening.
Complexity of Climate Systems
Climate models need to simulate highly complex processes occurring at different scales:
- Ocean currents
- Cloud formation
- Carbon cycles
AI models struggle to accurately represent these multi-scale processes and integrate them into global predictions. Climate models, for example, directly depend on feedback loops like ice melting, leading to reduced albedo and further warming, which can be challenging to model accurately over long periods.
Current approaches try different approaches, such as using different temporal step sizes for different scales of processes. An atmosphere-ocean coupled model could, for example, make six h steps for atmospheric processes and 24h steps for ocean processes.
Limited Generalization Across Scenarios
Data-driven climate models trained on current or past climate data may not generalise well to unprecedented future conditions. Models trained on historical temperature and carbon levels, for example, may struggle to predict impacts in a world with significantly higher CO₂ concentrations.
Technical and Practical Limitations
Lack of Explainability
Many AI models, especially deep neural networks, are "black boxes."This lack of transparency makes it difficult for scientists to:
- Interpret why specific predictions were made
- Trust AI-generated climate projections for policy decisions
- Understand the underlying assumptions and mechanisms in the models
These can be addressed by explainability methods. However, xAI is still in its infancy, and there is an explainability gap even with explainable models.
Computational Constraints
Running detailed climate models integrating AI with traditional physical simulations is computationally expensive.
Accurate long-term predictions require high-resolution global models with many variables (e.g., temperature, humidity, wind patterns, ocean temperatures, CO2) and simulations over decades or centuries. The data handling alone, as outlined above, is difficult. Still, the models also require significant hardware, e.g. deep learning accelerators with large amounts of RAM. These requirements can be prohibitive even to fairly decently funded research labs.
Challenges in Integrating Socioeconomic Factors
Predicting climate change impacts involves not just physical changes but also human societal responses:
- Energy consumption patterns
- Land use changes
- Adaptation strategies
Any model struggles to accurately forecast how socioeconomic factors will interact with the climate system over long periods. Human behaviour is unpredictable and subject to changes in policies, technologies, and economic conditions.
We can already see the potential impact of extreme weather events causing climate refugees, which itself changes how humans further impact the climate.
Moving Forward
Despite these limitations, machine learning continues to hold great promise for improving climate models, especially when combined with traditional physical simulations and expert knowledge. Ongoing research is exploring ways to address these limitations:
- Developing hybrid models that combine AI with physical models
- Improving data collection methods (e.g., enhanced satellite observations)
- Developing interpretable AI techniques to enhance transparency in predictions
- Integrating domain expertise with AI to improve model accuracy and relevance
Ultimately, while AI can help refine predictions and assist in understanding climate patterns, it is just one piece of the puzzle in predicting long-term climate impacts. Physical models and expert judgment will continue to play crucial roles in this endeavour.
Conclusion
As we continue developing and refining data-driven models for climate change predictions, we must be aware of these limitations. By understanding these challenges, researchers and policymakers can work towards more robust and reliable climate modelling systems that leverage the strengths of AI while compensating for its weaknesses.
Got this from a friend? Subscribe here!
Question of the Week
- What are the practical applications and limitations of reinforcement learning in real-world scenarios?
Post them on Mastodon and Tag me. I'd love to see what you come up with. Then I can include them in the next issue!
Tidbits from the Web
- The cybertruck this winter will be the cause of some really funny content
- Elden Ring, the notoriously difficult game, made to look easy
- Just horsing around
Jesper Dramsch is the creator of PythonDeadlin.es, ML.recipes, data-science-gui.de and the Latent Space Community.
I laid out my ethics including my stance on sponsorships, in case you're interested!