🌞 Summer is the time for sun-sational puns
In this issue, we have a summary of 900 open-source AI tools and an AI blunder. We’ll also talk about tropical cyclones and AI. I have started posting on social media again, which has been well-received.
Late to the Party 🎉 is about insights into real-world AI without the hype.
Hello internet,
I took some extra days off to manage my energy, and so far, I wish it had had more of an impact. So I’m probably closer to burnout again than I thought, which is pretty bad. I’ll keep taking things slow, but I’ll have to seriously adjust and make sure I stay on this side of the line. The sun coming out is pretty nice, though!
In this issue, we have a summary of 900 open-source AI tools and an AI blunder. We’ll also talk about tropical cyclones and AI. I have started posting on social media again, which has been well-received.
Let’s dive right into some machine learning!
The Latest Fashion
- Chip Huyen shares learnings 900 open-source AI tools
- Nvidia is the most valuable company in the world…
- Microsoft’s Recall AI was delayed after massive security concerns
Worried these links might be sponsored? Fret no more. They’re all organic, as per my ethics.
My Current Obsession
Microsoft seems to be trying to force Windows 11 and their Recall AI, which I’m highly sceptical of. I’m currently considering the move to Linux. If anyone has some tips on which distribution is better, I’d be keen. I tried this DistroChooser, and it’s saying “Debian or Arch”, which seem to be wildly differing choices…
We finally published the preprint of ECMWF’s AIFS! Pretty stoked about this!
Thing I Like
As it’s getting so hot again, I can tell you I’m still a massive fan of my mobile AC unit.
Hot off the Press
In Case You Missed It
My article on buying a laptop for machine learning is still the most popular writing I have ever done.
On Socials
Should I be shitposting about genAI more?
LinkedIn really enjoyed the Streamlit implementation of prettymaps, and I can understand why!
Python Deadlines
We have a few deadlines coming up. You have to be extra quick on PyCon Korea and Pyhep.dev.
Later this week, the PyCon Malaysia and the Swiss Python Summit are closing.
Machine Learning Insights
Last week, I asked, What are the current limitations of AI in predicting severe weather events, such as hurricanes or tornadoes?, here’s the gist of it:
Predicting severe weather events like hurricanes and tornadoes is a complex task that involves challenges, many of which current AI systems still struggle with. Here are some of the primary limitations:
- Data Quality and Quantity: - Sparse Data: For tornadoes, data collection is particularly challenging because these events are relatively rare and often occur in remote areas. This scarcity of high-quality data makes it difficult for AI models to learn effectively. - Historical Data: Hurricanes have more extensive historical data, but the quality and consistency of this data can vary significantly. Older data might lack the granularity and accuracy required for modern AI models.
- Complex Dynamics: - Nonlinear Interactions: Severe weather events involve highly nonlinear interactions between atmospheric parameters. Capturing these complex dynamics accurately is a significant challenge for AI models. - Multiscale Processes: Weather systems operate on multiple scales, from small-scale turbulence to large-scale atmospheric patterns. Modelling these interactions requires sophisticated algorithms and significant computational power.
- Computational Limitations: - Resource Intensive: High-resolution models that can capture detailed atmospheric processes are computationally expensive. Running these models requires substantial computational resources, which can limit the frequency and scope of predictions. - Real-time Processing: For real-time weather prediction, AI models must process vast amounts of data quickly. Ensuring that AI systems can operate in real time without sacrificing accuracy is a persistent challenge.
- Model Interpretability: - Black Box Models: Many AI models, particularly deep learning models, operate as “black boxes” where the decision-making process is not easily interpretable. For critical applications like severe weather prediction, understanding how and why a model makes certain predictions is crucial for trust and validation. - Uncertainty Quantification: It is essential to quantify the uncertainty in predictions, but many AI models struggle with providing reliable uncertainty estimates, making it hard to assess the confidence level of a forecast.
- Integration with Traditional Methods: - Hybrid Approaches: Combining AI models with traditional numerical weather prediction (NWP) models could enhance predictions, but integrating these approaches seamlessly is challenging. Ensuring that AI models complement rather than contradict NWP models requires careful calibration and validation. - Data Assimilation: Incorporating real-time observational data into AI models to update and refine predictions dynamically (a process known as data assimilation) is complex and requires advanced techniques. - Ensemble Prediction: As the weather is so complex, classical NWP usually implements ensemble predictions, where instead of one weather forecast, you generate 50. This produces probabilities, and we can evaluate how extreme events are covered by certain members of the ensemble. The ECMWF has just published its first AIFS ensemble forecast to take advantage of these advantages.
- Adaptation to Climate Change: - Shifting Baselines: Climate change is altering the patterns and frequencies of severe weather events. AI models trained on historical data may not accurately predict future events under changing climate conditions. - Scenario Modeling: Developing AI models that can adapt to and predict weather under various climate change scenarios is an ongoing area of research but remains challenging due to the inherent uncertainty in future climate projections.
Tornado Prediction: Tornadoes form rapidly and can have very short lifespans, making them particularly hard to predict. The key indicators of tornado formation, such as supercell thunderstorms and specific wind patterns, are difficult to model accurately and in real time.
Hurricane Path and Intensity: While the path of hurricanes is relatively better predicted, forecasting their intensity remains a significant challenge. Intensity depends on various factors, including sea surface temperatures, atmospheric moisture, and wind shear, all of which must be accurately modelled and predicted. This could also be due to the resolution that current AI models work at. A lower resolution tends to under-predict the intensity of extreme events.
Conclusion
Despite these limitations, AI continues to improve and holds promise for enhancing severe weather predictions. Ongoing advancements in machine learning algorithms, increased availability of high-quality data, and better integration with traditional forecasting methods will likely overcome many of these challenges in the future. However, achieving reliable and accurate predictions for severe weather events remains a complex task that requires continued research and development.
Got this from a friend? Subscribe here!
Question of the Week
- What’s the role of AI in achieving sustainable energy solutions?
Post them on Mastodon and Tag me. I’d love to see what you come up with. Then, I can include them in the next issue!
Tidbits from the Web
- Neurodivergent summer affirmations
- Have you seen this guy at a rave?
- Ok, Byeeee
Jesper Dramsch is the creator of PythonDeadlin.es, ML.recipes, data-science-gui.de and the Latent Space Community.
I laid out my ethics including my stance on sponsorships, in case you're interested!