🐴 Called my AI to understand horses Gemi-neigh
In this issue, we have a nice transformer explainer, a Python pickle security tool, and Everything Wrong with AI. I talk about a huge work achievement, a big move for Python Deadlines, I wrote a new blog post, and then I dive into non-hype AI for predicting ice melt patterns. We started a fun new event series in the Latent Space. And finally a bunch of cute animal videos!
Late to the Party 🎉 is about insights into real-world AI without the hype.
Hello internet,
it’s finally summer in Germany and I’ve been having a lovely time in the sun (when I’m not working). The biweekly schedule for the newsletter seems to be working out for me; I really enjoyed writing today’s issue!
In this issue, we have a nice transformer explainer, a Python pickle security tool, and Everything Wrong with AI. I talk about a huge work achievement, a big move for Python Deadlines, I wrote a new blog post, and then I dive into non-hype AI for predicting ice melt patterns. We started a fun new event series in the Latent Space. And finally a bunch of cute animal videos!
So, let’s dive right into our latest developments in machine learning!
The Latest Fashion
Transformers are driving much of modern AI, so this deep-dive explainer is pretty great.
Python pickles are a bit of a security problem, the library picklescan tries to address this!
The video Everything Wrong with AI seems to reflect the current mainstream sentiment about AI.
Worried these links might be sponsored? Fret no more. They’re all organic, as per my ethics.
My Current Obsession
I mentioned before that I’ve been working on something big at work. This isn’t an official announcement or anything, but here’s a sneak peek: ECMWF has now open-sourced the training code for Anemoi. It’s very much in beta status, but I’m incredibly proud of the work we’ve been doing here. It’s basically an ecosystem of packages to build graph-based data-driven weather forecasts. Will I make myself redundant with this? Maybe. Is it the right thing to do to ensure access to this incredible technology? Definitely! Please don’t share it too widely yet. There are still things to iron out. But it’s definitely ready to try. This is the culmination of over a year of work that I had a part in, and I’m glad to see the state it’s in now!
Unrelated to this, I also got a contract extension at ECMWF! Now I just have to sign it, and I’ll get to work on improving weather forecasts a little longer! How exciting!
I went climbing again yesterday, and I made it all the way to the top! Not bad for someone as heavy as me, honestly. I’m very proud and equally tired today. Will definitely try to go again.
I’m thinking of going for a live-aboard diving vacation later this year. If you have any experience with this or tips, I’d be very interested in hearing them!
Oh, and I’ve kinda found Sleeptoken as a new obsession for myself. If you’re enjoying heavy music, definitely give it a listen (and a second, I actually didn’t quite like them the first time around!)
Thing I Like
You know it’s my air conditioning again this week. I only run it in my small office when I’m home and it’s unbearable, but these days it’s an absolute game changer.
Hot off the Press
Look at that, I wrote a blog post again!
Why is predicting hurricanes with AI so difficult?
In Case You Missed It
Did Hydra enter the mainstream? Somehow my blog on config-driven development for machine learning is making the rounds!
On Socials
LinkedIn has been loving WeatherBench 2. On Mastodon, my post about Khuyen Tran’s book about efficient Python tricks resonated, and I said hi on science Threads, and it was an incredibly warm welcome!
I also shared some long-form writing about real-time weather forecasting inspired by our last issue here.
And I actually deleted my Twitter… 13 years. It was my first public social media. But it was time to kill it off. What a weird feeling.
Python Deadlines
I made an executive decision to remove all ads from Python Deadlines!
You’re the first to hear about it, honestly, but I believe it’s best to divest from Google.
I found a few new CfPs, mostly through official sources: PyLadiesCon and PyCascades.
We have a few deadlines coming up over the next two weeks: Python in Education Day, PyCon Sweden, Xtreme Python, Plone Conference, PyCon Ireland, PyCon South Africa, PyCon Poland
You can, of course, just find them at pythondeadlin.es.
Machine Learning Insights
Last week I asked, What machine learning techniques are best suited for analyzing ice melt patterns in polar regions?, and here’s the gist of it:
Before we dive into the machine learning techniques, let's talk about what ice melt patterns are and why they're so important in climatology.
Ice melt patterns refer to how ice in polar regions (like the Arctic and Antarctic) melts over time. This includes how much ice melts, where it melts, how quickly it melts, and how these factors change from year to year. These patterns are crucial indicators of global climate change.
Why do they matter?
Global temperature regulation: Polar ice acts like Earth's emergency blanket. As ice melts, it exposes darker land or water, which absorbs more heat instead of reflecting it back, leading to more warming and melting - a feedback loop.
Sea level rise: Melting land ice contributes directly to rising sea levels, affecting coastal communities worldwide.
Ocean circulation: Freshwater from melting ice can alter ocean currents, potentially affecting global weather patterns.
Ecosystem impacts: Changing ice patterns affect polar wildlife and can have ripple effects throughout global ecosystems.
By studying ice melt patterns, climatologists can better understand the pace and impacts of climate change, make more accurate predictions, and inform policy decisions.
Now, let's explore how one can analyze these critical patterns. This is a great example of how machine learning can intersect with Earth sciences, particularly climatology and glaciology.
Satellite Image Analysis:
One of the primary ways we study polar ice is through satellite imagery. Machine learning techniques like Convolutional Neural Networks (CNNs) are excellent for this. CNNs can be trained to identify and measure ice extent, differentiate between different types of ice, and track changes over time.
Time Series Analysis:
Ice melt patterns change over time, so time series analysis is crucial. Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM) networks, are well-suited for this task. They can process sequences of data and identify trends and patterns that might not be obvious or too labour-intensive to human observers.
Anomaly Detection:
Unusual melting events or patterns can be crucial to identify. Unsupervised learning techniques like autoencoders or isolation forests can help spot anomalies in ice melt data that might indicate significant climate events or changes.
Multi-source Data Fusion:
Ice melt is influenced by many factors - temperature, ocean currents, wind patterns, etc. Machine learning models that can integrate multiple data sources, like ensemble methods or deep learning models with multiple inputs, are valuable for creating a comprehensive picture.
Predictive Modeling:
To forecast future ice melt patterns, we often use a combination of physical climate models and machine learning. Techniques like Random Forests or Gradient Boosting Machines can be effective for making predictions based on historical data and current conditions on tabular data problems. Sometimes, the step-wise prediction of these models is not wanted by physical modellers, however.
Semantic Segmentation:
For detailed analysis of satellite imagery, semantic segmentation techniques can help identify and classify different types of ice, water, and land cover. This is particularly useful for tracking the breakup of ice shelves or the formation of melt ponds.
It's important to note that the "best" technique often depends on the specific question being asked and the data available. In practice, researchers often use a combination of these methods to get a comprehensive understanding of ice melt patterns.
Also, while machine learning can provide powerful insights, it's crucial to combine these techniques with domain expertise from glaciologists and climate scientists. The patterns and relationships identified by machine learning models need to be interpreted in the context of our physical understanding of ice dynamics and climate systems.
Got this from a friend? Subscribe here!
Data Stories
I actually had a data story to share today, but it’s already so late, I’d prefer to keep it for next time.
However!
In the Latent Space we started doing a Visual Vednesday, sharing a data visualisation or something cool we found. Would love to see you there!
Question of the Week
What are the challenges in using AI for real-time monitoring of deforestation activities?
Post them on Mastodon and Tag me. I'd love to see what you come up with. Then I can include them in the next issue!
Tidbits from the Web
What are your plans for summer?
This anteater is living its best life.
Jesper Dramsch is the creator of PythonDeadlin.es, ML.recipes, data-science-gui.de and the Latent Space Community.
I laid out my ethics including my stance on sponsorships, in case you're interested!