This year has seen seemingly exponential growth in the number of webinars happening in the geospatial industry. At UP42, we were incredibly keen to avoid adding another webinar to the busy end-of-year calendar without a clear differentiator; without a hook that provides real value to UP42 users. We believe that last Wednesday we did just that. We tapped into our network of incredible partners and an engaged user base to deliver a webinar on maritime surveillance full of constructive, technical, practical insights and walkthroughs.
A cross-section of attendees from geospatial enthusiasts to port operators tuned in to watch industry trailblazers from exactEarth, CCRi, Osir.io, and of course, UP42. Attendees learned if, how, and when to use different types of geospatial data for maritime surveillance. They learned about the benefits, limitations, and complementary nature of optical, radar, and AIS data; processing these data sets with machine learning can unlock insights into dark vessel identification, port boundaries, vessel routes, and ETA predictions.
——— Session 1 ———
Building a Port Boundary Layer Using exactEarth’s AIS Data and CCRi Machine Learning and AI
Our first session was a joint-presentation from Taylor Nicholls, Director of Data Products and Services at exactEarth, and Eric Nallon, Data Scientist at CCRi. The two speakers tackled a fascinating topic of using machine learning to develop port boundary layers for every port globally using exactEarth's AIS data. With a mix of a compelling presentation and a rather beautiful and insightful live demo, this session packed a punch.
exactEarth and AIS data
Taylor began by outlining the ability of exactEarth's AIS service to track ship movements anywhere in the world. He introduces the topic, letting attendees know that this presentation's focus was on understanding when ships arrive and depart from ports and that getting to that point requires an accurate port boundary layer.
What are the benefits of a port boundary layer?
But why is it important to understand when ships arrive or depart from ports? Taylor outlined some fascinating use cases, such as examining the effects of COVID-19 on port activity. Utilizing CCRi's port boundary layers, he was able to plot the number of ships arriving and departing for the Port of Long Beach over the past year. This visualization clearly showed a sustained slump in vessel activity starting in February 2020, when the effects of COVID-19 were beginning to be felt worldwide.
Port of Long Beach Cargo Vessel Activity Analysis
At this point, Eric took over, showing a visualization of the port boundary layer in the port of New York that was created using CCRi's clustering technique. Overlaid on the visualization was a heatmap of vessel arrivals within the port boundary. Using this data, Eric can dig into the arrival data, aggregated and split by vessel type, using AIS data.
Patterns of vessel activity
The time series plot shows an evident intraday variation at an aggregated level. But splitting it by vessel type indicates that this variation is driven almost entirely by fishing vessels that operate with a similar fishing schedule every day, other than on weekends when a reduction in activity can be observed. Using CCRi's model and exactEarth's AIS data, this type of analysis can be efficiently conducted for any port in the world automatically.
Vessel Activity in the Port of New York
CCRi's approach to port boundary layer development
CCRi takes a machine learning approach to developing port boundaries, differentiating them from the other options from this type of analysis. The first alternative is the World Point Index, which provides a simple static point geometry. The second alternative is other providers who have built manually-drawn port boundary layers. However, this process is both tedious to do and static. CCRi's port boundary layers update as the behavior of vessels inevitably evolves.
But CCRi's approach is not without complexity. Firstly, they had to handle 1.5 years of AIS data from exactEarth, which amounted to 160GB of data containing 812 million AIS records. But with kinematic filtering and computation with a RAM-heavy AWS EC2 instance, the boundary layers were computed within a couple of hours with parallelization.
The applications of a port boundary layer
The creation of the port boundary layer is just the beginning, only the foundational layer for other analysis, which Eric goes on to outline. Enabled by the port boundary layers, Eric and CCRi can determine vessel routes by identifying ships that leave one port's port boundary layer before entering another.
CCRi's vessel routes in Worldview
These routes are subsequently used for machine learning-driven ETA and next port predictions. As a vessel begins on a route, their model uses the vessel routes previously mapped with the vessel's current speed to predict its destination port and arrival time.
ETA predictions from CCRi
Live demo of CCRi's insights in Worldview
Eric ended with a live demo of their insights opened in Worldview with extremely insightful and frankly beautiful visualizations of their port layer, vessel routes, and even a one day timelapse of 1.7M AIS records.
Check out the presentation and live demo in the full session video below:
——— Session 2 ———
The Search for Dark Vessels: Fusing AIS Signals and Deep Learning-Based Ship Detection
The second session was delivered by UP42's very own Lead Data Science Engineer, Markus Müller. Before getting started with his topic, Markus introduced the UP42 platform as a "cool API for geospatial processing", turbocharged by our Python SDK.
Why are we interested in dark vessels?
Markus began his session with an introduction to the topic of dark vessels. He mentions that all ships above a specific size are obligated to send out AIS signals. Although sometimes the reasons for ships not communicating AIS signals are above board, sometimes there is a more sinister reason, such as illegal fishing. Therefore, many maritime surveillance organizations are interested in identifying these dark vessels, which can be done through geospatial analysis.
Overview of the analysis
The main process discussed in Markus's presentation was how to process satellite images with machine learning algorithms that detect ships before fusing the outputs with AIS data. If ships are detected but do not have corresponding AIS signals, they could be a candidate to be a dark vessel.
Practical demo
Markus walks us through a Jupyter Notebook that leverages the UP42 Python SDK to access the functionality of the UP42 platform. Markus initially imports the relevant libraries from the UP42 library to standard geospatial and data libraries.
He defines the areas of interest (AOIs) - in this case, Subaraya, Indonesia, and Gibraltar - before searching for SPOT 6/7 imagery through the UP42 catalog search - functionality that lets you search across a range of satellite images filtering by your requirements.
The images chosen for Markus's analysis
Markus reviews the quicklooks of the returned images and chooses an image from each AOI. These images are then processed through an analytics workflow consisting of DIMAP to GeoTIFF conversion, coordinate reference system conversion, raster tiling, and finally, an out-of-the-box ship identification processing block that automatically fuses the results of ship detection on SPOT imagery with AIS data.
Dark vessel candidates are outlined in red with AIS-transmitting ships otulined in orange
The entire workflow was run before the webinar began and took a total of 15 minutes, with the two AOIs being processed in parallel. Utilization of Matplotlib enables the overlay of the geometry of the identified ships on top of SPOT imagery, visualizing the vessels without corresponding AIS data in red with other vessels in orange.
A small disclaimer
The presentation comes to a close with a disclaimer that Markus is not implying that many ships around Surabaya and Gibraltar's ports are suspicious. Indeed, many of the vessels in the image are too small to be required to give out an AIS signal but large enough to be identified in the high-resolution SPOT imagery. Additionally, as previously mentioned, there are many other reasons why an AIS signal might not be found.
The aim of the presentation is instead to outline the principle of dark vessel detection. In reality, the analysis would occur along shipping routes, outside of ports, where dark vessels are more likely to operate.
Review Markus's session below and pull up the Jupyter Notebook to follow along yourself:
——— Session 3 ———
Why Is Synthetic Aperture Radar (SAR) Interesting for Maritime Object Verification and Identification?
Ron Hagensieker, CEO and Founder of Osir.io, brought a different perspective to the webinar, with vast experience and understanding of when, how, and why to use SAR data in the maritime industry.
The problem with optical satellite imagery
Ron began his presentation by somewhat controversially, putting SAR imagery directly against satellite imagery, outlining a fundamental limitation of using optical imagery for ship detection - the presence of clouds. This point is backed up by using time-lapses of satellite imagery of coastal areas showing significant obstructive cloud cover.
Timelapses of optical images of four coastal areas
He then takes aim directly at the Airbus ship detection algorithm used as an input for Markus's analysis. The clouds in underlying imagery weren't masked but rather were processed through "atmospheric correction". Ron points out that this is error-prone and can significantly affect the accuracy of the detection models.
The benefit of SAR data
He argues that detecting ships using SAR data products rather than satellite imagery circumvents these significant cloud coverage issues. Additionally, he argues that, despite its reputation, SAR data products are more accessible and easier to work with than most multispectral image types.
What SAR imagery is suitable?
But not all SAR imagery is created equal, Ron goes on to explain. There is, of course, a minimum and maximum range of SAR sophistication needed for ship detection. Ron delves deeply into SAR imagery's technical requirements, outlining how it is created by orbital movement along the azimuth direction, with the beams being sent out from the satellite, scattering back, and being collected by the sensor system.
Huang et al. 2018: OpenSARShip: A Dataset Dedicated to Sentinel-1 Ship Interpretation
This creates two different ground resolution types, which is vital for this type of vessel detection and for accessing data on the UP42 platform. Firstly, the resolution and range, which depends on the system's bandwidth, and secondly, the resolution in the azimuth depending on the time of illumination on the ground.
Ron delves deeper into the technical specifications of both Sentinel-1 SLC and Sentinel-1 GRD data, outlining each's ground resolution. However, to understand the chance of detection with SAR imagery, one must also understand the sensor's location relative to the ship route and the vessel size itself. Logically, the larger the vessel, the more likely it is to be detected.
Both of the Sentinel-1 data sources discussed in Ron's presentation are available on the UP42 platform, with the SLC data block benefiting from Raster Tiling processing. Moving into a live demo, the GRD data is then opened and calibrated in the ESA SNAP tool. Ron demonstrates his ship detection algorithm applied to the Sentinel-1 imagery.
A case for AI
In a section of his presentation titled, "A case for AI", Ron built upon the critical thread of his presentation, arguing that it would be beneficial to fuse SAR-driven ship detection with AIS data to detect, identify, and segment vessels.
Quoting a recent paper, Ron covers a large-scale analysis where 9000 tiles of SAR images were analyzed to detect 6015 ships, with 6003 vessels labeled as "small". These detection algorithms used the PyTorch library. However, one must keep in mind the balance between model complexity and accuracy, especially when applying the algorithms in the real world.
Parameters x mAP for ship identification algorithms
Deploying SAR-driven ship detection in production
On that note, one must review their use case before proceeding with the right approach to geospatial analysis. Ron posed the questions to our attendees, "Are you looking to conduct analysis to identify hotspots for activity or react in real-time for law enforcement functions?".
For the latter, advances in on-edge inference enable algorithms to be deployed on the satellite, meaning that you no longer need to wait for the entire satellite image to be downlinked to run your algorithm, which would prohibit you from taking timely action. However, to deploy on the edge, you need computationally and memory-efficient algorithms.
The limitations of optical and SAR analysis
Finally, Ron talks about the further limitations of this type of analysis. He poses a couple of theoretical loopholes, such as illegal fishing vessels operating solely on cloudy days or illegal vessels could install steam generators to evade the detection algorithms.
Additionally, he says SAR is not immune. Google engineers have already managed to fool LiDAR in Google's self-driving cars, and it is possible to trick SAR imagery with a similar approach. As a result, those working with SAR imagery need to be aware, just as with satellite imagery, that there may be loopholes, exceptions, and evasive strategies utilized by vessels to obscure surveillance.
Watch Ron's full presentation to make up your own mind about whether SAR or satellite imagery is best for your use case:
What’s on the horizon for UP42?
This webinar was something of a pilot for us. Will people show up? Will they appreciate the technical depth and practical nature of our excellent speakers' presentations? And we can now certainly say that our inaugural webinar was a resounding success with a cross-section of 800 registrants from across the globe interested in utilizing geospatial technology for maritime surveillance. We had more engagement and more insightful questions that we could have imagined, with way too many questions to have answered during the webinar.
As a result, we're already hard at work planning our next webinar. We'll be working with more incredible partners in the geospatial industry to bring you additional technical, practical insights into building geospatial products. Watch this space!
