Ten thousand images of plankton collected with the Plankton Imager on a research monitoring cruise on the North Sea.

Maritime research is on the cusp of a computer vision and robotics breakthrough

11 April 2024

Once every two months, we introduce you to on of our specialists. We give an insight into the person, their research and their expectations. This time, it’s marine biologist and computer scientist Jeroen Hoekendijk: “I listen to researchers and identify which problems seem similar from a computer vision perspective, so that we can tackle them efficiently in parallel.”

It was the huge six-foot dorsal fin rising out of the water that did it. Jeroen Hoekendijk, then 21, was stunned into silence. He knew that orcas were often seen along the east coast of Vancouver Island: that’s why he and his family had travelled to the Johnstone Strait in Canada. With its abundance of salmon, and the pebble beaches that serve as back scratchers, the channel is effectively a marine playground for orcas. But sitting in his kayak in a sheltered bay, waiting for the rest of the group to take to the water, the sudden appearance of the huge male orca still took Hoekendijk totally by surprise. “It really made its mark on me.” He got straight onto the phone and told his brother, who was still in the Netherlands, to sign him up for a degree in marine biology in Groningen. He managed to get his name down just before the deadline.

Automated image recognition with computer vision

While studying for his master’s degree, Hoekendijk ended up on Texel, at the Royal Netherlands Institute for Sea Research (NIOZ). After five years of research, he was awarded a PhD for his thesis: ‘Through the looking-glass: marine mammal monitoring in a changing world’. “Researchers have been taking aerial photographs of seals for years. They then count the seals manually to further their understanding of population trends. I wondered if you couldn’t do that automatically.”

Marine ecologists and biologists like to borrow from each other’s methodologies. They use programmes previously developed for other kinds of scientific research. This is because they are themselves often not pioneers of technology, Hoekendijk says. “But we do need to see those technologies in order to understand how to use them within a different context.” This is something that interests him. “As marine biologists, how can we apply the latest computer vision and robotics tools within our own bubble of ecological issues?”

Algorithm to count seals automatically

Referring to his PhD Thesis, Hoekendijk points out that seal photography takes advantage of the species’ ecology. Many of the animals can be found on the sandbanks of the Wadden Sea at certain times of the year. Specifically, during the mating season and when they moult. That’s the time to count them from the air. “If you want to achieve that within a tidal cycle, you really need an aircraft. You also want neighbouring countries to do it at the same time, to avoid double counting.”

Group of seals on the sandbanks of the Wadden Sea

Seals on the sandbanks of the Wadden Sea

For his PhD research, Hoekendijk used an algorithm that could automatically count seals. By way of comparison, if you wanted to distinguish between dogs and cats in photos, you would train the algorithm to recognise those two animals. For a picture of a dog, you’d say ‘this is a dog’, and for a cat ‘this is a cat’. If you repeat that often enough, the algorithm will automatically learn to recognise the difference between the two. “It works the same way for a seal census, where often sizeable groups of 10 to 20 or even 200 to 300 seals will be lying together on sandbanks.”

Deep learning technique regression and heatmaps

When training the programme for his research, he was able to rely on old data: the counts done in previous years. “You don’t tell the computer network where the seals are,” he says. “Instead, you show a picture with a group of seals and say how many there are.” One photo will say 26, the other 38, and so on. “After a while, a computer starts to understand which object it needs to recognise.” That deep learning technique is called regression. It doesn’t say where in the picture the seals are, but provides the number of seals as an output. “You teach the computer to just count and skip the localisation of the animals.”

If you do still want to know exactly where in the photo the seals are (“Is the algorithm counting what I hope it counts?”), you can look at where the counting network is most active, says Hoekendijk. This is done through the use of a type of heatmap. “On the heatmaps, sand is shown as darkness on the photo. Where a group of seals gathers together, you see a spot that lights up. The more seals, the stronger the light. So that lets you see where the seals are and what the network is responding to.”

Heatmap showing groups of seals gathered together

Heatmaps showing the location of seals gathered together. The more seals, the stronger the light.

Applying the same programming code to different biological problems

In the spring of 2023, just before he was awarded his PhD, Hoekendijk heard that Wageningen Marine Research (WMR) was looking for a computer scientist in Den Helder. They needed someone who could apply AI applications to biological problems. As a ‘biologist who has shifted a bit to the computer science side’, he fits that profile exactly. So he was able to start work at WMR in Den Helder straight away, and could continue to live on Texel. When he’s on the island, he loves to wander along its dunes and beaches. And of course, when something exciting turns up in the water, he’s all over it. Like when a handful of sperm whales beaches on Texel, or a walrus shuffles across the Harlingen pier. “Marine mammals aren’t my job, it’s a hobby,” he says.

Computer scientists and biologists each have their own way of thinking, says Hoekendijk. “With your computer science hat on, it’s often about exactly the same problem. You can sometimes apply the same programming code to different biological problems, even when at first glance they have nothing in common.” He points to research into how to determine the age of a fish, by way of example. That determination, he explains, is based on growth rings in otoliths, which are the fish’s ear stones. These develop a growth ring every year, just like trees. Researchers have been taking pictures of these for years. As with the seal censuses, you can use these old datasets to teach the algorithm to count growth rings. It’s done by constantly showing the computer a picture with a figure of the number of rings, until eventually it can count them on new pictures by itself. “That’s the trick we’re increasingly trying to apply. Modular thinking, and applying parts of methodologies from one study to another.”

Step up in efficiency

As Hoekendijk puts it, what he does is listen to researchers and identify which problems seem similar from a computer vision perspective, so that they can be efficiently tackled in parallel. It takes a lot of time and money to process data manually. In principle, automating the process makes it much faster, not to mention cheaper. “You can do more for the same money. The money you save can be put into gathering more data.” Imagine having to fly for 10 hours to collect enough aerial data. Then it takes 100 hours to process that data. If you can cut that time in half, you immediately have 50 hours at your disposal to do some more flying and get more data, which then allows you to answer your questions more accurately. “It’s a step up in efficiency.”

Hoekendijk doesn’t see his role as a pioneering one. On the contrary: pioneering is about discovery and experimentation. “I don’t develop too many complicated new things myself,” he says. “Instead, I look for smart existing computer vision and robotics solutions that we can use for our own research questions.” An additional advantage is that he gets to use WUR’s supercomputer Anunna for this purpose. “Using that, you can train algorithms very quickly and efficiently.”

Object detection

Maritime research is on the cusp of a computer vision and robotics breakthrough, according to Hoekendijk. So far at his new job, he has mainly been focusing on classification projects. Ideally, he wants to move on as quickly as possible to ‘object detection’, in which you do indicate specifically what you can see in an image, and where. He mentions the plankton project ‘Monitoring zooplankton phase 1’ as an example. “We used innovative methods in that project to monitor plankton. First, we collected images of individual organisms, and then we had them classified by an algorithm. A next step is to process photos showing multiple individuals and different species within a single image.” He also helps with the analysis of underwater recordings of fish and other organisms made with an underwater drone as part of the ‘Kobine ’ project. Without a doubt, there really is a lot to gain from integrating AI components from WMR research into the broader Vision + Robotics team. On one side you’ve got WUR’s Vision + Robotics specialists, and on the other you’ve got the WMR ecologists and fisheries biologists. “That collaboration has only been in place for a short time and you can already see great outcomes emerging from it.”

Bridge between computer scientists and ecologists

Take the Fully Documented Fisheries (FDF) automated catch registration project, for example, in which fish landed onto fishing vessels are placed on a conveyor belt for automatic detection and classification. The system uses a variety of sensors to capture the number of fish and their size distribution by species. Hoekendijk hopes to see more of this kind of collaboration between WMR and Vision+Robotics, so that everyone doesn’t have to keep reinventing the wheel within their own little patch. Instead, researchers can find one solution: a methodology that they can apply to different projects. “I kind of act as the bridge between the computer scientists, and the ecologists and biologists. And I want that bridge to become increasingly crowded. WMR already had a solid reputation for its ecological knowledge. By working with Vision + Robotics, we can now tackle an even wider range of projects.”

The Netherlands has two major institutes for marine research: NIOZ and WMR. “NIOZ focuses on fundamental research, while WMR is more about applying it,” says Hoekendijk. NIOZ has branches on Texel and Yerseke. WMR is based in Yerseke, IJmuiden and Den Helder and employs a few hundred people across the three sites. Every day, Hoekendijk travels back and forth to Den Helder by ferry with a dozen WMR colleagues who also live on Texel. “We’ll often be feeding the gulls and trying to photograph the rings around their legs, so we can see where they come from. No, researchers never stop.”

Header image by Lodewijk van Walraven. A collage of 10,000 images of plankton collected on 9 June 2023 at 08:00 on board Tridens in the North Sea northwest of Vlieland during a MONS North Sea monitoring cruise with the Plankton Imager. Huge blooms of sea sparkle (Noctiluca scintillans) were observed during the cruise, from the ship and in the plankton data. However, many other plankters can be seen among the sea sparks.  A total of 86 million plankton images have been collected, which are classified using machine learning to monitor plankton composition and variation along the entire Dutch coast.

Jeroen Hoekendijk Vision Robotics

JPA (Jeroen) Hoekendijk PhD

Computer scientist and Marine biologist

Categories: MarineTags: , ,

Contact Jeroen Hoekendijk, MSc