3D simulation of mackerel for fishing industry

Simulation with 3D fish: a building block for fishing industry automation

19 February 2024

In a few years, fishing vessels will be equipped with cameras and robots that were developed using synthetic data and machine learning (AI). Researcher Arjan Vroegop speaks about automation in the fishing industry, game engines, Masenro 3, and how to count fish even if you can’t see them all.

Plaice fall in a jumble onto a conveyor belt along with stones, shells, and other species of fish. Flop, there is another fish falling across the computer screen. Then a shell falls from the top of the screen and another one, and then we see more fish tumbling over each other. Behind the controls of the simulation is Vision+Robotics researcher Arjan Vroegop. “We are using synthetic data to ‘train’ a robot so it can learn the difference between plaice and by-catch (unwanted fish), among other things.”

Vroegop created the simulation with his colleagues at WANDER lab, a Wageningen University & Research group focusing on virtual and augmented reality. They created 3D fish for a virtual environment. On his screen several more plaice are passing by on the conveyor belt. Vroegop: “That conveyor belt moves underneath a camera, enabling us to automatically record data. Our goal is to ‘teach’ the robot to identify the species of fish it sees. Even if they are poorly visible or hidden.”

3D simulation to build an algorithm for various fishing industry applications

It looks like you’re watching some kind of game. This not surprising, because the images were created using a game engine. The 3D models were obtained from an online library. The next step is to scan actual fish and other objects to make the simulation even more realistic. Similarly, in the simulation Vroegop is attempting to mimic the behaviour of fish falling onto a conveyor belt. “The fish have to slide over and alongside each other in the most realistic way possible, so they end up on the belt in natural way.”

Training robots in a simulated (virtual) environment is a relatively new approach. According to Vroegop, much can be discovered and gained with this approach, especially in combination with the game engines. “In the auto industry, for example, they are already training self-driving cars partly in a virtual environment.” The goal of the fish simulation is to build an algorithm that is suitable for a range of applications in the fish sector. Once you have a suitable simulation, and you know how to add new 3D models, it doesn’t matter whether you are developing an application for use on a fishing vessel or in a fish processing plant. Vroegop: “The ideal application would be to have a robot on board a fishing vessel to sort the catch and select only fish that are suitable for consumption.”

The video shows the 3D simulation where the plaice are transported on a conveyor belt, intermingled with shells and stones. The computer not only sees every object, but it also knows whether it is a plaice or a shell or stone.

Reliable catch estimates in pelagic fishery

The pelagic fishery is very different from the cutter fishery. These large trawlers have their own freezers and can spend up to six weeks at sea catching mackerel, herring, and blue whiting. After catching and sorting, the fish are frozen immediately on board and packed in cardboard boxes. Inside the ship is a complete processing plant and an industrial freezer. “That’s where you literally have room for more automation and the combination of vision and robotics,” says Vroegop, who is – once again – using a simulation to try to teach a robot how to count mackerel. Counting fish is easy when they are lying neatly side-by-side on a conveyor belt. But that’s usually not the case. They are intermingled and lying on top of each other, which makes them more difficult to count. Vroegop: “We are looking at whether we can teach the robot to estimate the total amount of fish caught, even though it only sees part of the catch.” He believes that the solution is to collect artificial data in a simulated environment. “You then look not only at the top layer of fish, where you can count and measure individual fish, but you also scan the volume.” You then know the total volume of the fish on the conveyor belt. “Based on the information you have from the visible part, you can reliably estimate the total catch. The aim is to achieve a margin of error of less than 5% in either direction.”

3D simulation of mackerel

Mackerel are intermingled on a conveyor belt, but the computer knows exactly which fish is which: this can be used to train AI. 3D scans then determine if there are multiple layers on top of each other. The image on the right shows depth: the colours indicate the distance from the camera.

Automation a priority in fishing industry

Automation is becoming more and more of a priority in the fishing industry, partly due to the growing labour shortage. One of the innovation pacesetters is the WUR project Masenro 3 (Marine Sensing & Robotics 3). This is a collection of fishery projects with detection as a common denominator. By combining cameras with AI, you can already see a lot, but not everything as yet. That’s why Vroegop is also looking at how to use other types of sensors. In this case he does not work with simulations, but with real fish and real sensors. “Using X-ray sensors, we can look for damage such as internal bleeding in the fish. We are also going to use hyperspectral or multispectral sensors to see if we can measure the fat content of fish. In the case of a mackerel, this can even be done right through the skin.”

Technology and natural variation

Arjan Vroegop is also a Vision + Robotics expert in WUR’s greenhouse horticulture unit. This is a different world than commercial fishing, but the two industries are actually faced with similar challenges. Both deal with natural products with a lot of variation: every tomato, apple, mackerel and herring is different. For Vroegop, the combination of technology and natural variation is interesting. “A robot in a car factory operates in a defined environment. It is familiar with this environment and the product it works with.” A robot that packages a salmon fillet also operates in a familiar environment, but not every salmon is the same. There is more uncertainty about the product. That uncertainty becomes greater when you send a harvesting robot into a greenhouse or have a robot take a fish off a conveyor belt. The greater the uncertainty, the more difficult the task becomes for the robot. “That ability to deal with uncertainty is the biggest challenge for robotics in the agri and food sector.”

This research is being conducted by the Vision+Robotics team at Wageningen University & Research as part of the Masenro3 (Marine Sensing and Robotics 3) project. The research project is a public-private partnership funded by Coöperatieve Redersvereniging voor de Zeevisserij UA, VCU Robotics BV, Nationaal Overleg Visafslagen (NOVA), RapiD Engineering BV, and the Top Sector Agri & Food.

Arjan Vroegop Vision Robotics

JA (Arjan) Vroegop MSc

Researcher Vision+Robotics

Contact JA (Arjan) Vroegop MSc