While a vast number of researchers, companies and startups is working on robotic picking solutions, the OnePlanet Research Center decided to take on a different challenge and approach. A challenge fed by similar societal aspects yet from a whole different magnitude. With expert farmers retiring, their knowledge and expertise is also disappearing. Capturing that knowledge from pruning experts worldwide in data, digital twins and models by digitising real orchards and crop management decisions can help secure it for future generations.
OnePlanet Research Center is a unique multidisciplinary collaboration joining the expertise and know-how of Wageningen University & Research, Radboud University, Radboudumc, and imec to incorporate the latest chip and digital technologies in advanced data-driven solutions for among others agriculture. One of the research programmes is called ‘Digital Orchard’ in which researchers work cross-domain on digital solutions to prepare existing orchards for future food production. Bas Boom, Project Lead and Computer Vision Expert at imec-OnePlanet explains how OnePlanet’s solutions can help growers, students, experts and advisors to service orchards for healthy and efficient fruits and berries.
Digitising real orchards and crop management decisions
Boom explains: “We all know that the increasing costs and limited availability of labour is threatening not only today’s, but also future food production. While that is being recognised by fellow researchers and developers working on robotic picking solutions, we see another discrepancy arising. All over the world, the average age of farmers is rapidly increasing. With expert farmers retiring, their knowledge and expertise is also disappearing. Unique and highly valuable knowledge that isn’t always being secured for future generations. We aim to capture that knowledge from pruning experts across the world in data, digital twins and models by digitising real orchards and crop management decisions.” For starters, OnePlanet focuses on developing robotic technology solutions for apple orchards. Boom continues: “Apples are a high value crop and apart from robotic picking, we haven’t seen a solution like robotic pruning for apple trees yet. Pruning also proves to be a bigger challenge as it requires other and more expertise than picking and because the pruning season can last as long as three months. Whereas apple picking – in Western Europe at least – only takes place two or three weeks a year.”
Unique farmproof sensor suite/set creates digital twin of orchards
The researchers are certainly not trying to reinvent the wheel nor to start producing machinery themselves. “What we do is to upcycle, to upgrade or combine technologies to higher technology readiness levels (TRL’s).” One such example Boom likes to mention is the sensor suite/set developed by OnePlanet to create a digital twin of existing orchards. “Many researchers and companies use affordable and in itself reliable depth cameras (based on structured light or time of flight). The drawback of that type of cameras is the insufficient depth resolution of off-the-shelf industrial cameras and the reliability under sunlight conditions in harsh outdoor farming environments. Using high-end industrial stereo vision camera might tackle this, it however needs immense computing power to relate all the captured images to their exact location. Another potential drawback is the reliability, long-term durability and insufficient resolution of off-the-shelf industrial cameras in harsh outdoor farming environments. We therefore use a unique sensor suite/set consisting of two single RGB cameras enabling stereo (and thus depth) vision, SLAM (Simultaneous Localisation and Mapping) ánd a LiDAR sensor. The cameras capture the orchard in three dimensions while we use the LiDAR and SLAM technology to georeference the captured images in the orchard. The sensor suite/set is very lightweight and not only suitable to mount on a vehicle. You can also fit it to a drone/UAV. The sensor suite/set also is highly suitable for fruit tree breeders to quickly and objectively capture tree properties of interest from their breeding labour.” To further ruggedise the LiDAR technology, OnePlanet is currently developing an improved LiDAR sensor without the rotating part in it to prevent wear and to add to its durability and reliability.
The sensor suite/set scanning apple tree rows in the orchard at WUR Randwijk
The sensor suite/set and the machine learning connected to it are, as mentioned, deployed to create digital twins of existing orchards. By scanning an orchard up to eight times per year, not only tree and fruit growth becomes visible in detail. The data are also used to keep track of the effects of pruning operations and – in future – of blossom and root thinning operations. And, coming back to the gradually fading, retiring expertise on pruning, use real trees from the digitised orchard to capture pruning decisions from those experts. “We currently have digital trees that experts can prune virtually be means of a virtual reality (VR) headset. How, where and what they prune is recorded and analysed. We are not looking to find the best pruning method or style because every individual has a different style. We want to record different styles to create models for robotic pruning and to teach and educate future pruners. Praxis proven and viable solutions are key in this respect. By demonstrating the VR solution to different schools and pruning experts, we became aware that some pruners rely on buds to determine the exact pruning position. In the future we hope to include a ‘bud-layer’ in our VR environment.” Boom feels that this sort of pruning models could be a great asset. “In addition, our technology and models will be able to predict the effects of precision pruning, whether manual, virtual or robotic, on future fruit production at a very early stage. Pruning by all means is a way to steer and stabilise future fruit production and experts know that doing it wrong, affects future fruit harvests.”
Digital orchard scanning from left to right: branch pruning (January-March), leaf initiation (March-April), leaf growth (March-May)
Digital twins will result in precision pruning robots
Bas Boom is convinced that the models created based on the digital twins of orchards and virtual pruning by experts will result in robotic pruning solutions being able to prune up to or even more than 60 percent of the overall pruning. In apple orchards for starters, but the technology can quite easily be adapted for other types of fruit orchards such as pear. “It won’t even be limited to outdoor situations and might as well be suitable for Controlled Environment Agriculture (CEA) such as greenhouses. Our main challenge is not the pruning itself, but not to make mistakes. Because as I just mentioned, pruning decisions impact future harvests. Both positive (increased yields), as well as negative (lower yields) if you make mistakes.”
While the research and trials were initially focused on automated pruning of first year branches, it proved that the dual RGB cameras weren’t optimal for that because newly grown branches have a greener and different colour. “Hyperspectral cameras can assist us in determining what branches are first year branches because first year branches can be better separated using the infrared spectrum. And we even expect that with hyperspectral cameras we will be able to detect trees and branches suffering from plant diseases and insect attacks”, Boom emphasises.
Autonomous robotic pruning to redefine pruning strategies?
The Digital Orchard research programme is currently concentrating on modelling human pruning decisions and behaviour to teach autonomous robotic pruners. Boom firmly believes that the research can lead to changing pruning strategies. “By supplementing human labour and reducing labour costs, more pruning rounds are within reach. Autonomous / robotic pruning is capable of tree specific, tailored pruning. Pruning more often and/or pruning less trees per pruning round might result in an increased and better quality fruit production.”
To accelerate and ease the initial uptake of robotic pruning, Boom also sees a hybrid human – autonomous solution being deployed. “If a human pruner interacts with a robotic pruner, the person can tell the robot to prune a certain tree or branch that wasn’t selected for pruning by our models.”
Multi-use platform also for disease, fungi, and yield prediction
The research focuses on vision and robotics related to autonomous pruning but the unique features of the sensor suite/set open up much more possibilities. Especially in combination with the continuously increasing resolutions of cameras and LiDAR sensors. “It enables scanning, also for diseases, fungi and yield prediction, precision spraying, pruning, harvesting and picking and so on. And not only in apples as mentioned, but in much more types of fruits and berries and in other food production systems as well.”
To succeed in further optimising the autonomous pruning development and proposition, the researchers would like to get in touch with more (international) pruning experts and orchard owners. “Our solution is unique because we rely on digitising real trees and pruning expertise instead of on synthetic trees. But for that, we need more trees, more orchards, more locations, more varieties, more types of fruits and more input from growers and pruners. From a technology perspective, we would like to get in touch with companies interested in the models being developed for autonomous / robotic pruning and with educational institutions interested in our VR pruning solution for teaching and education.”
Read more about Digital Orchard here.