RU2796270C1 - Device for automated picking of tomatoes - Google Patents

Abstract

FIELD: agricultural engineering.

SUBSTANCE: invention relates to agricultural engineering, in particular, to automated devices for harvesting vegetables, namely tomatoes, based on computer vision technologies and intelligent image recognition technologies. The device consists of a mobile platform equipped with movement devices and a body, inside of which there is a control unit, a power source, whereas a remotely controlled manipulator equipped with a finger grip, a tomato recognition camera and a technical vision system is fixed outside. A container for picking tomatoes is fixed on the body and configured for placing cooling elements in it. Independent motor-wheels with omnidirectional motion fixed to the platform via a suspension are used as means of moving the platform. An on-board computer is installed in the case, which provides recognition and classification of tomatoes according to the degree of maturity using a pre-installed convolutional neural network. The suspension is a swinging trolley and includes swivel racks connected on one side to motor wheels and to swivel suspension elements and on the other side, which in turn are connected to linear actuators fixed on the sides of the platform.

EFFECT: stabilization of the body of the device in a horizontal position while increasing the maneuverability of the device.

5 cl, 5 dwg

Description

The field of technology to which the invention belongs

The invention relates to agricultural engineering, in particular, to automated devices for harvesting vegetables, namely tomatoes, based on computer vision technologies and intelligent pattern recognition technologies.

State of the art

From the prior art, technical solutions are known that provide automated harvesting of fruits. When using such devices, there is no need for sorting fruits, the autonomy of the harvesting process during daylight hours is ensured, and the labor costs of workers are reduced.

Most known robots do not have wheels that can rotate 360 (around their axis), which reduces the maneuverability of devices in conditions of limited space in greenhouses or in fields with narrow aisles. Moreover, many analogue robots move in greenhouses along heating pipes using they are like rails.Increased requirements of robots to the evenness of the surface for movement reduce the versatility of their application.Many robots use the circular Hough transform (http://wiki.technicalvision.ru/index.php/%D0%9F%D1%80%D0%B5 %D0%BE%D0%B1%D1%80%D0%B0%D0%B7%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5_%D0%A5%D0 %B0%D1%84%D0%B0_%D0%B4%D0%BB%D1%8F_%D0%BF%D0%BE%D0%B8%D1%81%D0%BA%D0%B0_%D0%BE % D0% BA% D1% 80% D1% 83% D0% B6% D0% BD% D0% BE% D1% 81% D1% 82% D0% B5% D0% B9) to search for tomatoes, although this method is faster is inferior to the sliding window method in a convolutional neural network (https://elib.bsu.by/bitstream/123456789/191723/1/108-111.pdf). cameras.

In the prior art, an apple-picking robot is known that navigates orchards using machine vision and a time-of-flight camera. The robot is equipped with a microcontroller, a CCD camera (short for charge-coupled device), that identifies apples. A three-finger manipulator with 6 degrees of freedom picks apples from trees. In order to avoid damage to the fruit and in order to increase the efficiency of collection, each grip of the manipulator is designed according to the principle of repeating the human index finger. To separate the apple from the stalk, the manipulator pulls it, moving according to the principle of a pendulum. To detect apples, the Circular Hough Transformation method is used, selected based on the rounded shape of the apples. However, apples partially hidden by foliage are not detected by this method, so Blob Analysis is used to detect them. In order not to identify the same apple each time, the fragments of which are visible to the camera, a clustering algorithm is used, which finds out that all the identified parts belong to the same fruit. [Davidson J.R., Silwal A., Hohimer C.J., Karkee M., Mo C., Zhang Q. Proof-of-concept of a robotic apple harvester // 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016 .- Pp.634-639. DOI: 10.1109/IROS.2016.7759119.]

The main disadvantage of the robot is its low maneuverability due to the lack of omnidirectional movement.

A device for collecting tomatoes is known, which sucks tomatoes with the help of a vacuum generator brought to them. The telescopic cylinder, expanding the sleeve of the vacuum generator, facilitates the entry of tomatoes through the sleeve into the air bag. When the sleeve rotates, the fruits are separated from the stems and leaves if they enter the sleeve along with the tomatoes. The robot needs 4 s to detect a tomato, 12 s to move the sleeve to the tomato, 8 s to suck the tomato, and 12 s to return the sleeve to its original position. [Wang X.N., Wu P.H., Feng Q.C., Wang G.H. Design and test of tomatoes harvesting robot // Journal of Agricultural Mechanization Research. - 2016. - Vol.38(4). - Pp.94-98].

The disadvantage of the device is that the vacuum generator sucks not only the fruits themselves, but also dust, leaves of plants as a result of the low efficiency of separating fruits from leaves by rotating the sleeve.

A device for picking tomatoes with a manipulator arm adaptive to the shape of tomatoes is known. Thanks to this feature, the robot does not damage fragile fruits during the picking process. The machine vision system uses the method of constructing chords, which is able to detect clusters of several tomatoes on one plant, fruits partially covered with leaves, and identify tomatoes in darkened images. The bowl that sucks tomatoes is not designed to completely suck them in like a vacuum cleaner, but only to attract a tomato in order to fix it, so that it would be easier for the fingers of the manipulator to grab the fruit. This bowl is placed next to the fingers of the manipulator. The tomato is pulled off the plant by tensile force as the arm moves towards the robot's container. [Peter L., Reza E., Ting K.C., Chi Y.-T., Ramalingam N., Klingman M.H., Draper C. Sensing and End-Effector for a Robotic Tomato Harvester // 2004 ASAE Annual Meeting. Ottawa, ON, Canada. - 2004. - Pp.1-12].

The fingers of the manipulator are made of ABS plastic, and, having the advantage of lightness, it is characterized by low reliability and increased fragility, especially in comparison with steel or aluminum. In this regard, the fingers of the manipulator have a high risk of damage, given the load when picking tomatoes. The robot does not have a cooling system for the container for collecting tomatoes, which can lead to spoilage.

Known robot for picking tomatoes in greenhouses, moving on 4 wheels, using automatic navigation and avoiding obstacles. To pick tomatoes, the device uses a robotic arm with 4 degrees of freedom and fingers with 1 degree of freedom. Neural networks are used as the basis of an intelligent device for the task of classifying fruits according to the degree of maturity. The implementation of the tomato recognition algorithm does not require a large amount of memory, high processor frequency, which reduces the cost of the device being developed, as well as improves the accuracy of fruit recognition. Its error in the process of recognizing tomatoes is about 14%. To search for ripe tomatoes, the device uses the normalized color difference method when acquiring grayscale images. Otcu algorithm is used for image segmentation. Fruit picking speed - 15 s / unit. with an accuracy of detection of tomatoes - 86%. [Wang L.L., Zhao B., Fan J.W., Hu X.A., Wei S., Li X.S., Wei C.F. Development of a tomato harvesting robot used in greenhouse // International Journal of Agricultural and Biological Engineering. - 2017. - Vol.10(4). - Pp.140-149]; [Wang Z.H., Xun Y., Wang Y.K., Yang Q.H. Review of smart robots for fruit and vegetable picking in agriculture // International Journal of Agricultural and Biological Engineering. - 2022. - Vol.15(1). - P. 36]. The disadvantage of the device is the lack of a cooling system in the container for storing harvested tomatoes. In this regard, there is a risk of spoilage or deterioration of the commercial quality of tomatoes, especially if harvesting takes a long time and is carried out in hot weather or if the greenhouse is overheated.

The closest in technical essence to the claimed invention is the design of a device for automated picking of tomatoes, disclosed in the publication "Design of robotic arms with a computer vision system for picking tomatoes" // Caspian journal: management and high technologies. - 2020. - No. 3 (51). - P.135-147, URL: https://www.elibrary.ru/item.asp?id=44189041. The well-known mobile robotic platform is a remotely controlled manipulator equipped with a power source, a camera and a fetal size assessment system, mounted on a turntable. The manipulator is also equipped with a specialized gripper and a technical (computer) vision system, which is an additional video camera and an ultrasonic rangefinder connected to the control unit. The video camera and rangefinder have a certain viewing angle and are designed to ensure accurate approach of the tong to the fruit located on the stem of the plant. The robot for identifying tomatoes uses a special program for determining the boundaries of tomato fruits and the coordinates of their centers on images with a resolution of 800 by 600 pixels. The color filter technique allows you to locate ripe tomatoes. The color filter parameters are set taking into account the illumination in the greenhouse and the desired degree of ripeness of the tomatoes.

The disadvantages of the prototype are expressed in the limited applications of this device - the robot operates only within greenhouses with a flat floor, since the wheels are not able to overcome bumps and bumps in the fields. The prototype does not have a system for stabilizing the position of the body, and therefore there is a high risk of shaking the body and camera, with subsequent deterioration in the quality of images of tomatoes. The container for collecting tomatoes is not cooled, which can cause a decrease in the commercial quality of the tomatoes.

Thus, the technical problem solved by the claimed invention is the need to overcome the disadvantages inherent in analogues and the prototype, by creating an automated device for picking tomatoes, characterized by high reliability and efficiency of picking tomatoes, not only in greenhouse conditions, but also in open fields with uneven ground due to the provision of omnidirectional wheel travel and the presence of a system for stabilizing the platform of the device.

Brief summary of the invention

The technical result achieved by using the invention is to ensure the stabilization of the device body in a horizontal position while increasing the maneuverability and maneuverability of the device, which contributes to more accurate recognition of tomatoes as a result of stabilizing the position of the HD camera that recognizes ripe tomatoes.

The device is also characterized by such technical advantages as:

- increasing the safety of harvested tomatoes through the introduction of a container cooling system for storing them during the collection process;

- improving the visibility of the device at night due to the use of LED-dimensions;

- more reliable capture of a tomato due to the use of a vacuum pump;

- greater accuracy of identification of ripe tomatoes due to the use of a convolutional neural network based on the YOLOv4 architecture;

- ensuring the safety of harvested tomatoes through the use of a container cooling system for their storage.

The technical result is achieved, among other things, due to the design of the suspension of the device, which ensures the horizontal position of the body in the process of moving across the field. This body position stabilization system helps to keep the HD camera still while taking photos of the tomatoes and the field, and thus ensures high quality photos, sufficient for 95% accuracy in recognizing ripe tomatoes. In analogue devices, due to the lack of such a stabilization system, photographs and videos are less clear, which negatively affects the accuracy of identifying ripe tomatoes.

The claimed technical result is achieved by the fact that in the device for automated picking of tomatoes, which is a mobile platform equipped with vehicles and a housing, inside of which there is a control unit, a power source, and a remotely controlled manipulator equipped with a finger grip, a tomato recognition camera and a system is fixed outside. technical view, according to the technical solution, a container for picking tomatoes is fixed on the body, made with the possibility of placing cooling elements in it, and independent motor-wheels with an omnidirectional stroke, fixed to the platform through a suspension, are used as a means of moving the platform, and an onboard a computer that provides recognition and classification of tomatoes according to the degree of maturity using a pre-installed convolutional neural network, while the suspension is a swinging cart and includes swivel racks connected on one side to motor wheels, and on the other hand, to swivel suspension elements that , in turn, are connected to linear actuators fixed on the sides of the platform. Dry ice or refrigerant packs can be used as cooling elements. The vision system is a video camera and an ultrasonic rangefinder connected to the control unit. The manipulator is equipped with a vacuum pump. The device is also equipped with a GPS-based navigation device.

Brief description of the drawings

The invention is illustrated by drawings, where

in fig. 1 shows a general view of the claimed device,

in fig. 2 - side view of the claimed device,

in fig. 3 - internal layout of the proposed device,

in fig. 4 - front view of the claimed device,

in fig. 5 - view of the claimed device from above.

The positions in the figures are:

1 - manipulator for capturing tomatoes;

2 - motor-wheel;

3 - rotary stand;

4 - turntable;

5 - frame;

6 - on-board computer;

7 - control device;

8 - rotary element;

9 - mount on-board computer;

10 - joystick;

11 - navigation device based on GPS;

12 - four-stroke gasoline generator;

13 - gyro sensor;

14 - linear actuator;

15 - suspension;

16 - vacuum pump;

17 - HD camera for tomato recognition;

18 - a container for collecting fruits;

19 - HD camera for terrain analysis.

Implementation of the invention

The device for providing mobility is made in the form of a platform mounted with the possibility of movement. The platform includes a frame on which a turntable 4 is fixed, designed to accommodate the manipulator 1. The frame 5, made of durable aluminum alloy, bears the load of the components of the device and the container with the harvested tomatoes. Motor-wheels are fixed on the sides of the frame on the suspension, ensuring the movement of the proposed device. The motor-wheels 2 of the device (in the prototype, six pieces were used, three on each side of the device) are connected to the rotary racks 3. The angle of rotation of the motor-wheels is regulated by the control systems of the rotary element 8. Due to the omnidirectional movement of the motor-wheels, the inventive device can change course of its movement without changing the position of its body, instead adjusting only the angle of rotation of the motor-wheels.

A vacuum pump is installed at the base of the manipulator, which controls the grip. The vacuum pump is a sphere about 10 cm in diameter made of vacuum-tight industrial rubber filled with small plastic granules. The vacuum pump seizes the tomatoes through the effect of the phase change of the state of the plastic granules when the external pressure changes. Small granules of plastic, being in a free state, are a mass resembling a liquid in physical properties, characterized by compliance, buoyancy and fluidity. By applying a certain force to the vacuum pump, for example, by pressing it against the fetus, small plastic granules become hard and durable. The elasticity of the rubber, combined with the pliability of the small plastic granules, allows the vacuum pump, when in a relaxed state, to transform into a shape corresponding to the shape of the tomato being grasped. Then a vacuum is formed in the vacuum pump until a vacuum is reached, which leads to compaction of small plastic granules and at the same time strong fixation of the captured tomato.

The manipulator is equipped with an HD camera 17 placed on its tip. The HD camera 17 provides the function of recognizing fruits and their degree of ripeness. The operation of the manipulator, the parameters of its inclination, rotation, capture of fruits are regulated by the control device 7 of the manipulator. An ultrasonic range finder (not shown) mounted on the tip of the manipulator helps the fingers of the manipulator and the vacuum pump to more accurately approach a ripe tomato. The control device 7 of the manipulator is placed inside the housing. On-board computer 6 provides control of all working units of the proposed device. The on-board computer 6 is placed on the mount 9, inside the housing. The on-board computer 6 has a graphics processor with an ultra-precise neural network necessary for the analysis of photographs of plants. In addition to the offline mode, the device can also be controlled by the operator, for example, through the joystick 10, in particular, when the offline mode is not possible due to unforeseen situations during the operation of the device. The device is oriented in space using the GPS 11 module located inside the case. The gyro sensor 13, located inside the case, controls the position of the proposed device on the ground. The height of the hull above the ground is adjusted using two linear actuators 14 of the suspension 15. Two linear actuators 14 are installed one by one on the lower part of the hull on the sides, connecting to the suspension 15. The suspension 15 is made according to the principle of a rocker cart or, in other words, a "rocker cart" (https://en.wikipedia.org/wiki/Rockerbogie). The presence of suspension arms allows the proposed device to overcome bumps, hills and uneven terrain without the risk of tipping over and/or slipping, allowing you to keep the body position in a horizontal position, which is important for the accuracy of the photos of tomatoes obtained by the HD camera. The device is also equipped with a Wi-Fi module to enable wireless connection with the operator's joystick. As a power source, the device is equipped with at least one lithium-ion battery that provides power to the device. The battery is located inside the case. A four-stroke gasoline generator 12 can be connected to the battery, designed to recharge the battery in case the charge is used up. Thanks to the gasoline generator 12, the battery will receive the charge necessary for the device to be able to drive from the field to the base for a full recharge. Manipulator fingers made of aluminum alloy have 6 degrees of freedom with a load capacity of up to 500 g, while the manipulator reaches 1.5-2 m in length. In addition to the fingers, the manipulator is equipped with a vacuum pump 16, which catches the tomato and holds it until it is sent to the harvest container. The principle of capturing objects with a vacuum pump is described at the link: https://dailytechinfo.org/robots/5549-kofeynyy-zahvat-versaball-pozvolit-robotam-manipulirovat-obektami-imeyuschimi-formu-lyuboy-slozhnosti.html. A container for storing harvested tomatoes is installed above the body. From the bottom and along the side walls from the inside, the container is lined with a refrigerant (for example, dry ice or cooling bags), thanks to which the tomatoes retain their marketable appearance and do not deteriorate in hot weather. A light sensor (not shown) can be installed on the front of the housing, which sends a signal to the control unit about the onset of darkness. In turn, the control unit sends a command to turn on the LED dimensions, which begin to glow, making the device visible in a densely overgrown field. LED dimensions are attached to the ribs (edges) on the front and back sides of the case. Since the top panel of the LED matrix is made of plexiglass from a special translucent film, the light spreads smoothly, creating aesthetic iridescent colors.

Tracking the capture of tomatoes is carried out using an additional camera installed in the manipulator to the right of the vacuum pump. An ultrasonic rangefinder mounted on the tip of the manipulator, estimating the distance from it to the tomato, helps the fingers of the manipulator and the vacuum pump to more accurately approach a ripe tomato.

The device is distinguished by the use of intelligent learning algorithms for recognizing fruits, accurately determining their coordinates, sizes, and degree of maturity, depending on the input data on tomato varieties.

The device can be additionally equipped with the following equipment: searchlight, dosimeter, gas analyzer, temperature and humidity sensors. The software part of the control unit of the proposed device is adapted to the requirements of the user.

The claimed device works as follows.

When harvesting fruits, the tomato is captured in a special vacuum pump and in three fingers of the manipulator, followed by separation from the stalk with a force of 25 N. The productivity of the device is about 220 kg / h, while there is no need for subsequent additional sorting of the harvested crop, the degree of damage to the tomatoes is insignificant.

Preliminarily, the on-board computer 6 determines the location of the device using a GPS navigation device, an inertial system and a lidar placed on the front of the case, to the right of the turntable 4. The initial coordinate is determined using the base station, then the inertial system registers the change in coordinates in space during movement. Lidar improves the positioning accuracy of the device in the field by analyzing the received data and comparing it with an existing task map. The on-board computer 6 processes information from the GPS navigation device, the inertial system and the lidar. Computer 6 then determines the location of the robot.

At the beginning of the tomato picking process, the device is placed on the field and the power source is activated. The device enters the tomato field, navigating with the help of the GPS system. The device moves on six wheels along the path set by the GPS navigation device. The HD camera, being directed towards one aisle, searches for tomato fruits, transmitting images to the on-board computer. The HD camera searches for ripe tomatoes. The camera takes pictures of the fruit and transfers the image file to the on-board computer. The neural network processes the received images of plants and fruits, while the algorithm based on the YOLOv4 architecture detects and classifies the images of tomatoes. YOLOv4 uses a "sliding window" system. The YOLOv4 architecture, dividing photos into conditional square blocks, predicts the probability of finding tomatoes for each block according to given criteria. This saves photo processing time, which makes the convolutional neural network preferable for work [Bosilj P., Aptoula E., Duckett T., Cielniak G. Transfer learning between crop types for semantic segmentation of crops versus weeds in precision agriculture // Journal of Field robotics. - 2020. - Vol.37(1). - Pp.7-19. DOI: 10.1002/rob.21869]. Thus, the classifier of the convolutional neural network based on the YOLOv4 architecture determines the ripeness of the tomato. If an unripe fruit is detected, the device continues its movement further. When a ripe fruit is detected as a result of analysis by the neural network, the on-board computer sends a corresponding signal to the control unit.

More specifically, when a tomato is detected in a photograph, the following processing steps for the image of the tomato in the photograph are initiated:

- gamma conversion (increasing the saturation of the color of the tomato);

- median filtering (used to remove noise from the tomato image);

- conversion to HSV (color model, in which color coordinates are hue, saturation and brightness);

- segmentation according to the Father algorithm (using a histogram of the distribution of pixel brightness values of a tomato bitmap with clustering);

- morphological processing (changing pixels on the image of a tomato in order to increase its clarity);

- search for the contour (selection of the borders of the tomato);

- obtaining information about the location of the tomato for transferring it to the manipulator, so that he would grab the tomato, tear it off the stalk and move it to the container for harvesting.

The control unit sends a signal to the motor-wheels so that the device drives closer to the plant with the detected ripe tomato and stops. After stopping, the control unit gives a command to start the manipulator, which starts moving towards the found ripe tomato. As soon as the manipulator reaches the tomato, the vacuum pump sucks the tomato, and the fingers of the manipulator clasp the fruit: as a result, it is grabbed by double force - fingers and vacuum pump. Further, the manipulator starts moving back and down, towards the body, and pulls the tomato along with it. The manipulator moves to a predetermined position, above the container. Under the force of tension, the stalk of the tomato is torn, and the tomato is separated from it, and the fingers of the manipulator lower the fruit, and it falls into the collection container. The cooling system prevents damage to the tomato, thanks to which the device may not return to the vegetable store until the container is completely filled with tomatoes. About 20 kg of tomatoes are placed in the container of the prototype device. As soon as one side of the row spacing in the field is passed by the device, the HD camera mount rotates to the other side of the row spacing and the HD camera starts searching for ripe tomatoes on the other side. The light sensor at the onset of darkness sends a signal to the control unit so that it gives the command to turn on the LED dimensions. LED-dimensions in the activated state make the device visible from afar, preventing other farmers from running into it and making it clear where the device is located in the field. As you drive, the suspension adapts to the terrain so that the body stays level at all times, so the HD camera doesn't change its angle. This achieves high quality photographs, without "blurring". As a result, it is easier for the neural network to recognize the degree of ripeness of tomatoes, focusing on high-quality photographs. The device's neural network classifier separates ripe tomatoes from unripe ones. Due to this, it is possible to avoid premature failure of unripe fruits. The percentage of detected tomatoes by a prototype device is 92%, unnoticed - 8%. The average time for picking tomatoes from one bush is 380 seconds. The GPS-based navigation system allows the proposed device to navigate while moving through a field, garden or greenhouse. The omnidirectional movement of six motor-wheels allows the proposed device to save space for a turn, thereby increasing its maneuverability and maneuverability.

The claimed device can move autonomously or be controlled remotely using a joystick. In this case, the operator has the ability to take photos of the fruits in real time. They are transmitted via a Wi-Fi router to the on-board computer 6. The weight of the prototype device is 400 kg with platform dimensions of 1250 - 1550 - 1750 mm (height - width - length). The weight of the manipulator is about 7,300 g, and its length is about 1.5 meters when unfolded.

Claims (5)

1. A device for automated picking of tomatoes, which is a mobile platform equipped with means of movement and a housing inside which is a control unit, a power source, and a remotely controlled manipulator equipped with a finger grip, a tomato recognition camera and a technical vision system is fixed outside, characterized in that that a container for picking tomatoes is fixed on the body, made with the possibility of placing cooling elements in it, and independent motor-wheels with omnidirectional travel, fixed to the platform through a suspension, are used as platform vehicles, and an on-board computer is installed in the body, providing recognition and classification tomatoes according to the degree of maturity using a pre-installed convolutional neural network, while the suspension is a swinging cart and includes swivel racks connected on one side to motor wheels, and on the other side to swivel suspension elements, which in turn are connected to linear actuators attached to the sides of the platform. 2. The device according to claim 1, characterized in that dry ice or cooling packs with refrigerant are used as cooling elements. 3. The device according to claim 1, characterized in that the vision system is a video camera and an ultrasonic rangefinder connected to the control unit. 4. The device according to claim 1, characterized in that the manipulator is equipped with a vacuum pump. 5. The device according to claim 1, characterized in that it is equipped with a GPS-based navigation device.

Images