WO2022222822A1

WO2022222822A1 - Method and device for identifying and positioning abelmoschus manihot on basis of cameras placed in non-parallel manner

Info

Publication number: WO2022222822A1
Application number: PCT/CN2022/086576
Authority: WO
Inventors: 桑一男; 巩卓成; 徐增莱; 汪琼; 唐海涛; 葛海涛
Original assignee: 桑一男; 江苏苏中药业研究院有限公司; 巩卓成; 江苏省中国科学院植物研究所
Priority date: 2021-04-21
Filing date: 2022-04-13
Publication date: 2022-10-27
Also published as: CN112949660A

Abstract

Provided in the present invention is a method for identifying the maturity of Abelmoschus manihot and positioning Abelmoschus manihot on the basis of cameras placed in a non-parallel manner. The method comprises the following steps: two cameras placed in a non-parallel manner capturing a plant from different directions, and marking, as flowers, color blocks in the two cameras, the colors of the color blocks both reaching a preset threshold value; binarizing each image that is marked as a flower, and then performing edge extraction, and then calculating the variance of the distance from each pixel point of a sideline of a closed graph to the geometric center point of the closed graph; determining the shape of the flower according to the magnitude of the variance; and parsing the position of a mature flower. Further provided in the present invention is a device for identifying the maturity of Abelmoschus manihot and positioning and detecting Abelmoschus manihot on the basis of cameras placed in a non-parallel manner. By means of the method and the device of the present invention, cameras placed in a non-parallel manner are used to collect an image of a target crop, and the three-dimensional coordinates of a flower are resolved according to the spatial coordinates, which are respectively obtained by two cameras, of the flower in two dimensions. In addition, the morphology and color information of the flower itself are used as the basis for determining whether the flower needs to be treated.

Description

A method and device for identifying and locating hollyhocks based on non-parallel cameras

technical field

The invention provides a low-cost target identification method and equipment for hollyhocks based on orthogonally placed cameras, belonging to the field of agriculture.

Background technique

Yellow Hollyhock [Abelmoschus manihot (L.) Medic.] is an annual or perennial erect herb of the Malvaceae family. Due to various factors such as blind excavation and deterioration of the ecological environment, the wild resources of hollyhocks in my country were on the verge of extinction in the 1980s. At present, the artificial cultivation of hollyhock is mainly distributed in Jiangsu, Anhui, Shandong, Hainan, Guangxi and other regions. The whole body of yellow hollyhock is a treasure. The leaves, tender pods and flowers of yellow hollyhock are rich in various vitamins and nutrients such as iron, calcium, crude protein, etc., which can prevent hardening of arteries, hair loss, prolong puberty, and enhance visceral function. The seed oil yield is 25% to 32%, and the extracted oil is clear and light yellow with strong antioxidant properties. It is a new type of edible oil crop. As a medicinal material, hollyhocks have been included in the "Chinese Pharmacopoeia".

With the advancement of large-scale agricultural planting and industrialization, the traditional manual picking technology is far from meeting the needs of modern agriculture. Robot technology has begun to be used in the agricultural field. However, at present, harvesting robots are mainly used on crops with clear goals such as apples and oranges. In order to ensure that these crops are not damaged after transportation, they often do not need to be picked when they are mature. For example, the patent CN 103279762 B provides a natural environment. A method for judging common growth patterns of fruits. Patent CN 101828469 B provides a binocular visual information acquisition device for a cucumber picking robot, which uses two cameras placed in parallel to perform binocular recognition, and obtains two sets of images with similar contents but with slight differences in shooting angles. Then, complex processing is performed on the obtained image to obtain target position information including depth information (ie, the distance information of the target from the position of the camera). When using parallel cameras for binocular recognition, the depth information of the target can only be obtained through coordinate transformation through the pixel difference of the obtained image. In order to achieve higher positioning accuracy, it is necessary to improve the resolution of the camera image and Perform multiple positioning, this solution requires high processing power of the computer and is susceptible to interference. The problem is that the hardware cost increases, which is not conducive to low-cost large-scale promotion.

For flowers, especially for hollyhocks, which are used as medicines, the maturity will affect the content of flavonoids, the active ingredient. Therefore, when picking hollyhocks by robots, it is particularly important to judge their maturity. Patent CN111723863A discloses a method, device, computer equipment and storage medium for identifying and obtaining the position of fruit tree flowers. The method inputs the partial image of the flower cluster into the flower detection model to obtain the flower types and types of all flowers in the partial image of the flower cluster. Corresponding flower positions, the flower detection model is trained according to the flower cluster local sample image and the flower position label and flower type label in the flower cluster local sample image. This method is used to automatically identify the type and position of flowers in fruit trees to guide the robot's subsequent flower thinning operations. It involves artificial intelligence and requires a large number of images to be collected in the early stage for model training. The application requires high hardware. This method can only solve the recognition of flowers by using AI recognition, but it is difficult to locate the space, and the training model and the execution of recognition have high hardware requirements, and a large amount of training data needs to be prepared in advance.

In addition, there is also a method of using a single camera and a distance sensor in the prior art, which is a very early idea. The distance sensor is easily interfered and thus loses distance information, which cannot solve practical problems.

SUMMARY OF THE INVENTION

In order to solve the defects of the prior art, the present invention provides a low-cost target recognition device and target recognition method based on non-parallel placement cameras, using the device and method to form a hardware solution at the lowest cost possible, while achieving detection High-precision localization of targets such as hollyhock flowers.

The present invention first provides a method for identifying and locating the maturity of the hollyhock flower based on non-parallel placement of cameras, comprising the following steps:

Step S10, two non-parallel cameras shoot plants from different directions, extract all pixel points with yellow color image peak characteristics in the picture, and form a color block to be determined, and the colors in the two cameras reach the set threshold. The color blocks are marked as flowers;

Step S20: After binarizing each image marked as a flower, edge extraction is performed, and after obtaining a closed edge, the area of the closed figure enclosed by the edge is calculated, and a closed figure with a suitable area is selected and its respective geometric center points are obtained. coordinates, and then calculate the variance of the distance from each pixel on the edge of the closed graph to its geometric center point;

S30. Determine the shape of the flower according to the variance. If the variance meets the set value, mark it as a mature flower and go to step S40; if the variance does not meet the set value, adjust the angles of the two non-parallel cameras to shoot from other different directions. For the plant, repeat the operations of step S10 and subsequent steps until the camera rotates once, and after one rotation, the flower shape is still not successfully determined to meet the standard, so the subsequent operations on the flower are abandoned;

S40, using the X-Y plane position and the X-Z plane position provided by the two cameras to jointly analyze the position of the mature flower.

Preferably, in the step S10, the non-parallel placement is that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal. Wherein, one of the two cameras is preferably located above the detected hollyhock plant.

Preferably, in the step S10, the method for determining that the yellow color reaches the threshold is: extracting the information of the red channel and the green channel of each pixel of the color camera, and when the values output by the two channels are greater than the respective set thresholds at the same time, determine the Pixels are yellow.

Preferably, in the step S20, the edge extraction method is as follows: the pixels marked in the step S10 are set to black, and the unmarked pixels are set to white. When there are 8 pixels around a black pixel When a white pixel or a black pixel is at the edge of the image, the black pixel is marked as an edge. When the above operations are performed on each pixel of the image, a closed figure surrounded by several closed edges will be obtained.

In the step S20, the closed graphics with a suitable screening area are closed graphics whose area is greater than or equal to a preset threshold after excluding closed graphics with too small area and closed graphics at the edge of the image.

In the step S30, when the shape of the flower is determined according to the variance, if the variance is small, it means that the flower is close to a circle, and the flower shape seen from this angle is qualified and can be picked; if the variance is large, it means that the shape does not meet the standard. , there will be the following situations: a. The flowers are not fully blooming, growing in strips, and can not meet the picking standards, you can give up picking; b. The shooting angle is wrong, it is horn-shaped, and the picking angle is not reached, you can further rotate the camera to shoot Handling; c. The working environment is complex, and several flowers grow together, which is inconvenient to carry out the positioning and picking operation.

In the step S40, the X-Y plane refers to a plane parallel to the ground, the Z axis refers to an axis orthogonal to the X-Y plane, and the positioning of the mature flower is realized by analyzing the position.

Another aspect of the present invention provides a method for identifying and locating the color and shape of a target based on non-parallel cameras, comprising the following steps:

In step S1, the two cameras placed in non-parallel photograph the detected objects from different directions, and the pixel positions of the color image in the picture with the target peak characteristic are extracted to form the color blocks to be determined. Color blocks are marked as preliminary targets;

Step S2: After binarizing the image of each marked preliminary target, perform edge extraction, after obtaining a closed edge, calculate the area of the closed figure enclosed by the edge, select a closed figure with a suitable area, and obtain its respective geometry. The coordinates of the center point, and then calculate the variance of the distance from each pixel point on the edge of the closed graph to its geometric center point;

Step S3, determine whether the shape of the preliminary target meets the standard according to the size of the variance. If the variance meets the set value, mark it as the target and go to step S40; if the variance does not meet the set value, adjust the angles of the two non-parallel cameras. Shoot the detected object from other different directions, and repeat the operations of step S10 and the subsequent steps until the camera rotates once, and after one rotation, if it is still not successfully determined as the target object, then give up the follow-up operation on the detected object;

Step S4, using the X-Y plane position and the X-Z plane position provided by the two cameras to jointly analyze the position of the target.

Preferably, in the step S1, the non-parallel placement is that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal. Wherein, one of the two cameras is preferably located above the detected object.

Preferably, in the step S1, the method for judging that the selected color reaches the threshold is: extracting the information of two specified color channels of each pixel of the color camera, when the output values of the two channels are greater than the respective set thresholds at the same time , determine that the pixel is the selected color.

Preferably, in the step S1, the detection object and the target object have different colors, especially contrasting colors, such as red and green, yellow and green, white and other colors, etc., preferably the target object is located in the detection object. On the outside, it is further preferred that the detection object be a cone or a cone-shaped upper tip and a lower large; the preferred detection object is a plant strain, more preferably a Malvaceae plant strain, especially a hollyhock plant strain; the target object is further preferably a brocade The plant flower or fruit of the family Auraceae is more preferably a hollyhock flower.

Preferably, in the step S2, the edge extraction method is: set the marked pixels in step S1 as black, and set the unmarked pixels as white, when there are 8 pixels around a black pixel When a white pixel or a black pixel is at the edge of the image, the black pixel is marked as an edge. When the above operations are performed on each pixel of the image, a closed figure surrounded by several closed edges will be obtained.

In the step S2, the closed graphics with suitable screening area are closed graphics whose area is greater than or equal to the preset threshold after eliminating closed graphics with too small area and closed graphics at the edge of the image.

In the step S3, the X-Y plane refers to a plane parallel to the ground, the Z axis refers to an axis orthogonal to the X-Y plane, and the position of the target object and the camera can be positioned by analyzing the position.

Another aspect of the present invention provides a target object recognition and positioning device based on non-parallel placement cameras, comprising a frame (1) fixed on a mobile carrier and a rotating platform (2); on the rotating platform (2) Two non-parallel cameras are fixed; the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can be mounted on the frame (1) turn up.

Preferably, in the target object recognition and positioning device, the non-parallel placement is that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal.

Preferably, in the target object identification and positioning device, the rotating platform (2) is a gantry-type bracket or an inverted L-shape. When the rotating platform (2) is in an inverted L shape, a Y-axis camera (3) and a Z-axis camera (4) are respectively fixed on two sides of the L-shaped rotating platform (2); with position coding The motor assembly (5) of the machine is fixed on the inner side of the top edge of the frame (1). 2) Turn.

The invention also provides a device for detecting the maturity and positioning of hollyhocks based on non-parallel cameras, comprising a frame (1) fixed on a mobile carrier and a rotating platform (2); Two non-parallel cameras are fixed; the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can be mounted on the frame (1) turn up.

Preferably, in the hollyhock flower maturity and positioning detection device, the non-parallel placement is such that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal.

Preferably, in the hollyhock flower maturity and positioning detection equipment, the rotating platform (2) is a gantry-type bracket or an inverted L-shape. When the rotating platform (2) is in an inverted L shape, a Y-axis camera (3) and a Z-axis camera (4) are respectively fixed on two sides of the L-shaped rotating platform (2); with position coding The motor assembly (5) of the machine is fixed on the inner side of the top edge of the frame (1). 2) Turn.

The device provided by the present invention uses the non-parallel cameras to capture the image of the target object, calculates the three-dimensional coordinates of the target object through the two-dimensional spatial coordinates of the target object obtained by each of the two cameras, and uses the shape and color information of the target itself. As a basis for whether it needs to be processed, such as picking flowers.

The method provided by the invention can obtain the result by judging the pixel threshold value of the color image, extracting the edge after binarization, and calculating the variance of the target image, and the required calculation amount is extremely low, and even a low-resolution color camera can be combined with a single-chip microcomputer to obtain a result. Achieving the identification of targets such as hollyhock flowers greatly reduces the difficulty of program development and the cost of practical application.

The method provided by the invention can solve the problem of maturity detection and positioning of the flower position when automatically picking the flowers of hollyhock. By screening the flowers on the plant to a certain extent, the spatial position and picking angle of the flowers of the hollyhock that can be picked are determined, and the flowers are provided to the picking machine. The arm guides it to carry out automatic picking, which involves the field of agricultural robots and the field of image recognition, especially a method for judging the pre-picking maturity and spatial positioning of fruits and vegetables or economically valuable crop flowers through image recognition. The application scenarios are not limited to this.

Description of drawings

FIG. 1 is a schematic structural diagram of a low-cost target recognition device based on orthogonally placed cameras according to the present invention.

Detailed ways

The present invention will be further described below. In the present invention, the X-Y plane refers to a plane parallel to the ground, and the Z axis refers to an axis orthogonal to the X-Y plane.

Example 1

A low-cost target recognition device based on orthogonally placed cameras, as shown in Figure 1, includes a frame (1) fixed on a mobile carrier and a rotating platform (2); the rotating platform (2) is fixed with non-parallel placed Two cameras; the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can be rotated on the frame (1).

The low-cost target recognition device of the present invention based on the orthogonal placement of cameras is described below with a more specific example:

See Figure 1, the frame (1) fixed on the mobile carrier (such as a crawler trolley) is connected to the L-shaped rotating platform (2) through the motor assembly (5), so that the rotating platform (2) can be wound around the motor assembly (5) the shaft rotates. The Z-axis camera (4) is fixed on the inner side of the top edge of the rotating platform 2, and its shooting screen is parallel to the ground, and provides image information in the X-Y plane direction. The Y-axis camera (3) is fixed on the inner side of the side of the rotating platform (2), and its shooting screen is perpendicular to the ground, providing image information in the direction of the X-Z plane.

Since the flowers of the hollyhock plant with picking value are yellow, which is significantly different from the green of the plant itself, and distributed in the outer part of the plant, it is not easy to be blocked by the branches and leaves of the plant, which provides an advantageous condition for the implementation of the present invention.

After the picking robot carrier stops beside the plants according to the designed route, the motor assembly (2) fixed on the frame (1) is almost directly above the plants. At this time, the Z-axis camera (4) shoots from the top to the bottom, and the Y-axis camera (3) shoots from the side, and extracts all the pixel points with the peak characteristic of the yellow RGB image in the picture to form the color block to be discriminated.

Afterwards, the motor assembly (5) drives the rotating platform (2) fixedly connected with the Z-axis camera (4) and the Y-axis camera (3) to rotate, so that the Y-axis camera (3) can monitor the flowers on the hollyhock plants from multiple angles. images are collected.

During the rotation, the color blocks whose color reaches the threshold are marked as flowers. At the same time, after binarizing each marked flower image, the edge is extracted, and the coordinates of the geometric center point of the edge are obtained. Afterwards, the variance of the distance from each point of the edge to the geometric center point (which can be understood as the radius) is made. If the flower is more like a circle, the variance is small, which means that the flower pattern seen at this angle is qualified and can be picked. If the variance is large, it means:

a. The flowers are not in full bloom, grow in strips, and fail to meet the picking standards;

b. At this time, the shooting angle is wrong, it is horn-shaped, and the picking angle is not reached. It is necessary to wait for the rotating platform to rotate to a suitable angle for identification and then pick it by the manipulator fixed on the rotating platform.

Through the above steps, the three X-Y-Z coordinates of the current position of the flower that is judged to be able to be picked can be jointly solved by the X-Y plane position and the X-Z plane position provided by the two cameras. At this time, the rotation of the rotating platform (2) is temporarily stopped, and the manipulator fixedly connected to it will pick flowers according to the X-Y-Z coordinates provided above.

Example 2

Basically the same as Embodiment 1, the difference is only that: the rotating platform (2) is the rotating platform of the gantry type support, and the top edge and bottom of the rotating platform (2) are respectively fixed to the Y-axis camera (3), and the inner wall of one side is fixed. Z-axis camera (4); the motor assembly (5) with the position encoder is fixed on the inside of the top edge of the frame (1), and the rotating shaft of the motor assembly (5) is connected to the outside of the top edge of the rotating platform (2) for Drive the rotating platform (2) to rotate.

The above-mentioned embodiments only represent several embodiments of the present invention, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can also be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

A kind of maturity identification and positioning method based on the hollyhock flower of non-parallel placement camera, it is characterized in that: comprise the following steps:

S10. Two non-parallel cameras shoot plants from different directions, extract all pixel positions with the peak characteristic of the yellow color image in the picture, and form a color block to be discriminated. Color blocks are marked as flowers;

S20. After binarizing each image marked as a flower, perform edge extraction, and after obtaining a closed edge, calculate the area of the closed figure enclosed by the edge, select a closed figure with a suitable area, and obtain the distance between its respective geometric center points. coordinates, and then calculate the variance of the distance from each pixel point on the edge of the closed graph to its geometric center point;

S30. Determine the shape of the flower according to the variance. If the variance meets the set value, mark it as a mature flower and go to step S40; if the variance does not meet the set value, adjust the angles of the two non-parallel cameras to shoot from other different directions. Plants, repeat the operations of step S10 and subsequent steps until the camera rotates once;

S40, using the X-Y plane position and the X-Z plane position provided by the two cameras to jointly analyze the position of the mature flower.
The method according to claim 1, wherein: in the step S10, the method for determining that the yellow color reaches the threshold value is: extracting the information of the red channel and the green channel of each pixel point of the color camera, and when the values output by the two channels are When it is greater than the respective set thresholds at the same time, it is determined that the pixel is yellow.
The method according to claim 1, wherein: in the step S20, the edge extraction method is: set the pixel points marked in the step S10 as black, set the unmarked pixels as white, when one When there is a white pixel in the 8 pixels around the black pixel or the black pixel is at the edge of the image, the black pixel is marked as an edge. When the above operations are performed on each pixel of the image, a number of A closed figure enclosed by closed edges.
The method according to claim 1, wherein one of the two cameras is located above the detected hollyhock plant.
A method for identifying and locating the color and shape of a target based on a non-parallel camera, comprising the following steps:

S1. Two non-parallel cameras shoot the detection object from different directions, extract the pixel points of the color image in the picture with the target peak characteristic, and form the color block to be judged. Blocks are marked as preliminary goals;

S2. Perform edge extraction after binarizing the image of each marked preliminary target. After obtaining a closed edge, calculate the area of the closed figure enclosed by the edge, select a closed figure with a suitable area, and obtain its respective geometric center. The coordinates of the point, and then calculate the variance of the distance from each pixel point on the edge of the closed graph to its geometric center point;

S3. Determine whether the shape of the preliminary target meets the standard according to the size of the variance. If the variance meets the set value, mark it as the target and go to step S40; Shoot the detected object in other different directions, and repeat step S10 and subsequent operations until the camera rotates once;

S4, using the X-Y plane position and the X-Z plane position provided by the two cameras to jointly analyze the position of the target.
A target object recognition and positioning device based on non-parallel placement cameras, characterized in that it comprises a frame (1) fixed on a mobile carrier, and a rotating platform (2); Two cameras placed in parallel; the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can be rotated on the frame (1).
The device according to claim 6, characterized in that: the rotating platform (2) is an inverted L-shape, and a Y-axis camera ( 3) and the Z-axis camera (4); the motor assembly (5) with the position encoder is fixed on the inner side of the top edge of the frame (1), and the rotating shaft of the motor assembly (5) is connected to the inverted L-shaped rotating platform (2). ) on the outside of the top edge to drive the rotating platform (2) to rotate.
A kind of hollyhock flower maturity and positioning detection equipment based on non-parallel placement camera, it is characterized in that: comprising a frame (1) fixed on a mobile carrier, a rotating platform (2); the rotating platform (2) is fixed on the There are two non-parallel cameras; the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can be mounted on the frame (1) turn.
The device according to claim 8, characterized in that: the rotating platform (2) is an upside-down L-shaped; Y-axis cameras (3) are respectively fixed on two sides of the L-shaped rotating platform (2). and the Z-axis camera (4); the motor assembly (5) with the position encoder is fixed on the inner side of the top edge of the frame (1), and the rotating shaft of the motor assembly (5) is connected to the top of the inverted L-shaped rotating platform (2). The connection on the outside of the side is used to drive the rotating platform (2) to rotate.
The non-parallel placement described in claims 1-9 is that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal.
The method according to claim 5 and the device according to claim 6, wherein the detection object and the target object have different colors, preferably the target object is located outside the detection object, and further preferably the detection object is a cone or The overall appearance is like a cone; the detection object is preferably a plant strain, more preferably a Malvaceae plant strain, especially a hollyhock plant strain; the target object is further preferably a Malvaceae plant flower or fruit, more preferably a hollyhock flower.
The device according to claim 6 or 7, characterized in that: the rotating platform (2) is a gantry-type bracket or an inverted L-shape.