CN111402318A

CN111402318A - Method and device for rapidly estimating yield of fruits on tree

Info

Publication number: CN111402318A
Application number: CN202010100307.4A
Authority: CN
Inventors: 钱建平; 吴文斌; 史云; 余强毅
Original assignee: Institute of Agricultural Resources and Regional Planning of CAAS
Current assignee: Institute of Agricultural Resources and Regional Planning of CAAS
Priority date: 2020-02-18
Filing date: 2020-02-18
Publication date: 2020-07-10
Anticipated expiration: 2040-02-18
Also published as: CN111402318B

Abstract

The invention discloses a method and a device for rapidly estimating the yield of fruits on a tree, wherein the method comprises the following steps: A. shooting images of the fruit trees and the reference sphere; B. performing image processing on the images of the fruit tree and the reference sphere, segmenting the outline of the fruit on the fruit tree, performing circle fitting on the outline of the fruit, and directly performing circle generation on the reference sphere according to the central point position and the diameter of the reference sphere; C. calculating the position of the fruit, and estimating the diameter of the fruit according to the position of the fruit, the position of the reference sphere, the diameter of the reference sphere and the proportional relation between the fruit fitting circle and the reference sphere; D. estimating the fruit weight according to the estimated fruit diameter, and accumulating the estimated fruit weight on the fruit tree to obtain the estimated fruit yield value on the tree. By adopting the method and the device for rapidly estimating the yield of the fruits on the trees, the yield of the fruits on the trees can be continuously and accurately estimated.

Description

Method and device for rapidly estimating yield of fruits on tree

Technical Field

The invention belongs to an agricultural automation technology, and particularly relates to an intelligent image processing and identifying technology in agricultural automation.

Background

China is a big fruit producing country, and the cultivation areas of apples, pears, peaches, litchis, kiwi fruits, persimmons and jujubes are the first place in the world. In many areas of Shandong, Shaanxi, Hebei provinces, etc., the fruit tree industry has become the local backbone industry. However, the orchard management level is relatively lagged behind and is limited by the manual management level and the natural climate, so that the annual fruit yield is greatly unstable, and further the market supply and income of fruit growers fluctuate. If the grower can estimate the apple yield by some relatively simple methods in the apple growing process and adjust the production management and sales strategies in time according to the estimated yield, the production data investment can be effectively saved, the income of the grower is improved, and the win-win effect of economic and ecological benefits is achieved.

The crop yield estimation is carried out by utilizing the image recognition technology, and the method has the advantages of easiness in obtaining images, low use cost, simplicity in operation and the like, and has become the characteristic of the invention. Such as:

the invention discloses an aerial photography system based on unmanned aerial vehicle near-ground imaging and a cotton yield estimation method (application number/patent number: 201710091713.7), and discloses the aerial photography system based on unmanned aerial vehicle near-ground imaging and the cotton yield estimation method thereof: building an unmanned aerial vehicle near-ground aerial photography system; acquiring a high-definition video stream of a cotton field by an aerial camera in the flying process; splitting by adopting a digital image processing means, and splicing the video streams to obtain a panoramic image; selecting a proper sample area, intercepting the digital image of the sample area, harvesting cotton in the area, and measuring the actual yield; and analyzing the digital image of the region and the actual cotton yield by an image training method to obtain the relationship between the cotton yield and the digital image.

The invention patent-a fruit tree yield estimation method and device (application number/patent number: 201410381889.2) discloses a fruit tree yield estimation method and device, the method comprises: s101, acquiring crown images and position data of single fruit trees in a mature period; s102, identifying the number of fruits in the image according to the color and shape characteristics of the fruits; s103, calculating the fruit yield data of the single fruit tree according to the recognition result; s104, generating a yield statistical graph according to the position data of the fruit trees and the calculated fruit yield data of the single fruit tree.

However, the existing yield estimation method based on image recognition is not suitable for fruit trees due to the characteristics of annual and annual harvest of field crops (rice, wheat, cotton, and the like) and the characteristics of population; the existing method for estimating the fruit tree yield based on image recognition has the following defects:

1) due to the multi-directional growth of the fruit trees, images are acquired from one side, so that a large error exists in the estimation of the yield of a single fruit tree; if the images are obtained from two sides in stages, on one hand, the efficiency is low, and on the other hand, the central points of the two-time obtained images are inconsistent, so that more repeated identification or missing identification is easy to cause;

2) the fruit size difference is large, and the yield is predicted only by the fruit number in the existing method, so the precision of yield estimation is not high;

3) the yield of a single fruit tree cannot be continuously obtained, and the rapid generation of a yield map is influenced.

Disclosure of Invention

In order to solve the problem that in the prior art, due to the multi-directional growth of the fruit trees, if an image is obtained from one side, a large error exists in the estimation of the yield of a single fruit tree; if the images are obtained from two sides in stages, on one hand, the efficiency is low, on the other hand, the central points of the two-time obtained images are inconsistent, so that more repeated identification or identification omission is easily caused, and in addition, the yield is predicted only through the number of fruits in the conventional method due to the large size difference of the fruits, so that the precision of yield estimation is not high.

In order to achieve the technical effect, the invention adopts the following technical scheme.

A method for rapidly estimating the yield of fruits on trees comprises the following steps:

A. shooting images of the fruit trees and the reference sphere;

B. performing image processing on the images of the fruit tree and the reference sphere, segmenting the outline of the fruit on the fruit tree, performing circle fitting on the outline of the fruit, and directly performing circle generation on the reference sphere according to the central point position and the diameter of the reference sphere;

C. calculating the position of the fruit, and estimating the diameter of the fruit according to the position of the fruit, the position of the reference sphere, the diameter of the reference sphere and the proportional relation between the fruit fitting circle and the reference sphere;

D. estimating the fruit weight according to the estimated fruit diameter, and accumulating the estimated fruit weight on the fruit tree to obtain the estimated fruit yield value on the tree.

One of the core objectives of this patent is to determine the yield of fruit on trees, and since the density of fruit is relatively constant, the fruit is roughly regarded as a sphere, and after determining the diameter of the sphere, the individual weight of the fruit can be estimated. However, the fruit size in the shot image is not the true fruit diameter, so that the reference sphere with the preset size is used as a reference in the patent, the fruit diameter is estimated according to the fruit position, the position of the reference sphere, the diameter of the reference sphere and the proportional relation between the fruit fitting circle and the reference sphere circle, and compared with the method for estimating the fruit yield on the fruit tree according to the fruit number only and without considering the fruit size in the prior art, the method has the characteristic of being more accurate.

In the patent, when the proportional relation between the fruit fitting circle and the reference sphere circle is considered, for the fruit, the actual size is unknown, and only the outline and the fitting circle can be segmented from the shot image, but for the reference sphere, the size/position of the reference sphere is a determined known quantity relative to the camera, so that a reference sphere circle can be generated by adopting direct projection to replace the sphere range in the shot image, and thus, the errors caused in the process of segmenting the reference sphere outline from the shot image and fitting the circle can be avoided, a 'standard' reference quantity is introduced, and the precision of the fruit diameter estimation on the tree is improved.

As another improvement of the patent, the image processing may be performed on the images of the fruit tree and the reference sphere, so as to segment the outline of the fruit on the fruit tree and the outline of the reference sphere in the image, and then fit a circle to the outline of the fruit on the fruit tree and the outline of the reference sphere respectively; correspondingly, the position of the fruit is calculated, and the diameter of the fruit is estimated according to the position of the fruit, the position of the reference sphere, the diameter of the reference sphere and the proportional relation of the fitting circle of the fruit and the reference sphere, so that the estimated value of the diameter of the other fruit can be obtained, and errors can be brought in the process of dividing the outline of the reference sphere from the shot image and fitting the circle relative to the estimated diameter of the fruit of the reference sphere serving as a standard reference; however, the overall deviation of the picture, such as the error caused by inaccurate parameters of the optical axis and the focal length, can be avoided, that is, if the camera and the picture have the overall deviation, the image of the fruit and the image of the reference sphere are both deformed, and then the geometric operation is performed by using the relationship between the two, so that the overall deviation is partially eliminated. Next, a confidence interval or other means, such as a weighted sum, is used to comprehensively consider the fruit diameter estimated using the reference sphere circle of the standard reference and the fruit diameter estimated using the fitted circle of the reference sphere to obtain an optimized fruit diameter estimate.

As another improvement of this patent, after the diameters of the fruits are estimated by using the above proportional relationship between the fitting circle and the reference sphere circle according to the fruit positions, the positions of the reference spheres, the diameters of the reference spheres and the fruit fitting circle, the conversion coefficients of the corresponding number of relative diameters and actual diameters can be obtained, and the average value of the conversion coefficients can be used to obtain the conversion coefficient suitable for the fruit diameter and actual fruit diameter of the fruit tree in the image under the photographic condition, so that the diameter estimation does not need to be performed separately for each fruit, but the same conversion coefficient is used, thereby simplifying the calculation and improving the efficiency of fruit tree yield estimation.

In addition, the step of taking the images of the fruit tree and the reference sphere comprises: simultaneously shooting images of fruit trees on two sides at the same horizontal height position in the opposite directions;

and, the calculating the position of the fruit in the step C includes: and (3) determining coordinates of the fruit center point in a three-dimensional space by using a binocular vision method for fitting a circle of the fruit outline in the fruit tree images on the two sides, and extracting the depth parameter of the fruit center point.

In the patent, an important parameter in the judgment of the diameter of the fruit is the position of the fruit (particularly the position of the center point of the fruit), and for this reason, it can be considered that a depth camera and other methods are used to acquire depth information (vertical distance relative to the camera) of the fruit, but the depth camera is found to be unsuitable for a scene such as a fruit tree through practical tests, because occlusion and interference of branches and leaves are very serious, and the detail area is too much, so that the accuracy of the result obtained by the depth camera is low. Therefore, the mode of imaging by the two cameras is adopted in the tree fruit yield rapid estimation device, the structure of the tree fruit yield rapid estimation device is more complex, the position of the fruit can be accurately calculated by the binocular vision mode, and compared with the scheme of utilizing a depth camera and the like, the tree fruit yield rapid estimation device has the advantages of being more accurate and higher in resolution ratio.

In addition, estimating the diameter of the fruit according to the position of the center point of the fruit, the position of the center point of the reference sphere, the diameter of the reference sphere and the proportional relationship between the fruit fitting circle and the reference sphere circle comprises:

estimating the diameter of the fruit based on the images respectively shot by the left camera and the right camera by respectively utilizing the position relation between the two cameras and the center point of the fruit, the position relation between the two cameras and the center point of a reference sphere, the diameter of the reference sphere and the proportional relation between a fitting circle of the fruit in the images shot by the cameras on the two sides and the circle of each reference sphere;

and carrying out weighted average on the fruit diameters estimated based on the images shot by the left camera and the right camera according to the image quality parameter as a weight to obtain the corrected estimated fruit diameter, wherein the image quality parameter is a parameter determined in the process of carrying out image processing on the images shot by the left camera and the right camera.

In the patent, the fruit weight and the fruit tree yield are estimated by estimating the fruit diameter, but the fruit diameter is calculated by a fitting circle of the fruit outline, and in the fitting process of the fruit outline, because the error occurs in the size of the fitting circle of the fruit outline due to the selection of parameters in the image processing process, or the definition of a picture, or the shielding condition in the picture, and the like, and the error correspondingly occurs in the estimated fruit diameter, the diameter of the fruit is estimated based on the images respectively shot by the left camera and the right camera by respectively utilizing the position relationship between the two cameras and the fruit central point, the position relationship between the two cameras and the reference sphere central point, the diameter of the reference sphere, and the proportional relationship between the fitting circle of the fruit in the images shot by the cameras at the sides and the reference sphere circle; furthermore, the fruit diameters estimated based on the images shot by the left camera and the right camera are weighted and averaged by taking the image quality parameters of the images shot by the left camera and the right camera respectively in the image processing process as weights to obtain the estimated fruit diameters after correction, so that the estimation results based on the photos shot by the left camera and the right camera can be comprehensively considered, and higher reliability is achieved.

In addition, the step A of simultaneously shooting the fruit tree images at the same horizontal height position in opposite lateral directions comprises the following steps: simultaneously shooting fruit tree images on two sides at different advancing positions respectively according to opposite two side directions and the same horizontal height position to form a plurality of groups of fruit tree images on two sides;

aiming at each group of fruit tree images on both sides, fitting a circle to the fruit outlines in the fruit tree images on both sides, determining the coordinates of the fruit center points in a three-dimensional space by using a binocular vision method, extracting the fruit center point depth parameters aiming at the group of fruit tree images on both sides, and forming a plurality of groups of fruit center point coordinate parameters, wherein each group of fruit center point coordinate parameters comprises the respective coordinate parameters of a plurality of fruit center points;

for each coordinate parameter group belonging to the same fruit center point in a plurality of groups of fruit center point coordinate parameters, carrying out weighted average according to image quality parameters in the image processing process of the bilateral fruit tree images corresponding to the group of fruit center point coordinate parameters, and obtaining the corrected coordinate parameters of the fruit center point;

and aiming at the corrected coordinate parameters of the center point of the fruit, the diameter of the fruit is estimated by combining the position of the center point of the reference sphere, the diameter of the reference sphere and the proportional relation between the fitting circle of the fruit and the circle of the reference sphere.

The estimated fruit diameter caused by the fact that the parameters of the camera or the diameter error generated in the processing process can be corrected by comprehensively considering the estimation results based on the photos taken by the left camera and the right camera, for example, the focal length of a certain camera is different from the actual situation, or the positioning position of a certain camera is not in a preset position, is larger or smaller, and the error can be reduced by comprehensively considering the estimation results based on the photos taken by the left camera and the right camera. However, the above method is not feasible for the error of inaccurate fruit center point position caused by camera parameters or position reasons, because the determination of the fruit center point position requires the matching of left and right cameras, the fitting of a circle to the fruit contour in the fruit tree images on both sides, the coordinate determination of the fruit center point in the three-dimensional space by using a binocular vision method, and the extraction of the depth parameter of the fruit center point, so that the estimation result of the photos taken by the left and right cameras cannot be used for cancellation.

Therefore, the method utilizes the steps that the images of fruit trees on two sides are shot at the same time at different advancing positions respectively according to the opposite directions on two sides and the same horizontal height position to form a plurality of groups of images of the fruit trees on two sides, the fitting circle of the fruit outline in the images of the fruit trees on two sides is aimed at each group of images of the fruit trees on two sides, the coordinate determination of the fruit center point in the three-dimensional space is carried out by utilizing a binocular vision method, the depth parameter of the fruit center point of the group of images of the fruit trees on two sides is extracted to form a plurality of groups of coordinate parameters of the fruit center point, and each group of coordinate parameters; and for each coordinate parameter group belonging to the same fruit center point in the multiple groups of fruit center point coordinate parameters, performing weighted average according to the image quality parameters as weights, and correcting the type of error in a mode of obtaining the corrected coordinate parameters of the fruit center point, wherein the image quality parameters are parameters determined in the process of performing image processing on the bilateral fruit tree images corresponding to the group of fruit center point coordinate parameters. In short, a plurality of sets of bilateral fruit tree images are shot at different positions, and the estimated fruit center point positions based on the plurality of sets of bilateral fruit tree images are comprehensively considered to obtain an optimized result, so that the accuracy is higher.

As another improvement scheme of the patent, the technical scheme that a plurality of groups of bilateral fruit tree images are shot at different positions, the optimized fruit center point position is obtained based on the estimated fruit center point positions of the plurality of groups of bilateral fruit tree images, and the estimated fruit diameter is corrected based on the photos shot by the left camera and the right camera can be comprehensively considered, so that the position error generated by shooting the photos is eliminated, the diameter error generated by shooting the photos is also eliminated, and the technical effect is better.

The image quality parameters in the process of image processing of the shot image are the number of original sampling points which form a fruit fitting circle and meet the preset edge condition in the process of circle fitting of the outline of the fruit.

For the technical scheme of shooting multiple groups of images of fruit trees on two sides at different positions and comprehensively considering the optimized fruit central point position obtained based on the fruit central point positions estimated by the multiple groups of images of fruit trees on two sides and the technical scheme of comprehensively considering the estimated fruit diameter corrected based on photos shot by left and right cameras, the fusion optimization problem of multiple groups of results (including the calculation result of the fruit positions or the estimation result of the fruit diameter) needs to be considered, and how to select the optimization means is an important factor directly influencing the accuracy of the on-tree fruit yield estimation of the patent.

The comparison shows that the main source of the yield estimation error in the method comes from the image processing process, particularly forms a contour of a fruit and obtains a fitting circle according to the contour of the fruit, and for the occasions with poor imaging quality environment or the areas with serious leaf shielding, the two links contribute more than 70% of the error of the method, therefore, when the weight is considered in the method, a weight factor is selected, namely, the image quality parameter is the number of original sampling points which form the fitting circle of the fruit and accord with the preset edge condition in the process of carrying out circle fitting on the contour of the fruit, because if the imaging quality is high or the interference factor for the image binarization process is less, more sampling points accord with the preset edge condition, namely, more sampling points accord with the real fruit edge, which indicates that the fitting circle after the image processing is closer to the edge of the fruit under the condition, it is more accurate to estimate the diameter based on the fitted circle. Therefore, in the method, the number of original sampling points which are in a fruit fitting circle and meet the preset edge condition is used as weight, weighted average is carried out on each coordinate parameter group which belongs to the same fruit center point in a plurality of groups of fruit center point coordinate parameters, or weighted average is carried out on fruit diameters estimated based on images shot by left and right cameras, and the precision can be obviously improved.

In addition, determining the coordinates of the fruit center point in a three-dimensional space by using a binocular vision method, and dividing the fruit in the three-dimensional space into two sides of respective fruits according to the position of the fruit center point relative to the cameras on two sides after extracting the depth parameter of the fruit center point;

and for the fruits included in each side, estimating the diameter of the fruits and the weight of the fruits according to the positions of the central points of the fruits, the position of the central point of the reference sphere, the diameter of the reference sphere and the proportional relation between the fitting circle of the fruits and the circle of the reference sphere;

and adding the weight of all fruits on each side, and adding the weight of all fruits on two sides to obtain the total yield of the fruits on the tree.

In this patent, because can utilize binocular vision's method to carry out the coordinate determination of fruit central point in three-dimensional space, extract the degree of depth parameter of fruit central point, consequently can belong to both sides with the fruit branch, carry out the fruit diameter estimation and the weight estimation of each side respectively, because the fruit of this side is nearer apart from the distance of this side camera, the formation of image is also comparatively clear, therefore this kind of mode of dividing the side processing can improve the precision of estimating to a certain extent.

In addition, the moving speed of the camera is determined according to the orchard line spacing and the image processing speed; and moving the camera according to the moving speed of the camera, and carrying out image shooting, image processing and yield estimation, so as to continuously estimate the yield of the fruits on the trees of the fruit trees in the orchard.

A device for rapidly estimating the yield of fruits on a tree comprises binocular image acquisition modules positioned at two sides of the fruit tree to be subjected to yield estimation, wherein the binocular image acquisition modules comprise left and right cameras respectively arranged at two sides of the fruit tree;

the camera comprises a left camera, a right camera, a left camera, a right camera and a left camera, wherein the left camera and the right camera are respectively connected with the left camera and the;

and a reference sphere disposed at a predetermined relative position between the left and right cameras;

and a control module connected to the control unit, the longitudinal adjustment link and the lateral adjustment link, the control module comprising:

the image processing unit is used for carrying out image processing on the images of the fruit tree and the reference sphere, segmenting the outline of the fruit on the fruit tree, carrying out circle fitting on the outline of the fruit and directly carrying out circle generation on the reference sphere according to the central point position and the diameter of the reference sphere;

the fruit diameter estimation unit is used for calculating the position of the fruit and estimating the diameter of the fruit according to the position of the fruit, the position of the reference sphere, the diameter of the reference sphere and the proportional relation between the fruit fitting circle and the reference sphere;

and the on-tree fruit yield estimation unit is used for estimating the fruit weight according to the estimated fruit diameter and accumulating the estimated fruit weight on the fruit tree to obtain the estimated on-tree fruit yield value.

In addition, the fruit diameter estimation unit is used for estimating the diameter of the fruit based on the images shot by the left camera and the right camera according to the position relationship between the two cameras and the center point of the fruit, the position relationship between the two cameras and the center point of the reference sphere, the diameter of the reference sphere and the proportional relationship between the fitting circle of the fruit in the images shot by the cameras on the sides and the circle of the reference sphere;

In addition, the device also comprises a synchronous walking roller connected with the two longitudinal adjusting connecting rods, and the synchronous walking roller is connected to the control module and used for walking under the control of the control module;

the control module utilizes the binocular image acquisition module to simultaneously shoot images of fruit trees on two sides at different walking positions of the synchronous walking roller respectively according to opposite directions on two sides and the same horizontal height position to form a plurality of groups of images of the fruit trees on two sides;

the fruit diameter estimation unit is used for determining coordinates of fruit center points in a three-dimensional space by aiming at each group of bilateral fruit tree images in the multiple groups of bilateral fruit tree images according to fitting circles of fruit outlines in the bilateral fruit tree images, extracting fruit center point depth parameters aiming at the group of bilateral fruit tree images, and forming multiple groups of fruit center point coordinate parameters, wherein each group of fruit center point coordinate parameters comprises respective coordinate parameters of multiple fruit center points;

carrying out weighted average on all coordinate parameters belonging to the same fruit center point in a plurality of groups of fruit center point coordinate parameters according to image quality parameters as weights to obtain modified coordinate parameters of the fruit center point, wherein the image quality parameters are parameters determined in the process of carrying out image processing on double-side fruit tree images corresponding to the group of fruit center point coordinate parameters;

Drawings

FIG. 1 is a schematic diagram of a device for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a method for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention.

FIG. 3 is a schematic diagram illustrating the principle of fruit diameter estimation in the method for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention.

FIG. 4 is a schematic diagram illustrating the principle of correcting the estimated diameter of fruit in a method for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention.

FIG. 5 is a schematic diagram illustrating the calculated positions of modified fruits in the method for rapidly estimating the yield of fruits on a tree according to an embodiment of the present invention.

Detailed Description

The present invention will be described in detail below with reference to the accompanying drawings.

Detailed exemplary embodiments are disclosed below. However, specific structural and functional details disclosed herein are merely for purposes of describing example embodiments.

It should be understood, however, that the intention is not to limit the invention to the particular exemplary embodiments disclosed, but to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure. Like reference numerals refer to like elements throughout the description of the figures.

Referring to the drawings, the structures, ratios, sizes, and the like shown in the drawings are only used for matching the disclosure of the present disclosure, so as to be understood and read by those skilled in the art, and are not used to limit the conditions that the present disclosure can be implemented, so that the present disclosure has no technical significance, and any structural modification, ratio relationship change, or size adjustment should still fall within the scope of the disclosure of the present disclosure without affecting the efficacy and the achievable purpose of the present disclosure. Meanwhile, the positional limitation terms used in the present specification are for clarity of description only, and are not intended to limit the scope of the present invention, and changes or modifications of the relative relationship therebetween may be regarded as the scope of the present invention without substantial changes in the technical content.

It will also be understood that the term "and/or" as used herein includes any and all combinations of one or more of the associated listed items. It will be further understood that when an element or unit is referred to as being "connected" or "coupled" to another element or unit, it can be directly connected or coupled to the other element or unit or intervening elements or units may also be present. Moreover, other words used to describe the relationship between components or elements should be understood in the same manner (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).

FIG. 2 is a flow chart of a method for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention, and FIG. 3 is a schematic diagram of a fruit diameter estimation method for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention. As shown, the present invention comprises a method for rapidly estimating the yield of fruit on a tree, the method comprising the steps of:

A. shooting images of the fruit trees and the reference sphere;

Under the condition that the position of the fruit and the position of the camera are known, the distance between the fruit and the camera can be calculated, so that the parameters actually related in the technical scheme of the invention comprise three parameters of the distance between the camera lens and the fruit, the distance between the camera lens and the reference sphere (determined by the position of the camera and the position of the reference sphere), the diameter proportion of the fruit and the reference sphere, (the distance between the camera lens and the fruit is obtained by referring to the depth of field of the picture), and finally the size of the fruit is obtained according to the size of the reference sphere and the two distances.

As shown in fig. 1, the reference sphere can be placed on both sides, which can bring in two "standard" references, so the solution of the invention can be used in a more flexible way, e.g. first one reference sphere (left or right), when the first reference sphere is not visible to one side camera, then the "standard" reference is set as the other reference sphere; or when the two reference spheres are visible by the cameras on the two sides, the two 'standard' reference quantities are adopted at the same time, so that the analysis result has redundancy, but the two reference spheres can be verified mutually, and the higher accuracy is achieved.

As shown in fig. 3, for example, the reference sphere has a diameter D_rFruit diameter of D_f(unknowns) because of the relative position of the reference sphere and the camera, the direct projection on the picture generates a reference sphere circle with a diameter D_r1The fitting circle diameter of the fruit is D_f1Then, according to the position of the fruit, if the reference sphere is moved to the position of the fruit, the circular diameter will be changed to D_r2，D_r1And D_r2Is only proportional to the distance between them, according to D_r2And D_f1The diameter D of the fruit can be obtained_f。

In addition, in an embodiment of the present invention, the step of capturing the images of the fruit tree and the reference sphere includes: simultaneously shooting images of fruit trees on two sides at the same horizontal height position in the opposite directions;

The binocular vision method is similar to the method that human eyes generate stereoscopic impression according to the distance difference of two eyes, and the binocular vision also calculates the position difference of the same fruit on the images of two cameras according to the distance between the optical axes of the two cameras, so that the real coordinates of the fruit are restored.

FIG. 4 is a schematic diagram illustrating the principle of correcting the estimated diameter of fruit in a method for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention. As shown, in one embodiment of the present disclosure, estimating the diameter of the fruit according to the position of the center point of the fruit, the position of the center point of the reference sphere, the diameter of the reference sphere, and the proportional relationship between the fitting circle of the fruit and the circle of the reference sphere includes:

FIG. 5 is a schematic diagram illustrating the calculated positions of modified fruits in the method for rapidly estimating the yield of fruits on a tree according to an embodiment of the present invention. As shown in the figure, in a specific embodiment of the present patent, the step a of simultaneously capturing the fruit tree images at the same horizontal height position in opposite lateral directions includes: simultaneously shooting fruit tree images on two sides at different advancing positions respectively according to opposite two side directions and the same horizontal height position to form a plurality of groups of fruit tree images on two sides;

As shown in fig. 5, for example, at the first walking position, a first set of bilateral fruit tree images, referred to as G01, G02, is formed by simultaneously photographing bilateral fruit tree images at the same horizontal height position in opposite lateral directions, and coordinates (F00, F01, F02, F03 … …, F0N) of a plurality of fruit center points are obtained by performing image processing and circle fitting on the first set of bilateral fruit tree images and using binocular vision, wherein F00 to F0N are all three-dimensional vectors.

In the same way, at the second walking position, a second set of bilateral fruit images, called G11 and G12, is taken and the coordinates of the center points of a plurality of fruits (F10, F11, F12, F13 … … F1N) are obtained, where F10-F1N are all three-dimensional vectors.

And (3) aiming at the coordinates of the center point of any one of the fruits, such as F00 and F10, obtaining F0 in a weighted average mode, wherein the F0 is used as the coordinate parameter of the center point of the fruit after correction.

In fig. 5, the comprehensive optimization on one coordinate is depicted only as an example, but those skilled in the art can appreciate that the comprehensive optimization on the three-dimensional coordinate of the center point of the fruit can be implemented according to the embodiment of the present invention.

In a specific embodiment of the present patent, the image quality parameter during the image processing of the captured image is the number of original sampling points that form a fruit fitting circle and meet a preset edge condition during the circle fitting process of the fruit contour.

The preset edge condition that the original sampling point accords with can be selected as required, for example, after the B-spline curve fitting is carried out on the previous sampling point for the second time, the distance from the next sampling point to the spline curve is less than a preset threshold value, and then the next sampling point accords with the preset edge condition; or the curvature radius of a curve formed by the sampling points and adjacent sampling points is within a certain range, and the like.

In addition, in a specific embodiment of the patent, coordinates of a fruit center point in a three-dimensional space are determined by using a binocular vision method, and after a depth parameter of the fruit center point is extracted, the fruit in the three-dimensional space is divided into two sides of respective fruits according to the positions of the fruit center point relative to two side cameras;

In addition, in a specific embodiment of the patent, the moving speed of the camera is determined according to the orchard line spacing and the image processing speed; and moving the camera according to the moving speed of the camera, and carrying out image shooting, image processing and yield estimation, so as to continuously estimate the yield of the fruits on the trees of the fruit trees in the orchard.

FIG. 1 is a schematic diagram of a device for rapidly estimating the yield of fruit on a tree according to an embodiment of the present invention. Corresponding to the method for rapidly estimating the yield of the fruit on the tree in the specific embodiment of the patent, the specific embodiment of the patent further comprises a device for rapidly estimating the yield of the fruit on the tree, wherein the device comprises binocular image acquisition modules positioned at two sides of the fruit tree to be subjected to yield estimation, and the binocular image acquisition modules comprise left and right cameras respectively arranged at two sides of the fruit tree;

In addition, the device for rapidly estimating the yield of the fruits on the tree in the specific embodiment of the patent further comprises a positioning module and a display module, wherein the positioning module is used for acquiring the position information of the fruit trees; and the display module is used for generating and displaying the orchard yield graph.

While the foregoing description shows and describes several preferred embodiments of the invention, it is to be understood that the invention is not limited to the forms disclosed herein, but is not to be construed as excluding other embodiments and is capable of use in various other combinations, modifications, and environments and is capable of changes within the scope of the inventive concept as expressed herein, commensurate with the above teachings, or the skill or knowledge of the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for rapidly estimating the yield of fruits on trees comprises the following steps:

A. shooting images of the fruit trees and the reference sphere;

2. The method for rapidly estimating the fruit yield on a tree as claimed in claim 1, wherein the step of capturing the images of the fruit tree and the reference sphere comprises: simultaneously shooting images of fruit trees on two sides at the same horizontal height position in the opposite directions;

3. The method of claim 2, wherein estimating the diameter of the fruit according to the position of the center point of the fruit, the position of the center point of the reference sphere, the diameter of the reference sphere, and the ratio of the fitting circle of the fruit to the circle of the reference sphere comprises:

4. The method for rapidly estimating the fruit yield on a tree as claimed in claim 2, wherein the step a of simultaneously photographing the fruit tree images at the same horizontal height position in opposite lateral directions comprises: simultaneously shooting fruit tree images on two sides at different advancing positions respectively according to opposite two side directions and the same horizontal height position to form a plurality of groups of fruit tree images on two sides;

5. The method according to claim 3 or 4, wherein the image quality parameter during the image processing of the captured image is the number of original sampling points that satisfy the predetermined edge condition and constitute a fruit fitting circle during the circle fitting of the fruit contour.

6. The method for rapidly estimating the yield of fruits on trees according to claim 2, wherein the coordinates of the center point of the fruit in the three-dimensional space are determined by using a binocular vision method, and after the depth parameter of the center point of the fruit is extracted, the fruit in the three-dimensional space is divided into fruits on two sides according to the positions of the center point of the fruit relative to the cameras on two sides;

and adding the weight of all fruits on each side, and adding the weight of all fruits on two sides to obtain the total yield estimation value of the fruits on the tree.

7. The method for rapidly estimating the yield of fruit on a tree as claimed in claim 1, wherein the moving speed of the camera is determined according to the orchard line spacing and the image processing speed; and moving the camera according to the moving speed of the camera, and carrying out image shooting, image processing and yield estimation, so as to continuously estimate the yield of the fruits on the trees of the fruit trees in the orchard.

8. A device for rapidly estimating the yield of fruits on a tree comprises binocular image acquisition modules positioned at two sides of the fruit tree to be subjected to yield estimation, wherein the binocular image acquisition modules comprise left and right cameras respectively arranged at two sides of the fruit tree;

9. The apparatus of claim 8, wherein the fruit diameter estimation unit is configured to estimate the diameter of the fruit based on the images captured by the left and right cameras according to the position relationship between the two cameras and the center point of the fruit, the position relationship between the two cameras and the center point of the reference sphere, the diameter of the reference sphere, and the ratio between the fitting circle of the fruit in the images captured by the cameras on each side and the circle of each reference sphere;

10. The apparatus for rapid estimation of on-tree fruit yield according to claim 8, further comprising a synchronous walking roller connected to two longitudinal adjustment links, the synchronous walking roller being connected to the control module for walking under the control of the control module;