CN104636722A - Fast tracking recognition method for overlapped fruits by picking robot - Google Patents

Abstract

The invention discloses a fast tracking recognition method for overlapped fruits by a picking robot. The fast tracking recognition method comprises the following steps: continuously collecting the latest ten frames of overlapped apple images through a camera; segmenting the collected first frame of image, and removing a background; determining the position of the circle center of overlapped apples by calculating the maximal value of the minimal distance from points in a circle to the edge of an outline; calculating the distance from the circle center to the edge of the outline to determine a radius; intercepting a subsequently matched template according to the circle center and the radius; determining the circle centers of overlapped apples in the continuously collected latest tens frames of images, and carrying out fitting and pre-judging on the motion path of the robot according to the circle center of each frame of image; determining the positions of overlapped apples in a next frame of image by synthesizing the radius and the pre-judging path, and intercepting the area of the overlapped apples; finally carrying out matching recognition by adopting a rapid normalized cross-correlation matching algorithm. According to the method, tracking recognition of near-spherical overlapped fruits such as the overlapped apples can be achieved; the running time is short; the picking efficiency of the picking robot can be effectively improved.

Description

A kind of quick Tracking Recognition method of overlapping fruit of picking robot

Technical field

The invention belongs to agricultural mechanical field, relate to the image-recognizing method of fruit and vegetable picking robot, particularly to the quick Tracking Recognition method of the overlapping fruit of the near-sphericals such as apple.

Background technology

From nineteen eighty-three the 1st tomato picking robot since the U.S. is born, the research and development of picking robot go through more than 30 year, and intelligent robot is plucked by various countries in succession all kinds of vegetables and fruits of registration study.But because discrimination and the problem such as harvesting rate is not high, picking robot also has certain distance from practical and commercialization, therefore, improves the picking efficiency of picking robot, and the Practical Performance strengthening picking robot is the key of current research.

The identification of fruit and location are top priority and the design difficulty of fruit picking robot, and the accuracy of identification and location is related to the work efficiency of picking robot.Chinese scholars has carried out large quantifier elimination for overlapping fruit, and achieves some preliminary achievements.Xiang Rong etc. (2012) propose the overlapping tomato of a kind of method identification based on edge, and the recognition correct rate for the overlapping tomato of slightly blocking is 90.9%.Song Huaibo etc. (2013), on the basis of K-means cluster segmentation, adopt the method based on convex hull to split overlapping apple, the experiment proved that, the apple target registration that the method obtains is 85.08%, and average localization error is 14.15%.Xu Yongwei etc. (2013) adopt the strawberry of the HOG operator identification overlap of support vector machine, and it identifies that successful accuracy rate is 87%.But these researchs are all under static condition, and static knowledge method for distinguishing can not be applicable to the dynamic harvesting of picking robot at motion process completely.Lv Jidong etc. (2014) have carried out Primary Study to the identification of dynamic fruit, and experiment proves to utilize the correlativity of front and back image effectively can reduce the time of image procossing, but less for the dynamic Tracking Recognition research of overlapping fruit.

Summary of the invention

The object of the invention is: a kind of quick Tracking Recognition method for the overlapping fruit of the near-sphericals such as apple is provided, solves the problem affecting robotic tracking's identification because fruit self-sow causes overlap.Its method is simple, and versatility is good, can accurately orient overlapping seed ball and improve the speed of picking robot.

The technical scheme of the quick Tracking Recognition method of the overlapping fruit of picking robot of the present invention comprises the following steps:

The quick Tracking Recognition method of overlapping fruit of picking robot, comprises the following steps:

Step 1, overlapping Apple image collection: adopt colored CCD camera continuous acquisition image;

Step 2, objective fruit is split: to the Image Segmentation Using collected, and removes background, and adopts Mathematical Morphology Method to carry out perfect to the image after segmentation, removes noise and hole;

Step 3, determines the center of circle and the radius of overlapping apple: the position finding the center of circle by finding the maximum value putting contour edge minor increment in circle; After the center of circle is determined, according to the distance determination radius of the center of circle to contour edge;

Step 4, extracts objective fruit template: add according to the center of circle of trying to achieve and radius the template that certain reserved value intercepts subsequent match;

Step 5, robot motion path anticipation: matching is carried out and anticipation to the motion path in the center of circle of fruit in the up-to-date 10 width images of continuous acquisition in robot motion, intercepts overlapping fruit region according to the center of circle of anticipation and the radius of fruit;

Step 6, match cognization: adopt quick normalized crosscorrelation to mate and match cognization is carried out to overlapping fruit.

Further, adopt Mathematical Morphology Method that the image after segmentation is carried out to perfect process and is in described step 2:

Step 2.1, carries out dilation operation with the disc-shaped structure element that radius is a pixel to image, expands frontier point, fills up some perforations;

Step 2.2, carries out holes filling with floodfill algorithm, fills up the aperture of calyx part, tries to achieve the largest connected region of image afterwards, is removed by isolated burr;

Step 2.3, carries out erosion operation to image, eliminates the noise of boundary member.

Further, described step 3 is specially:

Step 3.1, determines the step in the center of circle: define four direction of scanning A (x ⁺, y ⁺), B (x ^-, y ⁺), C (x ^-, y ^-), D (x ⁺, y ^-), in A direction, with from left to right, mode from top to bottom scans; In B direction, to turn left from the right side, mode from top to bottom scans; In C direction, to turn left from the right side, mode from top to bottom scans; In D direction, with from left to right, mode from top to bottom scans;

Step 3.2, determines the step of radius: the above-mentioned center of circle A (a obtaining overlapping apple _x, a _y), B (b _x, b _y); Obtain the straight-line equation y through the center of circle A, B again; Obtain the intersection point C (c of this straight line and fruit profile _x, c _y), D (d _x, d _y); Radius $r_1=\sqrt{{(a_x-c_x)}^2+{(a_y-c_y)}^2},r_2=\sqrt{{(b_x-d_x)}^2+{(b_y-d_y)}^2}.$

Further, the concrete processing procedure of described step 5 is as follows:

Step 5.1, the position in overlapping fruit two centers of circle in the image of the front 9 width continuous acquisition determined according to described step 3.1, fitting of a polynomial is carried out to the mid point in two centers of circle, simulate the path of robot motion, carry out anticipation in conjunction with robot movement velocity and sampling time again, estimate the position that next frame Circle in Digital Images is put in the heart;

Step 5.2, determines radius according to described step 3.2, obtains the maximal value r of two fruit radiuses _max, centered by the mid point in two of step 5.1 anticipation centers of circle, the intercepting length of side is 4*r _maxsquare as the region of successive image process.

Further, in described step 6, the simplify of arithmetic of normalized crosscorrelation coupling is:

$R_1(x,y)=\frac{F^{-1}\left\{F\right\{I\}\·F*\{T^{\′}\left\}\right\}}{\sqrt{\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{[I(x+u,y+\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}]}^2}}$

In formula, I is image to be matched (pixel is M × N); T is (x, y) is template image (pixel is m × n); (x, y) is subgraph I _x,ythe coordinate of the upper left corner in image I; (u, γ) is pixel coordinate in a template; for subgraph I _x,ypixel average;

Technique effect of the present invention is: the fruit overlapping cases caused due to self-sow for near-spherical fruits such as apples, this inventive method can accurately orient overlapping fruit under robot motion's state, and according to front and back image information, anticipation is carried out to robot motion path, reduce the workload of successive image process, therefore working time is short, and the real-time of harvesting obtains effective raising.

Accompanying drawing explanation

Fig. 1 is the quick Tracking Recognition process flow diagram of overlapping apple;

Fig. 2 is the image after overlapping Apple Iamge Segmentation and morphology operations, and wherein, Fig. 2 a is original image, and Fig. 2 b segments the image of dealing with problems arising from an accident;

Fig. 3 is the identification location map of apple original image overlapping shown in Fig. 2, wherein, Fig. 3 a is direction of scanning schematic diagram when determining the center of circle, Fig. 3 b is the minor increment three-dimensional function figure of the point in circle to contour edge, Fig. 3 c is the method schematic diagram determining radius, and Fig. 3 d is the recognition result figure of overlapping fruit image;

Fig. 4 is the matching template figure extracted;

Fig. 5 is robot motion's path curve fitted figure;

Fig. 6 is overlapping apple extracted region result figure, and wherein Fig. 6 a is the schematic diagram of overlapping apple extracted region, and Fig. 6 b is result figure;

Fig. 7 is quick normalized crosscorrelation matching result figure.

Embodiment

Be further described the specific embodiment of the present invention below in conjunction with accompanying drawing, idiographic flow of the present invention as shown in Figure 1.

1, overlapping Apple image collection

The present invention adopts colored CCD camera continuous acquisition image, and frequency acquisition is 10 frames/second, continuous acquisition image in robot kinematics, utilizes the overlapping Apple image of 10 up-to-date width.

2, objective fruit segmentation

The present embodiment to adopt based on the OTSU split plot design of color characteristic the Image Segmentation Using collected, and namely deducts G component by the R component under RGB color model.There is hole in the image after over-segmentation, burr, noise etc., therefore adopts Mathematical Morphology Method to carry out perfect to the image after segmentation.Concrete grammar first carries out dilation operation with the disc-shaped structure element that radius is 1 pixel to image, expands frontier point, fill up some perforations; Then carry out holes filling with Floodfill algorithm, fill up the aperture of calyx part, try to achieve the largest connected region of image afterwards, isolated burr is removed; Finally, then erosion operation is carried out to image, eliminate the noise of boundary member.Iamge Segmentation improves effect as shown in Figure 2.

3, the center of circle and the radius of overlapping apple is determined

3.1 determine the center of circle

Just the position in the center of circle can be found by finding the maximum value putting contour edge minor increment in circle.But, if calculate all point in circle inherently take a large amount of internal memories and real-time is poor to the distance of contour edge.The present embodiment adopts the method improved to scan the point in circle, defines four direction of scanning A (x ⁺, y ⁺), B (x ^-, y ⁺), C (x ^-, y ^-), D (x ⁺, y ^-).In A direction, with from left to right, mode from top to bottom scans; In B direction, to turn left from the right side, mode from top to bottom scans; In C direction, to turn left from the right side, mode from top to bottom scans; In D direction, with from left to right, mode from top to bottom scans, and schematic diagram as shown in Figure 3 a.

Get minimum value after point in profile compares with the point of its four neighborhood, can obtain minor increment function, its three-dimension curved surface design sketch as shown in Figure 3 b.Two place's maximum value of minor increment that what wherein red circle marked is, the namely home position of two overlapping apples.

3.2 determine radius

After the center of circle is determined, can determine radius according to the center of circle, but the center of circle can not be relied on merely to the maximal value of contour edge distance to determine radius, because in overlapping fruit situation, distance maximal value may be the distance to another apple profile.In order to avoid this situation, take herein with under type: the center of circle A (a obtaining overlapping apple according to 3.1 _x, a _y), B (b _x, b _y); Obtain the straight-line equation y through the center of circle A, B again; Obtain the intersection point C (c of this straight line and fruit profile _x, c _y), D (d _x, d _y); Radius $r_1=\sqrt{{(a_x-c_x)}^2+{(a_y-c_y)}^2},r_2=\sqrt{{(b_x-d_x)}^2+{(b_y-d_y)}^2}.$ Determine the schematic diagram of radius as shown in Figure 3 c.The design sketch of overlapping fruit location as shown in Figure 3 d.

4, objective fruit template is extracted

Determine that the center of circle and radius add the template that certain reserved value intercepts subsequent match according to 3.1,3.2.Its design sketch as shown in Figure 4.

5, robot motion path anticipation step: matching is carried out and anticipation to its motion path according to the center of circle of fruit in the up-to-date 10 width images of continuous acquisition in robot motion.Concrete treatment step is as follows:

Step1 is according to before by the position in overlapping fruit two centers of circle in the image of front 9 width continuous acquisition determined of method of 3.1, fitting of a polynomial is carried out to the mid point in two centers of circle, fitting precision is for being less than or equal to 0.5, as shown in Figure 5, simulate the path of robot motion, carry out anticipation in conjunction with robot movement velocity and sampling time again, estimate the position that next frame Circle in Digital Images is put in the heart.

Step2 determines radius by the method for 3.2, obtains the maximal value r of two fruit radiuses _max, centered by the mid point in two of Step1 anticipation centers of circle, the intercepting length of side is 4*r _maxsquare as the region of successive image process.Its schematic diagram as shown in Figure 6 a.

Overlapping fruit extracted region design sketch after treatment as shown in Figure 6 b.

6, match cognization

Adopt quick normalized crosscorrelation to mate and match cognization is carried out to overlapping fruit.Fig. 7 is quick normalized crosscorrelation matching result figure.

The algorithm steps of normalized crosscorrelation coupling is as follows:

If image I to be matched (pixel is M × N) and template image T (pixel is m × n), normalized correlation coefficient is defined as:

$R(x,y)=\frac{\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}[I(x+u,y+\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}][T(u,\mathrm{\γ})-\stackrel{\‾}{T}]}{\sqrt{\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{[I(x+u,y+\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}]}^2\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{[T(u,\mathrm{\γ})-\stackrel{\‾}{T}]}^2}}---\left(1\right)$

In formula: (x, y) is subgraph I _x,ythe coordinate of the upper left corner in image I; (u, γ) is pixel coordinate in a template;

${\stackrel{\‾}{I}}_{x,y}=\frac{1}{\mathrm{mn}}\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}\left[I(x+u,y+\mathrm{\γ})\right]---\left(2\right)$

For subgraph I _x,ypixel average;

$\stackrel{\‾}{T}=\frac{1}{\mathrm{mn}}\underset{i=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{j=0}{\overset{n-1}{\mathrm{\Σ}}}T(u,\mathrm{\γ})---\left(3\right)$

For the pixel average of template T.The scope of R (x, y) is between (0,1), and coefficient is larger, illustrates that the similarity between two matching templates is higher.

Normalized crosscorrelation matching primitives amount is excessive, and real-time is poor, therefore, adopts quick normalized crosscorrelation matching algorithm.Concrete steps are as follows:

Step 1 establishes then through simplifying, the molecular moiety of formula (1) can be rewritten as:

$\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}I(x+u,y+\mathrm{\γ})T^{\′}(u,\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{T}^{\′}(u,\mathrm{\γ})---\left(4\right)$

In formula $\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{T}^{\′}(u,\mathrm{\γ})=\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}[T(u,\mathrm{\γ})-\stackrel{\‾}{T}]=0$ Then molecule is reduced to further:

$R{(x,y)}_{\mathrm{numerator}}=\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}I(x+u,y+\mathrm{\γ})T^{\′}(u,\mathrm{\γ})---\left(5\right)$

According to the character of Fourier transform, molecule can be rewritten as

R(x,y) _numerator＝F ^-1{F{I}·F*{T'}} (6)

Step 2 for denominator part, because template is known, therefore be known definite value, find optimum solution when can not affect normalization coupling, can not calculate, so the denominator of formula (1) can be reduced to:

${R(x,y)}_{\mathrm{numerator}}=\sqrt{\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{[I(x+u,y+\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}]}^2}---\left(7\right)$

In sum, normalized correlation coefficient can be reduced to:

$R_1(x,y)=\frac{F^{-1}\left\{F\right\{I\}\·F*\{T^{\′}\left\}\right\}}{\sqrt{\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{[I(x+u,y+\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}]}^2}}---\left(8\right)$

Should understand above-mentioned example of executing only to be not used in for illustration of the present invention and to limit the scope of the invention, after having read the present invention, the amendment of those skilled in the art to the various equivalent form of value of the present invention has all fallen within the application's claims limited range.

Claims (5)

1. the quick Tracking Recognition method of the overlapping fruit of picking robot, is characterized in that, comprise the following steps:

Step 1, overlapping Apple image collection: adopt colored CCD camera continuous acquisition image;

Step 2, objective fruit is split: to the Image Segmentation Using collected, and removes background, and adopts Mathematical Morphology Method to carry out perfect to the image after segmentation, removes noise and hole;

Step 3, determines the center of circle and the radius of overlapping apple: the position finding the center of circle by finding the maximum value putting contour edge minor increment in circle; After the center of circle is determined, according to the distance determination radius of the center of circle to contour edge;

Step 4, extracts objective fruit template: add according to the center of circle of trying to achieve and radius the template that certain reserved value intercepts subsequent match;

Step 5, robot motion path anticipation: matching is carried out and anticipation to the motion path in the center of circle of fruit in the up-to-date 10 width images of continuous acquisition in robot motion, intercepts overlapping fruit region according to the center of circle of anticipation and the radius of fruit;

Step 6, match cognization: adopt quick normalized crosscorrelation to mate and match cognization is carried out to overlapping fruit.

2. the quick Tracking Recognition method of the overlapping fruit of picking robot according to claim 1, is characterized in that, adopts Mathematical Morphology Method to carry out perfect process to the image after segmentation to be in described step 2:

Step 2.1, carries out dilation operation with the disc-shaped structure element that radius is a pixel to image, expands frontier point, fills up some perforations;

Step 2.2, carries out holes filling with floodfill algorithm, fills up the aperture of calyx part, tries to achieve the largest connected region of image afterwards, is removed by isolated burr;

Step 2.3, carries out erosion operation to image, eliminates the noise of boundary member.

3. the quick Tracking Recognition method of the overlapping fruit of picking robot according to claim 1, is characterized in that, described step 3 concrete steps are:

Step 3.1, determines the step in the center of circle: define four direction of scanning A (x ⁺, y ⁺), B (x ^-, y ⁺), C (x ^-, y ^-), D (x ⁺, y ^-), in A direction, with from left to right, mode from top to bottom scans; In B direction, to turn left from the right side, mode from top to bottom scans; In C direction, to turn left from the right side, mode from top to bottom scans; In D direction, with from left to right, mode from top to bottom scans;

Step 3.2, determines the step of radius: the above-mentioned center of circle A (a obtaining overlapping apple _x, a _y), B (b _x, b _y); Obtain the straight-line equation y through the center of circle A, B again; Obtain the intersection point C (c of this straight line and fruit profile _x, c _y), D (d _x, d _y); Radius $r_1=\sqrt{{(a_x-c_x)}^2+{(a_y-c_y)}^2},r_2=\sqrt{{(b_x-d_x)}^2+{(b_y-d_y)}^2}.$

4. the quick Tracking Recognition method of the overlapping fruit of picking robot according to claim 1, is characterized in that, the concrete processing procedure of described step 5 is as follows:

Step 5.1, the position in overlapping fruit two centers of circle in the image of the front 9 width continuous acquisition determined according to described step 3.1, fitting of a polynomial is carried out to the mid point in two centers of circle, simulate the path of robot motion, carry out anticipation in conjunction with robot movement velocity and sampling time again, estimate the position that next frame Circle in Digital Images is put in the heart;

Step 5.2, determines radius according to described step 3.2, obtains the maximal value r of two fruit radiuses _max, centered by the mid point in two of step 5.1 anticipation centers of circle, the intercepting length of side is 4*r _maxsquare as the region of successive image process.

5. the quick Tracking Recognition method of the overlapping fruit of picking robot according to claim 1, is characterized in that, in described step 6, the simplify of arithmetic of normalized crosscorrelation coupling is:

$R_1(x,y)=\frac{F^{-1}\left\{F\right\{I\}\·F*\{T^{\′}\left\}\right\}}{\sqrt{\underset{\mathrm{\μ}=0}{\overset{m-1}{\mathrm{\Σ}}}\underset{\mathrm{\γ}=0}{\overset{n-1}{\mathrm{\Σ}}}{[I(x+u,y+\mathrm{\γ})-{\stackrel{\‾}{I}}_{x,y}]}^2}}$

In formula, I is image to be matched (pixel is M × N); T is (x, y) is template image (pixel is m × n); (x, y) is subgraph I _x,ythe coordinate of the upper left corner in image I; (u, g) is pixel coordinate in a template; for subgraph I _x,ypixel average.