CN115100644A - Fruit processing method and device, electronic equipment and computer readable storage medium - Google Patents

Abstract

The invention discloses a fruit processing method, a fruit processing device, electronic equipment and a computer readable storage medium. Wherein, the method comprises the following steps: acquiring a target image, wherein the target image comprises a single preset fruit and vegetable and a stacked preset fruit and vegetable; determining a first fruit edge of a single preset fruit and vegetable fruit and a second fruit edge of a stacked preset fruit and vegetable fruit in the target image; dividing a plurality of preset fruit and vegetable image according to the first fruit edge and the second fruit edge; and determining mature preset fruit and vegetable fruits from the preset fruit and vegetable images. The invention solves the technical problem that the detection of the preset fruits and vegetables is incomplete when the preset fruits and vegetables are detected aiming at a single preset fruit and vegetable in the related technology.

Description

Fruit processing method and device, electronic equipment and computer readable storage medium

Technical Field

The invention relates to the field of computers, in particular to a fruit processing method, a fruit processing device, electronic equipment and a computer readable storage medium.

Background

When the detection is carried out on the preset fruits and vegetables in the related technology, the detection processing is only carried out on the single preset fruits and vegetables, and the technical problem of incomplete detection exists.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The embodiment of the invention provides a fruit processing method, a fruit processing device, electronic equipment and a computer readable storage medium, which at least solve the technical problem that the detection of preset fruits and vegetables is incomplete due to the fact that the detection is carried out on a single preset fruit and vegetable fruit when the preset fruits and vegetables are detected in the related technology.

According to an aspect of an embodiment of the present invention, there is provided a fruit processing method including: acquiring a target image, wherein the target image comprises a single preset fruit and vegetable and a stacked preset fruit and vegetable; determining a first fruit edge of the single predetermined fruit and vegetable fruit and a second fruit edge of the stacked predetermined fruit and vegetable fruit in the target image; segmenting a plurality of preset fruit and vegetable image according to the first fruit edge and the second fruit edge, wherein any one preset fruit and vegetable image in the preset fruit and vegetable image comprises a target preset fruit and vegetable, and the target preset fruit and vegetable included in the preset fruit and vegetable image is not repeated; and determining mature preset fruit and vegetable fruits from the preset fruit and vegetable images.

Optionally, the determining a mature predetermined fruit and vegetable fruit from a plurality of target predetermined fruit and vegetable fruits according to the plurality of predetermined fruit and vegetable fruit images includes: acquiring pixel data of the preset fruit and vegetable image; determining a plurality of preset fruit and vegetable maturity degrees corresponding to the target preset fruit and vegetable according to the pixel data of the preset fruit and vegetable image, wherein the target preset fruit and vegetable maturity degrees correspond to the preset fruit and vegetable maturity degrees one to one; and determining the target predetermined fruit and vegetable fruit with the predetermined fruit and vegetable maturity being larger than the first predetermined threshold value as the mature predetermined fruit and vegetable fruit.

Optionally, the determining a mature predetermined fruit and vegetable fruit from a plurality of target predetermined fruit and vegetable fruits according to the plurality of predetermined fruit and vegetable fruit images includes: determining the integrity of a plurality of target preset fruit and vegetable fruits corresponding to the plurality of target preset fruit and vegetable fruits according to the plurality of preset fruit and vegetable fruit images; and determining mature preset fruit and vegetable fruits from the target preset fruit and vegetable fruits with the target preset fruit and vegetable integrity degree larger than a second preset threshold value.

Optionally, the determining, according to the predetermined fruit and vegetable image, the integrity of the target predetermined fruit and vegetable corresponding to the target predetermined fruit and vegetable includes: determining an incomplete preset fruit and vegetable image, wherein the incomplete preset fruit and vegetable image is a preset fruit and vegetable image with an angle at the edge of the image in the preset fruit and vegetable image; completing the incomplete predetermined fruit and vegetable image to obtain a complete predetermined fruit and vegetable image; and determining the integrity of the target preset fruit and vegetable corresponding to the incomplete preset fruit and vegetable image according to the incomplete preset fruit and vegetable image and the supplemented preset fruit and vegetable image.

Optionally, said determining a first fruit edge of said single predetermined fruit and vegetable fruit and a second fruit edge of said stacked predetermined fruit and vegetable fruits in said target image comprises: intercepting a fruit image according to the target image, wherein the fruit image comprises a single preset fruit and vegetable image and a stacked preset fruit and vegetable image; determining a first fruit edge of a single predetermined fruit and vegetable fruit in the single predetermined fruit and vegetable fruit image and a second fruit edge of a stacked predetermined fruit and vegetable fruit in the stacked predetermined fruit and vegetable fruit image.

Optionally, the determining a second fruit edge of the stacked predetermined fruit and vegetable fruit in the stacked predetermined fruit and vegetable fruit image comprises: adjusting image chromaticity data of the stacked preset fruit and vegetable fruit images; and determining a second fruit edge of the stacked preset fruit and vegetable in the stacked preset fruit and vegetable image according to the stacked preset fruit and vegetable image after the image chromaticity data is adjusted.

Optionally, the method further comprises: determining a target two-dimensional position of a target predetermined fruit and vegetable image, wherein the target predetermined fruit and vegetable image is a predetermined fruit and vegetable image corresponding to the mature predetermined fruit and vegetable; determining a target three-dimensional position of the mature predetermined fruit and vegetable according to the target two-dimensional position; and sending a picking instruction to a preset terminal so that the preset terminal picks the mature preset fruit and vegetable at the target three-dimensional position.

According to an aspect of an embodiment of the present invention, there is provided a fruit handling device comprising: the device comprises an acquisition module, a storage module and a display module, wherein the acquisition module is used for acquiring a target image, and the target image comprises a single preset fruit and vegetable fruit and a stacked preset fruit and vegetable fruit; a first determining module to determine a first fruit edge of the single predetermined fruit and vegetable fruit and a second fruit edge of the stacked predetermined fruit and vegetable fruit in the target image; a dividing module, configured to divide a plurality of predetermined fruit and vegetable image according to the first fruit edge and the second fruit edge, where any one of the predetermined fruit and vegetable image includes a target predetermined fruit and vegetable, and the target predetermined fruit and vegetable included in the predetermined fruit and vegetable image is not repeated; and the second determining module is used for determining mature preset fruits and vegetables from the preset fruit and vegetable images.

According to an aspect of an embodiment of the present invention, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the fruit handling method of any of the above.

According to an aspect of embodiments of the present invention, there is provided a computer-readable storage medium, wherein instructions of the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform any one of the above fruit processing methods.

In the embodiment of the invention, a target image comprising a single predetermined fruit and vegetable and a stacked predetermined fruit and vegetable is obtained, a first fruit edge of the single predetermined fruit and vegetable and a second fruit edge of the stacked predetermined fruit and vegetable are determined in the target image, so that a plurality of predetermined fruit and vegetable image are segmented according to the first fruit edge and the second fruit edge, wherein any one predetermined fruit and vegetable image in the plurality of predetermined fruit and vegetable image comprises a target predetermined fruit and vegetable, the target predetermined fruit and vegetable included in the plurality of predetermined fruit and vegetable image are not repeated, that is, each target predetermined fruit and vegetable has only one corresponding predetermined fruit and vegetable image, and further a mature predetermined fruit and vegetable is determined in the plurality of target predetermined fruit and vegetable according to the plurality of predetermined fruit and vegetable image. During detection of the target image, the fact that the plurality of predetermined fruit and vegetable fruit images are further divided by determining the fruit edges for subsequent processing is determined, so that the situation that single predetermined fruit and vegetable fruits or predetermined fruit and vegetable fruits in a string or stacked shape exist in a scene where the predetermined fruit and vegetable fruits are planted is considered, and various contents are considered during detection of the predetermined fruit and vegetable fruits, and the technical problem that detection of the single predetermined fruit and vegetable fruits is incomplete due to the fact that detection is carried out on the single predetermined fruit and vegetable fruits during detection of the predetermined fruit and vegetable fruits in the related technology is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

FIG. 1 is a flow diagram of a fruit treatment method according to an embodiment of the invention;

fig. 2 is a flow chart of a tomato fruit detection method provided in an alternative embodiment of the present invention;

fig. 3 is a block diagram of a fruit handling device according to an embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

In accordance with an embodiment of the present invention, there is provided an embodiment of a fruit handling method, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.

Fig. 1 is a flow chart of a fruit treatment method according to an embodiment of the present invention, as shown in fig. 1, the method comprising the steps of:

step S102, acquiring a target image, wherein the target image comprises single preset fruit and vegetable fruits and stacked preset fruit and vegetable fruits;

step S104, determining a first fruit edge of a single preset fruit and vegetable fruit and a second fruit edge of a stacked preset fruit and vegetable fruit in the target image;

step S106, segmenting a plurality of predetermined fruit and vegetable fruit images according to the first fruit edge and the second fruit edge, wherein any one predetermined fruit and vegetable fruit image in the plurality of predetermined fruit and vegetable fruit images comprises a target predetermined fruit and vegetable fruit, and the target predetermined fruit and vegetable fruit included in the plurality of predetermined fruit and vegetable fruit images is not repeated;

and S108, determining mature preset fruits and vegetables from the preset target fruits and vegetables according to the preset fruit and vegetable image.

Through the steps, the target images comprising the single preset fruit and the stacked preset fruit and vegetable are obtained, the first fruit edge of the single preset fruit and vegetable and the second fruit edge of the stacked preset fruit and vegetable are determined in the target images, and therefore the preset fruit and vegetable images are segmented according to the first fruit edge and the second fruit edge, wherein any preset fruit and vegetable image in the preset fruit and vegetable images comprises the preset target fruit and vegetable, the preset target fruit and vegetable included in the preset fruit and vegetable images are not repeated, namely each preset target fruit and vegetable has only one corresponding preset fruit and vegetable image, and the mature preset fruit and vegetable is determined in the preset target fruit and vegetable according to the preset fruit and vegetable images. When the target image is detected, the fact that the plurality of preset fruit and vegetable fruit images are further divided by determining the fruit edges for subsequent processing is determined, therefore, the situation that single preset fruit and vegetable fruit or the preset fruit and vegetable fruit in a string or stack shape exists in a scene where the preset fruit and vegetable fruit is planted is considered, when the preset fruit and vegetable fruit is detected, various contents are considered, and the technical problem that detection of the single preset fruit and vegetable fruit is incomplete due to the fact that detection of the preset fruit and vegetable fruit is carried out when the preset fruit and vegetable fruit is detected in the related technology is solved.

It should be noted that the predetermined fruit and vegetable may be a fruit or a vegetable that meets a predetermined rule, and both the predetermined fruit and vegetable and the predetermined rule may be set by self-definition according to actual applications and scenes. For example, the predetermined rule may be a round fruit or vegetable. In this case, the predetermined fruit and vegetable may be tomato, apple, etc.

As an alternative embodiment, a target image including a single predetermined fruit and vegetable and a stacked predetermined fruit and vegetable is obtained, wherein the target image may be obtained by shooting through a binocular depth camera, for example, an OAK-D depth camera, the binocular depth camera is a camera with two cameras, similar to the structure of human eyes, one left, one right, and the two cameras are on the same y axis and z axis, only a distance difference exists on the x axis, and the x, y, and z axis coordinates of the depth image are calculated through the parallax of the two cameras. Through using binocular depth camera, not only can acquire clear predetermined fruit vegetables image very to for the two-dimensional position of follow-up predetermined fruit vegetables fruit discernment is more accurate, and moreover, binocular depth camera is in fixed position department, and is follow-up when the two-dimensional position converts the three-dimensional position into, can be more accurate, quick, improves and picks efficiency.

As an alternative embodiment, a first fruit edge of a single predetermined fruit and vegetable fruit and a second fruit edge of a stacked predetermined fruit and vegetable fruit are determined in the target image. Through determining the fruit edge, can handle to predetermined fruit vegetables fruit better, get rid of other influence factors in the target image for it can be more accurate to handle.

It should be noted that, in the process of determining the first fruit edge and the second fruit edge in the target image, the first fruit edge and the second fruit edge may be directly determined from the target image, or the target predetermined fruits and vegetables of the single predetermined fruit and vegetable and the stacked predetermined fruits and vegetables in the target image may be detected, the fruit image including the single predetermined fruit and vegetable image and the stacked predetermined fruit and vegetable image is intercepted, and the first fruit edge of the single predetermined fruit and vegetable in the single predetermined fruit and vegetable image and the second fruit edge of the stacked predetermined fruit and vegetable in the stacked predetermined fruit and vegetable image are determined.

Optionally, when the target predetermined fruits and vegetables of the single predetermined fruits and vegetables and the target predetermined fruits and vegetables of the stacked predetermined fruits and vegetables in the target image are detected and the fruit image is intercepted, the target image can be detected by adopting a target detection model, that is, the single predetermined fruits and vegetables and the stacked predetermined fruits and vegetables in the target image can be obtained only by inputting the target image including the single predetermined fruits and vegetables and the stacked predetermined fruits and vegetables into the target detection model corresponding to the predetermined fruits and vegetables. And judging whether the target image contains a category meeting the requirement or not by using a target detection model, if so, outputting a series of labels with confidence coefficients to indicate the probability of the single predetermined fruit and vegetable and the stacked predetermined fruit and vegetable in the target image, and after the probability exceeds a certain value, considering that the target image contains the category meeting the requirement, and outputting a fruit image comprising the single predetermined fruit and vegetable fruit image and the stacked predetermined fruit and vegetable fruit image.

Optionally, when determining a first fruit edge of a single predetermined fruit and vegetable fruit in a single predetermined fruit and vegetable fruit image and a second fruit edge of a stacked predetermined fruit and vegetable fruit in a stacked predetermined fruit and vegetable fruit image, the single predetermined fruit and vegetable fruit image and the stacked predetermined fruit and vegetable fruit image may be processed in different manners. For example, when the first fruit edge of the single predetermined fruit and vegetable fruit in the single predetermined fruit and vegetable fruit image is determined, the single predetermined fruit and vegetable fruit image can be directly processed in a targeted manner in a semantic segmentation manner. When the second fruit edge of the stacked predetermined fruit and vegetable fruits in the stacked predetermined fruit and vegetable fruit image is determined, the edge of each predetermined fruit and vegetable fruit in the stacked predetermined fruit and vegetable fruits can be extracted in a semantic segmentation mode. The outline of the whole stacked preset fruit and vegetable fruits in the stacked preset fruit and vegetable fruits can be extracted firstly by using a semantic segmentation mode, then the image chromaticity data of the images of the stacked preset fruit and vegetable fruits is adjusted, and the second fruit edge of the stacked preset fruit and vegetable fruits in the images of the stacked preset fruit and vegetable fruits is determined according to the images of the stacked preset fruit and vegetable fruits after the image chromaticity data is adjusted. The pixel values of the predetermined fruit and vegetable fruits stacked together can be greatly and obviously distinguished after the adjustment of the image chromaticity data, and in this case, the edge of each predetermined fruit and vegetable fruit can be better determined in the outline of the whole stacked predetermined fruit and vegetable fruits.

As an alternative embodiment, a plurality of predetermined fruit and vegetable fruit images are segmented according to the first fruit edge and the second fruit edge, wherein any one of the predetermined fruit and vegetable fruit images includes a target predetermined fruit and vegetable fruit, and the target predetermined fruit and vegetable fruit included in the predetermined fruit and vegetable fruit images does not overlap. Namely, each target preset fruit and vegetable has only one corresponding preset fruit and vegetable image. It should be noted that, a plurality of predetermined fruit and vegetable image are divided according to the first fruit edge and the second fruit edge, and the divided predetermined fruit and vegetable image may be an irregular image including only predetermined fruit and vegetable, or a regular image in which all areas except predetermined fruit and vegetable are black.

Optionally, in the case of an irregular image, for a single predetermined fruit and vegetable fruit image, only the image within the fruit edge of the predetermined fruit and vegetable fruit is segmented. For stacking the predetermined fruit and vegetable image, the image within the fruit edge only including a certain predetermined fruit and vegetable fruit can be generated during traversal, and a new image within the fruit edge only including the predetermined fruit and vegetable fruit is generated for each predetermined fruit and vegetable fruit

Optionally, when the image is a regular image, for a single predetermined fruit and vegetable image, the pixel points outside the edge of the predetermined fruit and vegetable are set to be 0 (black). For stacking the predetermined fruit and vegetable fruit images, all pixels of other regions except a certain predetermined fruit and vegetable fruit can be set to be 0 during traversal, namely, a new image which only has the RGB value in the partial region of the predetermined fruit and vegetable fruit and has black other parts is generated for each predetermined fruit and vegetable fruit.

As an alternative embodiment, the mature predetermined fruit-vegetable fruit is determined among the plurality of target predetermined fruit-vegetable fruits based on the plurality of predetermined fruit-vegetable fruit images. Before the mature predetermined fruits and vegetables are determined, the fruit integrity of the predetermined fruits and vegetables can be determined. For example, according to the plurality of predetermined fruit and vegetable image, a plurality of target predetermined fruit and vegetable integrity degrees corresponding to the plurality of target predetermined fruit and vegetable fruits are determined, and the mature predetermined fruit and vegetable fruits are determined from the target predetermined fruit and vegetable fruits with the target predetermined fruit and vegetable integrity degrees larger than a second predetermined threshold value. Specifically, ResNet-101 can be used for carrying out secondary classification on a plurality of divided preset fruit and vegetable fruit images to determine the integrity of the target preset fruit and vegetable. Through the mode, the completeness of the preset fruits and vegetables is identified, so that the problem that the maturity of the preset fruits and vegetables is identified to cause errors due to fruit shielding caused by the overlapping of the preset fruits and vegetables is avoided, and the accuracy of the identification of the preset fruits and vegetables is ensured.

As an alternative embodiment, there are various ways to determine the integrity of the predetermined fruit or vegetable, for example, the following ways are adopted: determining an incomplete predetermined fruit and vegetable image, wherein the incomplete predetermined fruit and vegetable image is a predetermined fruit and vegetable image with an angle at the edge of the image in a plurality of predetermined fruit and vegetable images, wherein in an irregular image, the angle at the edge of the image refers to the edge angle of the irregular image, and in a regular image, the angle at the edge of the image refers to the angle in the image part with the RGB value; completing the incomplete preset fruit and vegetable image to obtain a complete preset fruit and vegetable image; and determining the integrity of the target preset fruit and vegetable corresponding to the incomplete preset fruit and vegetable image according to the incomplete preset fruit and vegetable image and the complete preset fruit and vegetable image. The area ratio of the part before the completion of the fruit and the part before the completion of the fruit can be calculated, so that the completeness of the preset fruit and vegetable can be determined, and the calculation accuracy of the preset fruit and vegetable is improved.

As an optional embodiment, after the mature predetermined fruit and vegetable fruit is determined in the target predetermined fruit and vegetable fruits according to the predetermined fruit and vegetable images, the method further includes the step of picking the mature predetermined fruit and vegetable fruit, that is, the target two-dimensional position of the target predetermined fruit and vegetable image can be determined, wherein the target predetermined fruit and vegetable image is a predetermined fruit and vegetable image corresponding to the mature predetermined fruit and vegetable fruit; determining a target three-dimensional position of a mature predetermined fruit and vegetable fruit according to the target two-dimensional position; and sending a picking instruction to the preset terminal so that the preset terminal picks the mature preset fruits and vegetables at the target three-dimensional position. The preset terminal can be a machine capable of picking the preset fruits and vegetables, such as a mechanical arm. Through this optional embodiment, realized the integration setting that predetermined fruit vegetables fruit space orientation was taken for this scheme is more comprehensive, and intelligence has more the practicality.

Based on the above embodiments and alternative embodiments, an alternative implementation is provided, which is described in detail below.

The invention provides a tomato fruit detection method in an optional embodiment, and the method integrates tomato fruit detection, tomato fruit maturity detection and tomato fruit space positioning. Fig. 2 is a flow chart of a tomato fruit detection method provided in an alternative embodiment of the present invention, as shown in fig. 2, which is specifically described below:

s1, acquiring a tomato fruit image in a tomato planting scene;

the tomato fruit image can be acquired by the binocular depth camera at the fixed position, the tomato fruit image is acquired by the binocular depth camera at the fixed position, clear tomato images can be acquired in a very arriving mode, two-dimensional position recognition of follow-up tomato fruits is more accurate, and the binocular depth camera is located at the fixed position, and can be more accurate and rapid when the two-dimensional position is converted into a three-dimensional position, so that picking efficiency is improved.

S2, realizing target fruit detection of single tomato fruits and stacked tomato fruits;

specifically, the YOLOv5x target detection model can be used to directly perform target detection recognition on tomato fruit images. In this process, the identified tomato fruits are divided into two categories, one being a single target individual with only one fruit; the other is a string-like overlapping target string where multiple targets are aggregated, overlapping.

S3, intercepting a target image, wherein the target image comprises a single tomato fruit image and a stacked tomato fruit image;

when the fruit image is cut out by classifying the positioned target into the recognized target frame after the target in the image is positioned, the cut-out fruit image has a few pixels of the partial background image, and therefore, the processing is performed again by the processing method in step S4.

S4, determining a first fruit edge of a single tomato fruit and a second fruit edge of a stacked tomato fruit in the target image, and dividing a plurality of tomato fruit images according to the first fruit edge and the second fruit edge;

specifically, different types of target images may be processed specifically using deplab 3+ semantic segmentation.

1) And (2) directly performing semantic segmentation on a single tomato fruit image, extracting the edge of the tomato fruit to generate the tomato fruit image, wherein the tomato fruit image can be an irregular image only comprising the tomato fruit, or can be a regular image except for the black area of the tomato fruit, and when the tomato fruit image is the regular image, setting the pixel points except the edge of the tomato fruit to be 0 (black).

2) For stacked tomato fruit images, semantic segmentation is performed on each tomato fruit, extracting the edges of each tomato fruit. And traversing each tomato fruit to generate a tomato fruit image, wherein the tomato fruit image can be an irregular image only comprising the tomato fruit, or can be a regular image in which all areas except the tomato fruit are black, and when the tomato fruit image is a regular image, all pixels of the areas except a certain tomato fruit can be set to 0 during traversal, that is, a new image in which only part of the area of the tomato fruit is with RGB value and all the other parts are black is generated for each tomato fruit.

It should be further noted that after the plurality of tomato fruit images are obtained, the plurality of tomato fruit images may be numbered, where any one of the plurality of tomato fruit images includes one target tomato fruit, and the target tomato fruit included in the plurality of tomato fruit images is not repeated, and through the numbering process, the target tomato fruit can be identified more easily, so that the method provided in the optional embodiment of the present invention is more ordered.

S5, determining a plurality of tomato ripeness degrees corresponding to a plurality of target tomato fruits according to the pixel data of the plurality of tomato fruit images;

focusing on the tomato fruit images obtained from the stacked tomato fruit images, in particular, the segmented tomato fruit images can be classified two times using ResNet-101, and images with an occlusion degree of no more than 20% are considered as intact fruits (slight occlusion has little influence on the detection of the overall maturity). When the training data set is made, fruits not exceeding 20% and fruits exceeding 20% need to be classified for training ResNet-101 to make two classifications to judge whether the fruits are occluded or not, identify the fruits which are not occluded, and send the identified tomato fruits with the occlusion degree not exceeding 20% to S6.

S6, according to the tomato fruit images, determining a mature tomato fruit in the target tomato fruits;

specifically, the method can use an RGB color model to calculate the mean value, traverse the pixel points in the input tomato fruit image, extract the R value in the tomato fruit image pixel, and calculate the mean value. Identifying the pluckable tomato fruits and the pluckable-unpeelable tomato fruits, determining the number and the three-dimensional position information corresponding to the pluckable tomato fruits, and determining the plucking instruction carrying the number and the three-dimensional position information.

Specifically, a binocular depth camera is used for determining a two-dimensional position where the tomato fruits can be picked according to the tomato fruit images, then three-dimensional positioning is carried out on the picked tomato fruits, the mass center pixel points of the picked fruits are calculated through the pixels on the edges of the tomato fruits, and the three-dimensional coordinates of the target mass center are calculated.

And S7, sending a picking instruction to the three-axis mechanical arm to control the three-axis mechanical arm to pick the picked tomato fruits at the three-dimensional position of the target.

According to the three-dimensional position information, whether the tomato fruits are in the picking range of the current three-axis mechanical arm position or not can be determined, the tomato fruits exceeding the picking range of the three-axis mechanical arm are ignored firstly, the nearby tomato fruits are picked firstly, namely the tomato fruits can be picked at the current three-axis mechanical arm position, after the nearby tomato fruits are picked, the three-axis mechanical arm picks the residual three-dimensional coordinate information of the picked tomato fruits, in the process, the three-dimensional coordinate information can be sorted according to the distance, then the three-dimensional coordinate information can be picked in turn, and the picking efficiency is improved. After picking at one position is finished, a forward movement signal is sent to the moving chassis, and picking at the next position is carried out. After the fruits identified in the first round are picked, the second round of identification is carried out from S1 until no tomato fruits can be picked in the picking range of all positions of the three-axis mechanical arm.

Through the above alternative embodiment, at least the following advantages can be achieved:

(1) the target detection algorithm is used for positioning, tomato fruit images are cut out and sent to the semantic segmentation algorithm, so that the precision of the semantic segmentation algorithm is improved, optimization improvement is made aiming at the condition that small targets in a large image are not accurately identified, and the small image is more favorable for semantic segmentation to segment target tomato fruits;

(2) the integrity of the tomato fruits is identified, so that errors caused by tomato fruit shielding due to overlapping of the tomato fruits in identifying the maturity of the tomato fruits are avoided;

(3) the method for extracting the RGB color of the image after semantic segmentation recognition and calculating the maturity of the tomato ensures that the extracted RGB values are all on the tomato fruit, avoids the influence of the color of the background image on the calculation of the maturity of the tomato fruit when the RGB values are extracted, and enables the calculation of the maturity of the tomato fruit to be more accurate and credible.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.

Example 2

According to an embodiment of the present invention, there is also provided an apparatus for implementing the above fruit processing method, and fig. 3 is a block diagram of a structure of a fruit processing apparatus according to an embodiment of the present invention, as shown in fig. 3, the apparatus includes: an acquisition module 302, a first determination module 304, a segmentation module 306, and a second determination module 308, which are described in detail below.

An obtaining module 302, configured to obtain a target image, where the target image includes a single predetermined fruit and vegetable and stacked predetermined fruit and vegetable; a first determining module 304, connected to the obtaining module 302, for determining a first fruit edge of a single predetermined fruit and vegetable fruit and a second fruit edge of a stacked predetermined fruit and vegetable fruit in the target image; a dividing module 306, connected to the first determining module 304, for dividing a plurality of predetermined fruit and vegetable image according to the first fruit edge and the second fruit edge, where any one of the predetermined fruit and vegetable image includes a target predetermined fruit and vegetable, and the target predetermined fruit and vegetable included in the predetermined fruit and vegetable image is not repeated; a second determining module 308, connected to the dividing module 306, for determining a mature predetermined fruit and vegetable from the predetermined fruit and vegetable images.

It should be noted here that the obtaining module 302, the first determining module 304, the dividing module 306 and the second determining module 308 correspond to the implementation of steps S102 to S108 in the fruit processing method, and a plurality of modules are the same as the implementation examples and application scenarios of the corresponding steps, but are not limited to the disclosure in the foregoing embodiment 1.

Example 3

According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including: a processor; a memory for storing processor executable instructions, wherein the processor is configured to execute the instructions to implement the fruit handling method of any of the above.

Example 4

According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, wherein when instructions of the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the above fruit processing methods.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

In the above embodiments of the present invention, the description of each embodiment has its own emphasis, and reference may be made to the related description of other embodiments for parts that are not described in detail in a certain embodiment.

In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be an indirect coupling or communication connection through some interfaces, units or modules, and may be electrical or in other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may also be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention, which is substantially or partly contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and various media capable of storing program codes.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be construed as the protection scope of the present invention.

Claims (10)

1. A method of treating fruit, comprising:

acquiring a target image, wherein the target image comprises a single preset fruit and vegetable and a stacked preset fruit and vegetable;

determining a first fruit edge of the single predetermined fruit and vegetable fruit and a second fruit edge of the stacked predetermined fruit and vegetable fruit in the target image;

segmenting a plurality of preset fruit and vegetable image according to the first fruit edge and the second fruit edge, wherein any one preset fruit and vegetable image in the preset fruit and vegetable image comprises a target preset fruit and vegetable, and the target preset fruit and vegetable included in the preset fruit and vegetable image is not repeated;

and determining mature preset fruit and vegetable fruits from the preset fruit and vegetable images.

2. The method of claim 1, wherein said determining a mature predetermined fruit-vegetable fruit among a plurality of target predetermined fruit-vegetable fruits based on said plurality of predetermined fruit-vegetable fruit images comprises:

acquiring pixel data of the plurality of predetermined fruit and vegetable image;

determining a plurality of preset fruit and vegetable maturity degrees corresponding to the plurality of target preset fruit and vegetable fruits according to the pixel data of the plurality of preset fruit and vegetable fruit images, wherein the plurality of target preset fruit and vegetable fruits correspond to the plurality of preset fruit and vegetable maturity degrees one to one;

and determining the target predetermined fruit and vegetable fruit with the predetermined fruit and vegetable maturity being larger than the first predetermined threshold value as the mature predetermined fruit and vegetable fruit.

3. The method of claim 1, wherein said determining a mature predetermined fruit-based fruit of the plurality of target predetermined fruit-based fruits based on the plurality of predetermined fruit-based fruit images comprises:

determining the integrity of a plurality of target preset fruit and vegetable fruits corresponding to the plurality of target preset fruit and vegetable fruits according to the plurality of preset fruit and vegetable fruit images;

and determining mature preset fruit and vegetable fruits from the target preset fruit and vegetable fruits with the target preset fruit and vegetable integrity degree larger than a second preset threshold value.

4. The method of claim 3, wherein said determining a plurality of target predetermined fruit-vegetable fruit integrity degrees corresponding to said plurality of target predetermined fruit-vegetable fruits based on said plurality of predetermined fruit-vegetable fruit images comprises:

determining an incomplete predetermined fruit and vegetable fruit image, wherein the incomplete predetermined fruit and vegetable fruit image is a predetermined fruit and vegetable fruit image with an angle at the edge of the image in the plurality of predetermined fruit and vegetable fruit images;

completing the incomplete predetermined fruit and vegetable image to obtain a complete predetermined fruit and vegetable image;

and determining the integrity of the target preset fruit and vegetable corresponding to the incomplete preset fruit and vegetable image according to the incomplete preset fruit and vegetable image and the supplemented preset fruit and vegetable image.

5. The method of claim 1, wherein said determining a first fruit edge of said single predetermined fruit-vegetable fruit and a second fruit edge of said stacked predetermined fruit-vegetable fruit in said target image comprises:

intercepting a fruit image according to the target image, wherein the fruit image comprises a single preset fruit and vegetable image and a stacked preset fruit and vegetable image;

determining a first fruit edge of a single predetermined fruit and vegetable fruit in the single predetermined fruit and vegetable fruit image and a second fruit edge of a stacked predetermined fruit and vegetable fruit in the stacked predetermined fruit and vegetable fruit image.

6. The method of claim 5, wherein determining a second fruit edge of the stacked predetermined fruit and vegetable fruit in the stacked predetermined fruit and vegetable fruit image comprises:

adjusting image chromaticity data of the stacked preset fruit and vegetable fruit images;

and determining a second fruit edge of the stacked preset fruit and vegetable in the stacked preset fruit and vegetable image according to the stacked preset fruit and vegetable image after the image chromaticity data is adjusted.

7. The method of any one of claims 1 to 6, further comprising:

determining a target two-dimensional position of a target predetermined fruit and vegetable image, wherein the target predetermined fruit and vegetable image is a predetermined fruit and vegetable image corresponding to the mature predetermined fruit and vegetable;

determining a target three-dimensional position of the mature predetermined fruit and vegetable according to the target two-dimensional position;

and sending a picking instruction to a preset terminal so that the preset terminal picks the mature preset fruit and vegetable at the target three-dimensional position.

8. A fruit handling device, comprising:

the device comprises an acquisition module, a storage module and a display module, wherein the acquisition module is used for acquiring a target image, and the target image comprises a single preset fruit and vegetable fruit and a stacked preset fruit and vegetable fruit;

a first determining module for determining a first fruit edge of the single predetermined fruit and vegetable fruit and a second fruit edge of the stacked predetermined fruit and vegetable fruit in the target image;

a dividing module, configured to divide a plurality of predetermined fruit and vegetable image according to the first fruit edge and the second fruit edge, where any one of the predetermined fruit and vegetable image includes a target predetermined fruit and vegetable, and the target predetermined fruit and vegetable included in the predetermined fruit and vegetable image is not repeated;

and the second determining module is used for determining mature preset fruits and vegetables from the preset fruit and vegetable images.

9. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the fruit handling method of any of claims 1 to 7.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the fruit handling method of any of claims 1-7.

Images