CN115995017A - A method, device and medium for fruit identification and positioning - Google Patents

Abstract

The invention discloses a fruit identification and positioning method, comprising the following steps: photographing the fruit under different lighting conditions, classifying the photographing results to obtain a training image data set; marking the images in the training image data set, and Carry out label setting to labeling result; Utilize described training image data set and labeling result to fruit target detection model to train; Collect the image of several pieces of fruit to be detected, carry out the fruit in described image through the fruit target detection model that training is completed Identify and locate, and obtain the maturity and location information of the fruit to be detected. The invention can effectively solve the problems of low accuracy, non-universality and high cost of data acquisition in the prior art.

Description

A fruit identification and positioning method, device and medium

Technical Field

The present invention relates to the technical field of fruit identification and positioning, and in particular to a fruit identification and positioning method, device and medium.

Background Art

Fruit recognition and positioning are the premise and basis for automatic harvesting. Existing fruit recognition and positioning methods, such as the fruit positioning method and device proposed in patent document CN111126296A, and a ripe pomegranate positioning method based on Mask R-CNN and 3D sphere fitting proposed in patent document CN112529948A, use threshold segmentation or instance segmentation to identify fruit targets in images. This method has complex algorithms, is easily affected by the environment, and requires a large amount of data to be processed, and cannot guarantee real-time performance.

Existing target technology uses the color, shape, texture and other information of fruits in color images to separate the target from the background in the image, so as to realize the recognition of fruits in the image. This method has strict requirements on the environment and is easily disturbed, resulting in omissions and recognition errors, which cannot meet the requirements of fruit recognition in orchards. For example, the light conditions in the orchard vary greatly under different weather conditions and at different times of the day; on the other hand, the fruits in the orchard grow on fruit trees, and there are close and occluded situations with leaves and branches, which makes the background of fruit images collected in the orchard very complex. Existing technology cannot well avoid the interference caused by the above factors, and the recognition accuracy in the orchard environment is low and not universal. The instance segmentation method needs to mark the outline points of the target when marking the data set, which is labor-intensive and inefficient. The above two recognition methods need to process a large amount of data, which is slow to process and cannot guarantee real-time performance. The point cloud data required for positioning is difficult to obtain and the cost is high.

Summary of the invention

The embodiments of the present invention provide a fruit identification and positioning method, device and medium, which can effectively solve the problems of low accuracy, non-universality and high data acquisition cost of the prior art.

An embodiment of the present invention provides a fruit identification and positioning method, comprising the following steps:

The fruits are photographed under different lighting conditions, and the photographing results are classified to obtain a training image dataset;

Annotating the images in the training image dataset and setting labels for the annotation results;

Using the training image data set and the annotation results to train a fruit target detection model;

Collect several images of fruits to be detected, identify and locate the fruits in the images through the trained fruit target detection model, and obtain the maturity and position information of the fruits to be detected.

Compared with the prior art, the fruit recognition and positioning method disclosed in the embodiment of the present invention is to shoot under different weather conditions to ensure the diversity of environmental conditions for image acquisition in the data set, so that the characteristics of litchi fruit targets under various conditions can be learned when training the orchard litchi target detection model, overcome the difficulties caused by light changes, and ensure that the target detection model can accurately identify litchi fruit targets under different environmental conditions. By combining the results of target detection and the depth image to locate the target, compared with the method of using point cloud data for positioning, the present invention only needs to use a depth sensor for shooting, which is low in cost and simple in data acquisition method.

Furthermore, photographing the fruits under different lighting conditions and classifying the photographing results to obtain a training image data set specifically includes:

A fixed number of fruit images are taken under various lighting conditions, and all the captured fruit images are classified according to the lighting conditions and combined into a training image dataset.

When making the litchi fruit image dataset, the images were taken under different weather conditions to ensure the diversity of environmental conditions for acquiring the images in the dataset. This allows the characteristics of litchi fruit targets in various situations to be learned when training the orchard litchi target detection model, overcoming the difficulties caused by light changes and ensuring that the target detection model can accurately identify litchi fruit targets under different environmental conditions.

Furthermore, the step of labeling the images in the training image data set and setting labels for the labeling results specifically includes:

The fruits in the image data set are annotated by using an annotation tool, the fruit area in the image is framed by a geometric frame, and the obtained geometric frames are labeled according to the maturity of the fruits in the image, and the label types include mature and unripe.

When marking data, you only need to set a geometric frame that surrounds the target, without drawing points on the target's outline, which reduces the workload during the marking process.

Furthermore, the training of the fruit target detection model using the training image dataset and the annotation results specifically includes:

Loading an image data set, inputting the training image data set and the annotation results into a fruit target detection model, obtaining initial model parameters and calculating initial losses after model operation, then continuously updating model parameters and calculating losses using back propagation iterations, and terminating training when model performance meets requirements to obtain a fruit target detection model that has been trained.

Among them, the fruit target detection model includes: a feature extraction network, a neck, and a detection part; the feature extraction network is composed of a convolutional neural network and an attention function, and the attention function is a multi-head attention function formed by splicing the scaled dot product attention function after parallel calculation for multiple times; the neck adopts two structures: a feature pyramid and a path aggregation network. The feature pyramid structure is used to overlap the high-level feature map and the low-level feature map through upsampling, and the path aggregation network is used to transfer the positioning information from the shallow layer to the deep layer; the detection part outputs a target detection output frame according to the feature map generated by the feature extraction network and the neck, and the output frame includes a number of prior frames and prediction frames, the prior frames are distributed in each pixel of the feature map and have different sizes, and the prediction frame is obtained by calculating the prior frame and the feature map.

As a preferred embodiment, the training is terminated when the model performance reaches the requirement, specifically including:

The model performance meets the requirements specifically as follows: the loss is less than the preset error value;

The loss is obtained by adding the positioning loss, confidence loss and classification loss, and is used to determine the error between the model prediction result of the current parameter and the actual situation. When the loss is less than the preset error value, the training ends.

Furthermore, the collecting of a plurality of images of fruits to be detected, identifying and locating the fruits in the images by using the trained fruit target detection model, and obtaining the position information of the fruits to be detected specifically includes:

Load the trained fruit object detection model, initialize the shooting parameters of the shooting device, and set the resolution of the captured image;

The photographing device is used to collect a plurality of images of the fruit to be inspected; wherein the images include color images and depth images, and the photographing device is specifically a depth sensor;

The fruit in the color image is detected using a fruit target detection model to obtain a plurality of target detection output frames, and the horizontal and vertical coordinates of the center points of the plurality of output frames in the color image are recorded respectively; wherein the target detection output frame also includes a label, and the label is divided into mature and unripe, and is used to identify the maturity of the fruit;

Obtaining depth values of center points of the plurality of output frames in the depth image;

The horizontal and vertical coordinates are combined with the depth value to obtain the position information of the fruit in the spatial coordinate system.

The target is located by combining the results of target detection and the depth image. Compared with the method of using point cloud data for positioning, this method only needs to use a depth sensor for shooting, which is low-cost and has a simple data acquisition method.

Another embodiment of the present invention provides a fruit recognition and positioning device, including: an image acquisition and annotation module, a model training module and a fruit recognition and positioning module;

The image acquisition and annotation module is used to shoot fruits under different lighting conditions, classify the shooting results, obtain a training image data set, annotate the images in the training image data set, and set labels for the annotation results;

The model training module is used to train the fruit target detection model using the training image data set and the annotation results;

The fruit recognition and positioning module is used to collect a number of images of fruits to be detected, identify and locate the fruits in the images through the trained fruit target detection model, and obtain the maturity and position information of the fruits to be detected.

Compared with the prior art, the fruit recognition and positioning device disclosed in the embodiment of the present invention takes pictures under different weather conditions to ensure the diversity of environmental conditions for image acquisition in the data set, so that the characteristics of litchi fruit targets under various conditions can be learned when training the orchard litchi target detection model, overcoming the difficulties caused by light changes, and ensuring that the target detection model can accurately identify litchi fruit targets under different environmental conditions. By combining the results of target detection and the depth image to locate the target, compared with the method of using point cloud data for positioning, this device only needs to use a depth sensor for shooting, which is low in cost and simple in data acquisition method.

Furthermore, the fruit recognition and positioning module is used to collect a number of images of fruits to be detected, identify and locate the fruits in the images through the trained fruit target detection model, and obtain the maturity and position information of the fruits to be detected, specifically including:

Load the trained fruit object detection model, initialize the shooting parameters of the shooting device, and set the resolution of the captured image;

The photographing device is used to collect a plurality of images of the fruit to be inspected; wherein the images include color images and depth images, and the photographing device is specifically a depth sensor;

The fruit in the color image is detected using a fruit target detection model to obtain a plurality of target detection output frames, and the horizontal and vertical coordinates of the center points of the plurality of output frames in the color image are recorded respectively; wherein the target detection output frame also includes a label, and the label is divided into mature and unripe, and is used to identify the maturity of the fruit;

Obtaining depth values of center points of the plurality of output frames in the depth image;

The horizontal and vertical coordinates are combined with the depth value to obtain the position information of the fruit in the spatial coordinate system.

Another embodiment of the present invention provides a fruit identification and positioning device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, the fruit identification and positioning method described in the above-mentioned embodiment of the invention is implemented.

Another embodiment of the present invention provides a storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the fruit identification and positioning method described in the above-mentioned embodiment of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flow chart of a fruit identification and positioning method provided in accordance with an embodiment of the present invention.

FIG. 2 is a schematic diagram of a network structure of a fruit target detection model provided by an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of a fruit identification and positioning device provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Referring to FIG. 1 , it is a schematic flow chart of a fruit identification and positioning method provided by an embodiment of the present invention, comprising:

S101: photographing fruits under different lighting conditions, classifying the photographing results, and obtaining a training image data set;

S102: annotating images in the training image data set, and setting labels for the annotated results;

S103: training a fruit object detection model using the training image dataset and the annotation results;

S104: Collect several images of fruits to be detected, identify and locate the fruits in the images through the trained fruit target detection model, and obtain the maturity and position information of the fruits to be detected.

A fruit recognition and positioning method provided by an embodiment of the present invention is to shoot under different weather conditions to ensure the diversity of environmental conditions for image acquisition in the data set, so that the characteristics of litchi fruit targets under various conditions can be learned when training the orchard litchi target detection model, overcome the difficulties caused by light changes, and ensure that the target detection model can accurately identify litchi fruit targets under different environmental conditions. By combining the target detection results and the depth image to locate the target, compared with the method of using point cloud data for positioning, the present invention only needs to use a depth sensor for shooting, which is low in cost and simple in data acquisition method.

For step S101, specifically, a fixed number of fruit images are taken under various lighting conditions, and all the taken fruit images are classified according to the lighting conditions and then combined into a training image data set.

In a preferred embodiment, on a sunny day, images under direct sunlight are obtained by shooting in the front light, and images under side light are obtained by shooting in the side light; images under low brightness are obtained by shooting in the evening; and images under diffuse light conditions are obtained on a cloudy day. When preparing the data set, it is ensured that the number of images shot in the above four environments of direct sunlight, photometry, low brightness, and diffuse light in the training data set is equal.

For step S102, specifically, the fruits in the image data set are annotated by using an annotation tool, the fruit area in the image is framed by a geometric frame, and the obtained geometric frames are labeled according to the maturity of the fruits in the image, and the label types include mature and unripe.

In a preferred embodiment, the fruits in the images of the training data set are manually annotated. When annotating, only a rectangular frame surrounding the target is required, and no points are drawn on the outline of the target. The fruit area in the image is annotated with an annotation tool using a rectangular frame to obtain a real frame and set a corresponding label. The label types include ripe and unripe to distinguish ripe and unripe fruits.

For step S103, specifically, an image data set is loaded, and the training image data set and the annotation results are input into the fruit target detection model. After the model operation, the initial model parameters are obtained and the initial loss is calculated. Then, the model parameters are continuously updated and the loss is calculated by back propagation iteration. When the model performance meets the requirements, the training is terminated to obtain the fruit target detection model that has been finally trained.

Among them, the fruit target detection model includes: a feature extraction network, a neck, and a detection part; the feature extraction network is composed of a convolutional neural network and an attention function, and the attention function is a multi-head attention function formed by splicing the scaled dot product attention function after parallel calculation for multiple times; the neck adopts two structures: a feature pyramid and a path aggregation network. The feature pyramid structure is used to overlap the high-level feature map and the low-level feature map through upsampling, and the path aggregation network is used to transfer the positioning information from the shallow layer to the deep layer; the detection part outputs a target detection output frame according to the feature map generated by the feature extraction network and the neck, and the output frame includes a number of prior frames and prediction frames, the prior frames are distributed in each pixel of the feature map and have different sizes, and the prediction frame is obtained by calculating the prior frame and the feature map.

In a preferred embodiment, the model is trained using a back propagation iteration method to obtain model parameters suitable for orchard litchi target detection. The training steps include loading data, building a model, updating model parameters, calculating losses, evaluating the model, judging the conditions for ending training, and saving model parameters. Among them, the conditions for judging technical training are "model performance meets the requirements or the number of training times is greater than the set value", and the requirement is "the change value of the loss function is less than the set value".

In particular, the calculation loss uses an improved target detection loss function, including positioning loss, confidence loss, and classification loss, which reflects the error between the model prediction result using the current parameters and the actual situation. The calculation method is:

Loss＝Loss _cls +Loss _obj +Loss _box

The classification loss and confidence loss use the binary cross entropy loss function, and the calculation method is expressed as:

In the formula, p represents the predicted value, x represents the sample, y represents the target value, n represents the total number of samples, and L represents the final calculation result of the binary cross entropy loss.

The positioning loss uses α-CIoU loss Loss _α-CIoU , which is calculated as follows:

In the formula, A and B represent the output box and the true box respectively, |A∩B| represents the area of the intersection of A and B, |A∪B| represents the area of the union of A and B, and C represents the area of the minimum rectangle enclosing A and B. α is an adjustable parameter. Comparing the detection results at different values to determine the value of α can improve the flexibility of debugging the target detection model. b and b ^gt are the center points of the output box and the true box respectively, ρ(·) is the Euclidean distance, and c is the diagonal length of the minimum enclosing box of the two boxes. β is a positive trade-off parameter, and v measures the consistency of the aspect ratio. The calculation methods of β and v are respectively expressed as:

Where w ^gt and h ^gt are the width and height of the real box, respectively, and w and h are the width and height of the output box, respectively.

In a preferred embodiment, the fruit target detection model is an orchard target detection model based on improved YOLOv5, including a feature extraction network, a neck, and a detection part. See Figure 2 for the specific network structure.

In particular, the feature extraction network is a "convolution-attention structure" composed of a convolutional neural network and an attention function. The convolutional neural network constitutes three structures of the feature extraction network: Conv (Convolution), SPP (Spatial Pyramid Pooling) and CSP bottleneck layer. Conv includes a convolution layer, a batch normalization layer, and an activation function. The activation function is a leaky linear rectification function. The CSP bottleneck layer includes a convolution layer, batch normalization, an activation function, and a residual network structure. The attention function uses a scaled dot product attention function, and the calculation method is expressed as:

为比例因子。In the formula, Q, K, V are the input feature maps, K ^T represents the transposed matrix of K,

is the scale factor.

The scaled dot product attention function is calculated multiple times in parallel and concatenated to form a multi-head attention function. The calculation method is expressed as:

MultiHead(Q,K,V)＝Concat(head ₁ ,…,head _□ )W ^O

Where head is the output of the scaled dot product attention function.

The multi-head attention function and convolutional neural network are combined to form a feature extraction network.

The neck adopts two structures: feature pyramid and path aggregation network. The feature pyramid adopts a top-down approach to overlap high-level feature maps with low-level feature maps through upsampling. The path aggregation network adopts a bottom-up approach to transfer positioning information from shallow layers to deep layers.

The detection part outputs the target detection output box based on the feature extraction network and the feature map generated by the neck. Multiple boxes of different sizes are generated in each pixel of the feature map, called prior boxes. The sizes of the prior boxes are 10×13, 16×30, 33×23, 30×61, 62×45, 59×119, 116×90, 156×198, and 373×326. The prediction box is obtained by calculating the prior box and the feature map. The calculation method is:

b _x =σ(t _x )+c _x

_by =σ( _ty )+ _cy

Where σ(t _x ) and σ(t _y ) are the offsets based on the coordinates of the upper left corner of the grid center, and σ is the sigmoid function. p _w , p _□ are the width and height of the prior box. _{b x} , _by , b _w , and b _□ are the horizontal coordinate, vertical coordinate, width, and height of the center point of the predicted box, respectively.

For step S104, specifically, the collecting of a plurality of images of fruits to be detected, identifying and locating the fruits in the images by using the trained fruit target detection model, and obtaining the position information of the fruits to be detected specifically includes:

Load the trained fruit object detection model, initialize the shooting parameters of the shooting device, and set the resolution of the captured image;

The photographing device is used to collect a plurality of images of the fruit to be inspected; wherein the images include color images and depth images, and the photographing device is specifically a depth sensor;

The fruit in the color image is detected using a fruit target detection model to obtain a plurality of target detection output frames, and the horizontal and vertical coordinates of the center points of the plurality of output frames in the color image are recorded respectively; wherein the target detection output frame also includes a label, and the label is divided into mature and unripe, and is used to identify the maturity of the fruit;

Obtaining depth values of center points of the plurality of output frames in the depth image;

The horizontal and vertical coordinates are combined with the depth value to obtain the position information of the fruit in the spatial coordinate system.

In a preferred embodiment, an Intel RealSense D435 depth sensor is used as a camera, the parameters of the camera are initialized, and the resolution of the acquired color image and depth image is set to 640×480. The depth sensor is used to collect a color image and a depth image in front of the fruit, and the target detection model is used to detect the fruit in the color image, obtain the target detection output frame, and record the coordinates (x, z) of the center point of the output frame in the color image. Then, the depth value of the midpoint (x, z) of the depth image is obtained as the distance y between the fruit and the shooting point, and (x, y, z) represents the position information of the fruit in the spatial coordinate system.

3 is a schematic diagram of the structure of a fruit recognition and positioning device provided by an embodiment of the present invention, comprising: an image acquisition and annotation module 201, a model training module 202 and a fruit recognition and positioning module 203;

The image acquisition and annotation module 201 is used to shoot fruits under different lighting conditions, classify the shooting results, obtain a training image data set, annotate the images in the training image data set, and set labels for the annotation results;

The model training module 202 is used to train the fruit object detection model using the training image data set and the annotation results;

The fruit recognition and positioning module 203 is used to collect a number of images of fruits to be detected, and to recognize and locate the fruits in the images through the trained fruit target detection model to obtain the maturity and position information of the fruits to be detected.

The fruit recognition and positioning device provided by the embodiment of the present invention is shot under different weather conditions to ensure the diversity of environmental conditions for image acquisition in the data set, so that the characteristics of litchi fruit targets under various conditions can be learned when training the orchard litchi target detection model, overcoming the difficulties caused by light changes, and ensuring that the target detection model can accurately identify litchi fruit targets under different environmental conditions. By combining the target detection results and the depth image to locate the target, compared with the method of using point cloud data for positioning, the device only needs to use a depth sensor for shooting, which is low in cost and simple in data acquisition method.

Furthermore, the fruit recognition and positioning module 203 is used to collect a number of images of fruits to be detected, and to recognize and locate the fruits in the images through the trained fruit target detection model to obtain the maturity and position information of the fruits to be detected, which specifically includes:

Load the trained fruit object detection model, initialize the shooting parameters of the shooting device, and set the resolution of the captured image;

The photographing device is used to collect a plurality of images of the fruit to be inspected; wherein the images include color images and depth images, and the photographing device is specifically a depth sensor;

The fruit in the color image is detected using a fruit target detection model to obtain a plurality of target detection output frames, and the horizontal and vertical coordinates of the center points of the plurality of output frames in the color image are recorded respectively; wherein the target detection output frame also includes a label, and the label is divided into mature and unripe, and is used to identify the maturity of the fruit;

Obtaining depth values of center points of the plurality of output frames in the depth image;

The horizontal and vertical coordinates are combined with the depth value to obtain the position information of the fruit in the spatial coordinate system.

An embodiment of the present invention also provides a fruit identification and positioning device. The fruit identification and positioning device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned various fruit identification and positioning method embodiments are implemented, such as step S101 shown in Figure 1. Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are implemented, such as the fruit identification and positioning module 203.

Exemplarily, the computer program can be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the fruit identification and positioning device. For example, the computer program can be divided into an image acquisition and annotation module 201, a model training module 202, and a fruit identification and positioning module 203, and the specific functions of each module are as follows:

The image acquisition and annotation module 201 is used to shoot fruits under different lighting conditions, classify the shooting results, obtain a training image data set, annotate the images in the training image data set, and set labels for the annotation results;

The model training module 202 is used to train the fruit object detection model using the training image data set and the annotation results;

The fruit recognition and positioning module 203 is used to collect a number of images of fruits to be detected, and to recognize and locate the fruits in the images through the trained fruit target detection model to obtain the maturity and position information of the fruits to be detected.

The fruit identification and positioning device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The fruit identification and positioning device can include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram is only an example of a fruit identification and positioning device and does not constitute a limitation on the fruit identification and positioning device. It can include more or less components than shown in the figure, or combine certain components, or different components. For example, the fruit identification and positioning device can also include input and output devices, network access devices, buses, etc.

The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the fruit identification and positioning device, and uses various interfaces and lines to connect the various parts of the entire fruit identification and positioning device.

The memory can be used to store the computer program or module, and the processor realizes various functions of the fruit identification and positioning device by running or executing the computer program or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

Wherein, if the module integrated with the fruit identification and positioning device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signal and telecommunication signal.

It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without creative work.

The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims (10)

1. The fruit identifying and positioning method is characterized by comprising the following steps:

shooting fruits under different illumination conditions, and classifying shooting results to obtain a training image dataset;

labeling the images in the training image data set, and setting labels for labeling results;

training the fruit target detection model by utilizing the training image data set and the labeling result;

and acquiring images of a plurality of fruits to be detected, and identifying and positioning the fruits in the images through a fruit target detection model after training to obtain the maturity and position information of the fruits to be detected.

2. The method for identifying and positioning fruits according to claim 1, wherein the steps of photographing fruits under different illumination conditions, classifying photographing results, and obtaining a training image dataset comprise:

and respectively shooting a fixed number of fruit images under various illumination conditions, classifying all shot fruit images according to the illumination conditions, and combining the fruit images into a training image data set.

3. The method for identifying and positioning fruits according to claim 1, wherein the steps of labeling the images in the training image dataset and labeling the labeling result comprise:

labeling fruits in the image data set through a labeling tool, framing out fruit areas in the image by using geometric figure frames, and setting labels according to the maturity of the fruits in the image, wherein the label types comprise ripeness and immature.

4. The method for identifying and locating fruits according to claim 1, wherein training the fruit target detection model by using the training image dataset and the labeling result comprises:

loading an image dataset, inputting the training image dataset and a labeling result into a fruit target detection model, obtaining initial model parameters and calculating initial loss after model operation, continuously updating the model parameters and calculating the loss by using a back propagation iteration mode, and ending training when the model performance meets the requirement to obtain a fruit target detection model after final training is completed;

wherein, the fruit target detection model comprises: a feature extraction network, a neck, and a detection portion; the feature extraction network is composed of a convolutional neural network and attention functions, wherein the attention functions are multi-head attention functions formed by performing parallel calculation on the dot product scaling attention functions for a plurality of times and then splicing the dot product scaling attention functions; the neck adopts two structures of a feature pyramid structure and a path aggregation network, the feature pyramid structure is used for overlapping high-level feature mapping and low-level feature mapping through up-sampling, and the path aggregation network is used for transmitting positioning information from a shallow layer to a deep layer; the detection part outputs a target detection output frame according to the feature image generated by the feature extraction network and the neck, the output frame comprises a plurality of prior frames and a prediction frame, the prior frames are distributed in each pixel of the feature image and have different sizes, and the prediction frame is obtained through calculation of the prior frames and the feature image.

5. The method for identifying and locating fruits according to claim 4, wherein said training is finished when the model performance meets the requirement, specifically comprising:

the model performance meets the requirements specifically as follows: the loss is less than a preset error value;

the loss is obtained by adding the positioning loss, the confidence coefficient loss and the classification loss, is used for judging the error between the model prediction result of the current parameter and the real situation, and ends training when the loss is smaller than a preset error value.

6. The method for identifying and positioning fruits according to claim 1, wherein the collecting a plurality of images of fruits to be detected, identifying and positioning fruits in the images by a trained fruit target detection model, and obtaining position information of the fruits to be detected specifically comprises:

loading a trained fruit target detection model, initializing shooting parameters of shooting equipment, and setting resolution of a shot image;

collecting a plurality of images of fruits to be detected through the shooting equipment; the image comprises a color image and a depth image, and the shooting equipment is specifically a depth sensor;

detecting fruits in the color image by using a fruit target detection model to obtain a plurality of target detection output frames, and respectively recording the horizontal coordinates and the vertical coordinates of the central points of the plurality of output frames in the color image; the target detection output frame further comprises labels, wherein the labels are divided into ripe and unripe labels and are used for identifying the ripeness of fruits;

obtaining depth values of central points of the plurality of output frames in the depth image;

and combining the abscissa and the ordinate with the depth value to obtain the position information of the fruit in the space coordinate system.

7. A fruit identification and positioning device, comprising: the device comprises an image acquisition and labeling module, a model training module and a fruit identification and positioning module;

the image acquisition and labeling module is used for shooting fruits under different illumination conditions, classifying shooting results to obtain a training image data set, labeling images in the training image data set, and setting labels for labeling results;

the model training module is used for training a fruit target detection model by utilizing the training image data set and the labeling result;

the fruit recognition and positioning module is used for collecting images of a plurality of fruits to be detected, recognizing and positioning the fruits in the images through the trained fruit target detection model, and obtaining maturity and position information of the fruits to be detected.

8. The fruit identification and positioning device according to claim 7, wherein the fruit identification and positioning module is configured to collect images of a plurality of fruits to be detected, and identify and position the fruits in the images through a trained fruit target detection model, so as to obtain maturity and position information of the fruits to be detected, and the fruit identification and positioning device specifically comprises:

loading a trained fruit target detection model, initializing shooting parameters of shooting equipment, and setting resolution of a shot image;

collecting a plurality of images of fruits to be detected through the shooting equipment; the image comprises a color image and a depth image, and the shooting equipment is specifically a depth sensor;

detecting fruits in the color image by using a fruit target detection model to obtain a plurality of target detection output frames, and respectively recording the horizontal coordinates and the vertical coordinates of the central points of the plurality of output frames in the color image; the target detection output frame further comprises labels, wherein the labels are divided into ripe and unripe labels and are used for identifying the ripeness of fruits;

obtaining depth values of central points of the plurality of output frames in the depth image;

and combining the abscissa and the ordinate with the depth value to obtain the position information of the fruit in the space coordinate system.

9. A fruit identification and locating device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the fruit identification and locating method according to any one of claims 1 to 6 when executing the computer program.

10. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored computer program, wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the fruit identification and localization method according to any one of claims 1 to 6.

Images