CN111951233B - Fishbone residue detection method and system - Google Patents
Fishbone residue detection method and system Download PDFInfo
- Publication number
- CN111951233B CN111951233B CN202010724230.8A CN202010724230A CN111951233B CN 111951233 B CN111951233 B CN 111951233B CN 202010724230 A CN202010724230 A CN 202010724230A CN 111951233 B CN111951233 B CN 111951233B
- Authority
- CN
- China
- Prior art keywords
- detection
- fishbone
- image
- fish
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 312
- 241000251468 Actinopterygii Species 0.000 claims abstract description 100
- 238000012549 training Methods 0.000 claims abstract description 77
- 238000000034 method Methods 0.000 claims description 37
- 210000000988 bone and bone Anatomy 0.000 claims description 34
- 235000013372 meat Nutrition 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 210000003484 anatomy Anatomy 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 abstract description 7
- 238000004458 analytical method Methods 0.000 abstract description 6
- 238000004422 calculation algorithm Methods 0.000 abstract description 4
- 235000019688 fish Nutrition 0.000 description 63
- 241000277263 Salmo Species 0.000 description 17
- 238000003860 storage Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000011176 pooling Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 235000013305 food Nutrition 0.000 description 3
- 241000972773 Aulopiformes Species 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 210000001035 gastrointestinal tract Anatomy 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 235000019515 salmon Nutrition 0.000 description 2
- 235000002566 Capsicum Nutrition 0.000 description 1
- 239000006002 Pepper Substances 0.000 description 1
- 235000016761 Piper aduncum Nutrition 0.000 description 1
- 235000017804 Piper guineense Nutrition 0.000 description 1
- 244000203593 Piper nigrum Species 0.000 description 1
- 235000008184 Piper nigrum Nutrition 0.000 description 1
- 210000002808 connective tissue Anatomy 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000001094 effect on targets Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 235000013622 meat product Nutrition 0.000 description 1
- 201000001231 mediastinitis Diseases 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000004400 mucous membrane Anatomy 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 235000016709 nutrition Nutrition 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 150000003839 salts Chemical class 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30128—Food products
Landscapes
- Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a fishbone residue detection method and a fishbone residue detection system, comprising the following steps: acquiring a fish original image, and dividing a detection sub-image with detection value on the fish original image; inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the fishbone original image according to an output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label. According to the fishbone residue detection method and the fishbone residue detection system, the fishbone detection network model is trained by using the deep learning target detection algorithm, and the high-value areas in the images are segmented according to the original fish images and are used as detection sub-images to be input into the fishbone detection network model, so that the fishbone residue analysis result can be directly obtained, the real-time accurate detection of the fishbone residue can be realized, the labor intensity is reduced, and the labor cost is reduced.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a fishbone residue detection method and a fishbone residue detection system.
Background
The fish bone is an important food-borne foreign body which is frequently present in fish foods. Careless intake of fishbone by consumers can lead to complications such as damage of digestive tract mucous membrane, perforation of digestive tract, mediastinitis and the like, and even life-threatening in severe cases. Because of good mouthfeel, color, and nutritional value, china has become one of the main importation countries for Atlantic salmon.
Because of the density characteristic of the fishbone, the detection is generally performed by adopting an X-ray detection technology and a hyperspectral detection technology in the prior art. However, the X-ray detection technology is easy to confuse the fishbone with connective tissues and muscle tissues in the fish meat product, so that missed detection and false detection are caused; and strong light sources adopted by the hyperspectral detection technology are easy to damage the surfaces of fish food products.
Therefore, most processing enterprises in China currently hire a large amount of labor force to manually remove the bone remained after the cut of the fish fillets, and the bone remained of the Atlantic salmon is inspected one by one mainly in a mode of human eye identification and manual touch.
In view of this, developing an efficient, accurate and rapid automated fishbone detection method is a technical problem to be solved.
Disclosure of Invention
The embodiment of the invention provides a fishbone residue detection method and a fishbone residue detection system, which are used for solving the defects of low detection efficiency and poor detection precision in the fishbone detection process in the prior art.
In a first aspect, an embodiment of the present invention provides a method for detecting fishbone residues, which mainly includes: acquiring a fish original image, and dividing a detection sub-image with detection value on the fish original image; inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the original fish image according to the output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
Optionally, before inputting the detection sub-image into the fishbone detection network model, the method further comprises: acquiring a plurality of original fish sample images; respectively acquiring at least one detection sample sub-image in each fish original sample image; setting a fishbone residual result label for each detection sample sub-image; combining each detection sample sub-image with the corresponding fishbone residual result label to be used as a training sample, and constructing a training sample set; and pre-training the fishbone detection network model by using a training sample set.
Optionally, before setting a fishbone residual result label for each detection sample sub-image, the method further comprises: expanding the sub-image of the detection sample by using geometric transformation based on image processing; and denoising and filtering each expanded detection sample sub-image, and then constructing a detection sample sub-image set.
Optionally, before the training sample set is used to pretrain the fishbone detection network model, the method may further include: setting an intersection ratio of each training sample for pre-training of the fishbone detection network model; the cross ratio is the overlapping rate of the image area and the mark image area in the detection frame of the fishbone detection network model.
Optionally, pre-training the fishbone detection network model using the training sample set may include: training the fishbone detection network model by utilizing each training sample in sequence; setting a confidence threshold value for selecting and rejecting a detection frame generated by the fishbone detection network model in each training process; if the confidence coefficient value of the detection frame is larger than the confidence coefficient threshold value, training the fishbone detection network model by using the detection sample sub-image and the fishbone residual result label corresponding to the detection frame; if the confidence coefficient value of the detection frame is smaller than the confidence coefficient threshold value, discarding the sub-image of the detection sample and the fishbone residual result label corresponding to the detection frame, and continuing to train the fishbone detection network model by using the next training sample.
Alternatively, the confidence threshold includes: an accuracy threshold, a recall threshold, an F1 score threshold, an average accuracy threshold, and an average detection time threshold; a confidence value for a detection frame, comprising: accuracy, recall, F1 score, average accuracy, and average detection time.
Alternatively, the calculation method of the accuracy is:
The recall rate calculation method comprises the following steps:
The calculation method of the F1 fraction comprises the following steps:
the average precision is the area under the curve of the precision rate and the calling rate;
The calculation method of the average detection time comprises the following steps:
Wherein, For accuracy, TP is the number of fishbones successfully identified by the model, FP is the number of fishbones incorrectly identified by the model as non-fishbone portions, FN is the number of fishbones not detected by the model,/>Is the recall rate; F1-Score represents the F1 Score, which is the harmonic mean of accuracy and recall; AVERAGE TIME is the average detection time,/>To detect total time,/>Is the total number of samples tested.
Alternatively, the fishbone detection network model is constructed with Faster-RCNN as the target detection framework and VGG19 as the target detection base network.
In a second aspect, an embodiment of the present invention further provides a fishbone residue detection system, which mainly includes: an image acquisition unit and a model operation unit; the image acquisition unit is mainly used for acquiring an original fish image and dividing a detection sub-image with detection value on the original fish image; the model operation unit is mainly used for inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the fishbone original image according to the output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the steps of any one of the above-mentioned fishbone residual detecting methods are implemented when the processor executes the program.
In a fourth aspect, embodiments of the present invention also provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a fishbone residual detection method as described in any of the above.
According to the fishbone residue detection method and the fishbone residue detection system, the fishbone detection network model is trained by using the deep learning target detection algorithm, and the high-value areas in the images are segmented according to the original fish images and are used as detection sub-images to be input into the fishbone detection network model, so that the fishbone residue analysis result can be directly obtained, the real-time accurate detection of the fishbone residue can be realized, the labor intensity is reduced, and the labor cost is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a fishbone residue detection method according to an embodiment of the present invention;
FIG. 2 is a photograph of Atlantic salmon meat with fish bone residue collected in accordance with an embodiment of the present invention;
FIG. 3 is a detection sub-image segmented based on FIG. 2;
FIG. 4 is a schematic illustration of the effect of labeling according to a sample;
FIG. 5 is a schematic representation of the results of Atlantic salmon residual fishbone detected using a fishbone detection network model;
fig. 6 is a schematic structural diagram of a fishbone residual detecting system according to an embodiment of the invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a schematic flow chart of a method for detecting fishbone residue according to an embodiment of the invention, as shown in fig. 1, the method includes but is not limited to the following steps:
Step S1: acquiring a fish original image, and dividing a detection sub-image with detection value on the fish original image;
step S2: inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the original fish image according to the output result of the fishbone detection network model;
The fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
It should be noted that, in all embodiments of the present invention, the detection of the fishbone residue in the fish of the atlantic salmon is taken as an example for convenience of description, and the description will not be repeated.
Specifically, the method for detecting fishbone residue provided by the embodiment of the invention firstly shoots an image (original fish image) of Atlantic salmon flesh to be detected, as shown in fig. 2. Considering that the size and image quality of the fish bone in the image will directly affect the detection accuracy, a simple mechanical device may be used in this embodiment to fix the relative spatial position of the fish bone and the shooting camera and set the camera parameters. Wherein the lens distance of the camera is ensured to be a fixed value, and the camera is kept vertical to the bottom plate for placing the fish meat. In addition, various camera parameters, such as focal length, exposure time, maximum aperture, resolution of the acquired image, etc., are set to ensure that the image of Atlantic salmon flesh in the captured image is clear and the details are apparent.
Further, a high value region within the acquired image is determined and segmented according to the physiological structure of salmon. Considering that the distribution of the fish bones on the atlantic salmon fillet conforms to the physiological structure thereof, the specific distribution position of the atlantic salmon bones on the fillet is obtained according to the physiological anatomy principle to cut the obtained original image of the fish meat and obtain a high-value area image, i.e., a detection sub-image, as shown in fig. 3.
Further, in step S2, a deep learning analysis is performed on the detected sub-images inputted into the model by using the pre-trained fish bone detection network model, so as to obtain a fish bone residual result corresponding to the inputted detected sub-images, i.e. a fish bone residual detection result of the atlantic salmon meat to be detected can be derived therefrom.
It should be noted that, the fishbone detection network model provided by the embodiment of the invention is obtained after training according to the detection sample sub-image with the fishbone residual result label.
Optionally, the fishbone detection network model provided in the embodiment of the invention may include a convolution layer, a pooling layer, a full connection layer, a logistic regression layer, and the like, where the detection sub-image is input to the convolution layer and the pooling layer of the preset neural network, the feature extraction is performed on the detection sub-image by using the convolution layer and the pooling layer, the two-dimensional feature vector corresponding to the detection sub-image is output, the two-dimensional feature vector is input to the full connection layer of the preset neural network, the two-dimensional feature vector is converted into a one-dimensional feature vector by using the full connection layer, and the one-dimensional feature vector is output; and finally, inputting the one-dimensional feature vector into a logistic regression layer of the neural network, outputting the prediction probability of the fishbone residue corresponding to the detection sub-image, and obtaining a fishbone residue result diagram corresponding to the fish original image according to the prediction probability.
According to the fishbone residue detection method provided by the embodiment of the invention, the fishbone detection network model is trained by using the deep learning target detection algorithm, and the high-value area in the image is segmented according to the original image of the fish meat and is used as a detection sub-image to be input into the fishbone detection network model, so that the fishbone residue analysis result can be directly obtained, the real-time accurate detection of the fishbone residue can be realized, the labor intensity is reduced, and the labor cost is reduced.
Based on the foregoing embodiment, as an alternative embodiment, before inputting the detection sub-image into the fishbone detection network model, the method further includes:
Acquiring a plurality of original fish sample images; respectively acquiring at least one detection sample sub-image in each fish original sample image; setting a fishbone residual result label for each detection sample sub-image; combining each detection sample sub-image with the corresponding fishbone residual result label to be used as a training sample, and constructing a training sample set; and pre-training the fishbone detection network model by using a training sample set.
Specifically, before the detected sub-image corresponding to each fish original image is input to the fishbone detecting network model, the fishbone detecting network model is further required to be pre-trained, and the specific training process is as follows:
first, a large number of raw images of atlantic salmon flesh are taken to obtain a plurality of raw sample images of flesh, i.e., an image as shown in fig. 2. And (3) calibrating the high-value area of each fish original sample image respectively, and acquiring one or more detection sample sub-images corresponding to the high-value area, wherein the detection sample sub-images are any image schematic diagram as shown in fig. 3.
The fishbone residual result label corresponding to each detection sample sub-image is established, namely the fishbone residual result corresponding to each detection sample sub-image is known and marked by the fishbone residual result label, as shown in fig. 4. The specific implementation process of obtaining the detection sample sub-image from the fish original sample image may be referred to the above method embodiment, and will not be described herein.
Further, the combination of the detection sample sub-image corresponding to each fish original sample image and the fishbone residual result label is used as a training sample, namely, each detection sample sub-image with the fishbone residual result label is used as a training sample, so that a plurality of training samples can be obtained. After a plurality of training samples are obtained, the training samples are sequentially input into the fishbone detection network model, namely, the sub-images of the detection samples and the fishbone residual result labels in each training sample are simultaneously input into the fishbone detection network model, each output result according to the fishbone detection network model is obtained, model parameters in the fishbone detection network model are adjusted, and finally the training process of the fishbone detection network model is completed. For example, the model parameters in the fishbone detection network model are adjusted by using the sample sub-image with the marked sample shown in fig. 4 and the residual fishbone result diagram shown in fig. 5, so as to realize one training of the model.
Alternatively, the corresponding training set and test set may be designed based on the constructed training sample set. And labeling the fishbone in the image according to the photographed original sample image of the fish meat to obtain a training sample set. And then dividing the training sample set into two independent parts, namely a training set and a testing set, based on a random distribution principle. In the dividing process, too many samples with the same type, the same position and the same number are avoided in the same data set as much as possible, so that the robustness of the fishbone detection network model is improved.
According to the fishbone residue detection method provided by the embodiment of the invention, the fishbone detection network model is pre-trained based on the deep learning thought, so that the fishbone detection network model learns the characteristics of the detection sample sub-images corresponding to different fishbone residue degrees, and the detection of the fishbone residue result by using the trained fishbone detection network model is facilitated, so that the prediction precision of the fishbone detection network model is improved.
Based on the foregoing, as an alternative embodiment, before setting a fishbone residual result tag for each detection sample sub-image, the method may further include: expanding the sub-image of the detection sample by using geometric transformation based on image processing; and denoising and filtering each expanded detection sample sub-image, and then constructing a detection sample sub-image set.
Specifically, in the embodiment of the present invention, data expansion may be further performed on each acquired sub-image of the detection sample to increase the number of sample images in the training sample set. Alternatively, geometric transformations based on image processing, such as: the expansion of the sample image is performed by random cropping, random rotation, random flipping, etc. Furthermore, proper noise and filtering means can be adopted to remove the defects of Gaussian noise, salt and pepper noise, blurring and the like in each detection sample sub-image, namely, after the detection sample sub-image is processed, fishbone in each sub-image is labeled by using a label, and finally, a standardized training sample set is obtained.
According to the fishbone residual detection method provided by the embodiment of the invention, the training quality is effectively improved and the model prediction precision is improved by expanding the detection sample sub-image and preprocessing the image.
Based on the foregoing embodiment, as an optional embodiment, before the training sample set is used to pretrain the fishbone detecting network model, the method further includes: setting an intersection ratio of each training sample for pre-training of the fishbone detection network model; the cross ratio is the overlapping rate of the image area and the mark image area in the detection frame of the fishbone detection network model.
The intersection ratio is the overlapping rate of the image area in the model detection frame and the label mark image area, and the specific expression is as follows:
wherein DetectionResult represents the image region within the detection frame, groundTruth represents the image region within the annotation frame
Alternatively, the overlap ratio (IOU) may be set to 0.4. The fishbone residue detection method provided by the embodiment of the invention is the same as that of setting the cross-over ratio so as to improve the effect of pre-training the fishbone detection network model by using the training sample set.
Based on the foregoing, as an optional embodiment, the pre-training the fishbone detecting network model using the training sample set specifically includes:
training the fishbone detection network model by using each training sample in sequence; setting a confidence threshold value for selecting and rejecting a detection frame generated by the fishbone detection network model in each training process; if the confidence coefficient value of the detection frame is larger than the confidence coefficient threshold value, training the fishbone detection network model by using the detection sample sub-image and the fishbone residual result label corresponding to the detection frame; if the confidence coefficient value of the detection frame is smaller than the confidence coefficient threshold value, discarding the sub-image of the detection sample and the fishbone residual result label corresponding to the detection frame, and continuing to train the fishbone detection network model by the next training sample.
When the fishbone residue detection method provided by the embodiment of the invention is used for carrying out fishbone residue detection by utilizing the fishbone detection network model, the network model is firstly required to be pre-trained, and in the process, the optimal confidence threshold value is optimized, so that the detection model has the optimal detection effect.
The confidence threshold is a criterion value required for model training, which is set according to actual conditions, such as detection accuracy.
In the training process, the detection frames generated by the detection of the fishbone detection network model have confidence values, if the confidence values are lower than the confidence threshold value, the detection frames are directly cleared, and if the confidence values are higher than the confidence threshold value, the detection frames are reserved.
Alternatively, the confidence threshold mainly includes: an accuracy threshold, a recall threshold, an F1 score threshold, an average accuracy threshold, and an average detection time threshold; the confidence of the detection frame mainly comprises the following benefit values, including: accuracy, recall, F1 score, average accuracy, and average detection time.
Specifically, the indexes for measuring the detection effect are Precision (Precision), recall (Recall), average Precision (AP), and F1 Score (F1-Score). In addition, the average detection time of the model for a certain amount of pictures is calculated (AVERAGE TIME).
Accuracy refers to the proportion of the fish bone sample judged as "fish bone" to all the fish bones judged as "fish bone", i.e. how much of the detection frames judged as "fish bone" are true fish bones:
Wherein TP (True Positive) represents the number of fishbones successfully identified by the target detection model, FP (False Positive) represents the number of fishbones erroneously determined by the target detection model as "fishbones" by the non-fishbone portion.
The Recall (Recall) is the ratio of the fishbone judged to be "fishbone" by the target detection model to the total fishbone sample, i.e., how high the ratio of true fishbone is detected by the target detection model.
Wherein FN (Flase Negative) represents the number of fishbone that the fishbone detection network model fails to detect.
The Average Precision (AP) is the area under the precision-recall curve, and is a commonly used evaluation index.
The F1 Score (F1-Score) is the harmonic average of the two index values, precision and recall, and allows a more comprehensive evaluation of the model:
Substituting the accuracy and recall rate of the target detection model corresponding to the different confidence coefficient thresholds into a calculation formula of the F1 score, and adopting the confidence coefficient threshold with the F1 score taking the maximum value as the optimal threshold of the target detection model. Meanwhile, the accuracy and recall ratio of the target detection model calculated by adopting the threshold value are also used as application reference values of actual production.
The average detection time (AVERAGE TIME) is the time required for the object detection model to detect a picture, and is in ms.
As shown in Table I, the test results of the fishbone test network model were tested using the test set.
List one
Comparing the obtained detection results with corresponding thresholds respectively, and if all the detection results meet the requirements, pre-training the model by using the sample sub-image; if any parameter is not satisfactory, the sample sub-image is discarded.
According to the fishbone residue detection method provided by the embodiment of the invention, the appropriate confidence threshold is preferably selected, and docile samples are screened to optimize the training process of the fishbone detection network model, so that the fishbone detection network model has an optimal detection effect, and the detection precision of the model is effectively improved.
Based on the above embodiments, as an alternative embodiment, the fishbone detection network model is constructed with fast-RCNN as the target detection framework and VGG19 as the target detection base network.
Specifically, in the embodiment of the invention, faster-RCNN is adopted as a target detection framework, and compared with other detection frameworks such as R-FCN, retinaNet, faster-RCNN has higher detection precision and better comprehensive performance, but lower detection speed. However, for food quality safety detection, a certain detection speed can be sacrificed to pursue higher accuracy, so that fast-RCNN is more suitable for the application scene.
Further, in the embodiment of the present invention, the base network for target detection is VGG19. Compared with convolutional neural networks such as Alexnet, the VGG19 adopts the convolutional kernel size and the maximum pooling size with the same size, so that the network structure is simpler; meanwhile, the VGG19 adopts a plurality of small filter convolution layers to replace the original large filter convolution layer, so that the accuracy effect on target identification is higher.
The embodiment of the invention also provides a fishbone residual detecting system, as shown in fig. 6, including but not limited to an image acquisition unit 1 and a model operation unit 2, wherein:
The image acquisition unit 1 is mainly used for acquiring a fish original image and dividing a detection sub-image with detection value on the fish original image; the model operation unit 2 is mainly used for inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the fishbone original image according to the output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
Specifically, the fishbone residual detecting system provided by the embodiment of the invention firstly takes an image of the Atlantic salmon flesh to be detected (fish flesh original image) with the image acquiring unit 1. In view of the fact that the size and image quality of the fish bone in the image will directly affect the detection accuracy, it is possible in this embodiment to fix the relative spatial position of the fish bone and the camera in the image acquisition unit 1 with simple mechanical means and to set the camera parameters therein. Wherein the distance between the shooting lens and the Atlantic salmon fish is ensured to be a fixed value, and the shooting lens is kept perpendicular to a bottom plate for placing the fish. In addition, various camera parameters, such as focal length, exposure time, maximum aperture, resolution of the acquired image, etc., are set to ensure that the image of Atlantic salmon flesh in the captured image is clear and the details are apparent.
Further, the image acquisition unit 1 is also configured to determine and segment a high-value region within the acquired image based on the physiological structure of salmon. Taking into account that the distribution of the fish bones on the atlantic salmon fillet conforms to the physiological structure thereof, according to the physiological anatomical principle, the specific distribution position of the atlantic salmon bones on the fillet is obtained to cut the obtained original image of the fish meat, and a high-value area image, i.e., a detection sub-image, is obtained.
Further, the detected sub-images are uploaded to the model operation unit 2, the model operation unit 2 performs deep learning analysis on the detected sub-images input into the model by using the stored pre-trained fish bone detection network model, so as to obtain a fish bone residual result corresponding to the input detected sub-images, namely, a fish bone residual detection result of the Atlantic salmon meat to be detected can be obtained through deduction.
According to the fishbone residue detection system provided by the embodiment of the invention, the fishbone detection network model is trained by using the deep learning target detection algorithm, and the high-value area in the image is segmented according to the original image of the fish meat and is used as a detection sub-image to be input into the fishbone detection network model, so that the fishbone residue analysis result can be directly obtained, the real-time accurate detection of the fishbone residue can be realized, the labor intensity is reduced, and the labor cost is reduced.
It should be noted that, when the fishbone residual detecting system provided in the embodiment of the present invention is specifically implemented, the fishbone residual detecting method may be implemented based on any of the above embodiments, which is not described in detail in this embodiment.
Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 430, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may call logic instructions in memory 730 to perform a fishbone residual detection method comprising: acquiring a fish original image, and dividing a detection sub-image with detection value on the fish original image; inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the original fish image according to the output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, embodiments of the present invention further provide a computer program product, including a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions which, when executed by a computer, enable the computer to perform the fishbone residual detection method provided in the above method embodiments, the method including: acquiring a fish original image, and dividing a detection sub-image with detection value on the fish original image; inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the original fish image according to the output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
In still another aspect, an embodiment of the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the method for detecting fishbone residue provided in the above embodiments, the method including: acquiring a fish original image, and dividing a detection sub-image with detection value on the fish original image; inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the original fish image according to the output result of the fishbone detection network model; the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The fishbone residue detection method is characterized by comprising the following steps:
setting camera parameters to obtain a fish original image, and dividing a detection sub-image with detection value on the fish original image; the camera parameters include at least one of a focal length, an exposure time, a maximum aperture, and a resolution of an acquired image;
Inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the fishbone original image according to an output result of the fishbone detection network model;
Wherein, the obtaining the original image of the fish flesh comprises the following steps: fixing the relative spatial position of the fish meat and the shooting camera, ensuring that the distance between a camera lens of the shooting camera and the fish meat is a fixed value, ensuring that the shooting camera is perpendicular to a bottom plate on which the fish meat is placed, and controlling the shooting camera to shoot the fish meat so as to acquire an original fish meat image;
The splitting of the detection sub-image with detection value on the fish original image comprises the following steps: according to the physiological structure and physiological anatomy principle of fish, specific distribution positions of fish bones are obtained, the original fish image is cut, a high-value area image is obtained, and the high-value area image is used as a detection sub-image;
the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
2. The fish bone residual detection method according to claim 1, further comprising, before inputting the detection sub-image to a fish bone detection network model:
Acquiring a plurality of original fish sample images;
respectively acquiring at least one detection sample sub-image in each fish original sample image;
Setting a fishbone residual result label for each detection sample sub-image;
Combining each detection sample sub-image with the corresponding fishbone residual result label to be used as a training sample, and constructing a training sample set;
and pre-training the fishbone detection network model by using the training sample set.
3. The fish bone residue detection method according to claim 2, further comprising, before setting a fish bone residue result tag for each of the detection sample sub-images:
Expanding the sub-image of the detection sample by using geometric transformation based on image processing;
And denoising and filtering each expanded detection sample sub-image, and then constructing a detection sample sub-image set.
4. The method of claim 2, further comprising, prior to pre-training the fish bone detection network model with the training sample set:
setting an intersection ratio of each training sample for pre-training of the fishbone detection network model;
And the intersection ratio is the overlapping rate of the image area and the marked image area in the detection frame of the fishbone detection network model.
5. The method for detecting fishbone as claimed in claim 2, wherein the pre-training the fishbone detecting network model using the training sample set comprises:
training the fishbone detection network model by using each training sample in sequence;
setting a confidence threshold value for selecting and rejecting a detection frame generated by the fishbone detection network model in each training process;
If the confidence coefficient value of the detection frame is larger than a confidence coefficient threshold value, training the fishbone detection network model by using the detection sample sub-image and the fishbone residual result label corresponding to the detection frame;
If the confidence coefficient value of the detection frame is smaller than the confidence coefficient threshold value, discarding the sub-image of the detection sample and the fishbone residual result label corresponding to the detection frame, and continuing to train the fishbone detection network model by using the next training sample.
6. The method of claim 5, wherein the confidence threshold comprises: an accuracy threshold, a recall threshold, an F1 score threshold, an average accuracy threshold, and an average detection time threshold;
the confidence value of the detection frame comprises: accuracy, recall, F1 score, average accuracy, and average detection time.
7. The method for detecting fish bone residue according to claim 6, wherein,
The calculation method of the accuracy comprises the following steps:
；
The calculation method of the recall rate comprises the following steps:
；
The F1 fraction calculating method comprises the following steps:
；
the average precision is the area under the curve of the precision rate and the calling rate;
The method for calculating the average detection time comprises the following steps:
；
Wherein, For accuracy, TP is the number of fishbones successfully identified by the model, FP is the number of fishbones incorrectly identified by the model as non-fishbone portions, FN is the number of fishbones not detected by the model,/>Is the recall rate; F1-Score represents the F1 Score, which is the harmonic mean of accuracy and recall; AVERAGE TIME is the average detection time,/>To detect total time,/>Is the total number of samples tested.
8. The method according to claim 1, wherein the fishbone detecting network model is constructed with fast-RCNN as a target detecting framework and VGG19 as a target detecting base network.
9. A fish bone residue detection system, comprising:
An image acquisition unit for setting camera parameters to acquire a fish flesh original image, and dividing a detection sub-image having a detection value on the fish flesh original image; the camera parameters include at least one of a focal length, an exposure time, a maximum aperture, and a resolution of an acquired image;
The model operation unit is used for inputting the detection sub-image into a fishbone detection network model, and acquiring a fishbone residual result corresponding to the fishbone original image according to the output result of the fishbone detection network model;
Wherein, the obtaining the original image of the fish flesh comprises the following steps: fixing the relative spatial position of the fish meat and the shooting camera, ensuring that the distance between a camera lens of the shooting camera and the fish meat is a fixed value, ensuring that the shooting camera is perpendicular to a bottom plate on which the fish meat is placed, and controlling the shooting camera to shoot the fish meat so as to acquire an original fish meat image;
The splitting of the detection sub-image with detection value on the fish original image comprises the following steps: according to the physiological structure and physiological anatomy principle of fish, specific distribution positions of fish bones are obtained, the original fish image is cut, a high-value area image is obtained, and the high-value area image is used as a detection sub-image;
the fishbone detection network model is obtained after training according to the detection sample sub-image with the fishbone residual result label.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the fish bone residue detection method according to any one of claims 1 to 8 when the program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724230.8A CN111951233B (en) | 2020-07-24 | 2020-07-24 | Fishbone residue detection method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724230.8A CN111951233B (en) | 2020-07-24 | 2020-07-24 | Fishbone residue detection method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111951233A CN111951233A (en) | 2020-11-17 |
| CN111951233B true CN111951233B (en) | 2024-05-14 |
Family
ID=73338074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010724230.8A Active CN111951233B (en) | 2020-07-24 | 2020-07-24 | Fishbone residue detection method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111951233B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112508930A (en) * | 2020-12-17 | 2021-03-16 | 无锡日联科技股份有限公司 | Food foreign matter detection method and device based on deep learning |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117877A (en) * | 2018-08-02 | 2019-01-01 | 南京师范大学 | A kind of Pelteobagrus fulvidraco and its intercropping kind recognition methods generating confrontation network based on depth convolution |
| CN109685152A (en) * | 2018-12-29 | 2019-04-26 | 北京化工大学 | A kind of image object detection method based on DC-SPP-YOLO |
| CN110243805A (en) * | 2019-07-30 | 2019-09-17 | 江南大学 | Fishbone detection method based on Raman high light spectrum image-forming technology |
| CN111110335A (en) * | 2019-12-31 | 2020-05-08 | 锐驰机器人（深圳）有限公司 | Intelligent esophagus fish bone pulling device and method |
-
2020
- 2020-07-24 CN CN202010724230.8A patent/CN111951233B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117877A (en) * | 2018-08-02 | 2019-01-01 | 南京师范大学 | A kind of Pelteobagrus fulvidraco and its intercropping kind recognition methods generating confrontation network based on depth convolution |
| CN109685152A (en) * | 2018-12-29 | 2019-04-26 | 北京化工大学 | A kind of image object detection method based on DC-SPP-YOLO |
| CN110243805A (en) * | 2019-07-30 | 2019-09-17 | 江南大学 | Fishbone detection method based on Raman high light spectrum image-forming technology |
| CN111110335A (en) * | 2019-12-31 | 2020-05-08 | 锐驰机器人（深圳）有限公司 | Intelligent esophagus fish bone pulling device and method |
Non-Patent Citations (3)
| Title |
|---|
| Automated fish bone detection using X-ray imaging;Domingo等;《Journal of food engineering》;论文第2节 * |
| Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks;Shaoqing Ren等;《IEEE》;论文第3节 * |
| 速冻食品异物检测与剔除技术研究;王强;《硕士电子期刊》;论文正文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111951233A (en) | 2020-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111325713B (en) | Neural network-based wood defect detection method, system and storage medium | |
| Huang et al. | Real-time classification of green coffee beans by using a convolutional neural network | |
| CN111178197A (en) | Mass R-CNN and Soft-NMS fusion based group-fed adherent pig example segmentation method | |
| CN109697716B (en) | Identification method and equipment of cyan eye image and screening system | |
| TWI673683B (en) | System and method for identification of symptom image | |
| JP6913953B2 (en) | Digitization method of quality standard of meat quality grade, digital processing method of quality evaluation using the quality standard, automatic meat quality evaluation device using them, and software installed in it | |
| CN111972700B (en) | Cigarette appearance detection method and device, equipment, system and medium thereof | |
| CN111161295B (en) | Dish image background stripping method | |
| Rossi et al. | FishAPP: A mobile App to detect fish falsification through image processing and machine learning techniques | |
| CN114581709A (en) | Model training, method, apparatus, and medium for recognizing target in medical image | |
| CN114219754A (en) | Thyroid-related eye disease identification method and device based on eye CT image | |
| CN111951233B (en) | Fishbone residue detection method and system | |
| CN112750104A (en) | Method and device for automatically matching optimal camera by monitoring ship through multiple cameras | |
| CN113822927B (en) | Face detection method, device, medium and equipment suitable for weak quality image | |
| CN114445843A (en) | Card image character recognition method and device of fixed format | |
| CN111753775B (en) | Fish growth assessment method, device, equipment and storage medium | |
| CN115272340B (en) | Industrial product defect detection method and device | |
| CN110097080B (en) | Construction method and device of classification label | |
| JPWO2021140602A5 (en) | Image processing system and program | |
| CN116311211A (en) | Detection and identification method and system of gas detector and storage medium | |
| JP2021051376A (en) | Image processing apparatus, image processing method, and program | |
| CN110084276B (en) | Splitting method and device of training set | |
| CN114565538B (en) | Endoscopic image processing method, system, storage medium and equipment | |
| CN110084278B (en) | Splitting method and device of training set | |
| Elfakharany et al. | Halalnet: a deep neural network that classifies the halalness slaughtered chicken from their images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |