JP2021042955A - Food inspection device, food inspection method and learning method of food reconstruction neural network for food inspection device - Google Patents

Abstract

To provide a food inspection device capable of improving the robustness of food inspection for confirming that no foreign matter is mixed.SOLUTION: The food inspection device is provided with light irradiation means 3 that irradiates an inspection object A with light, imaging means 4 for capturing an image of the inspection object A, and an identification processing device 6. The identification processing device 6 includes a food reconstruction neural network that has learned only good food products in advance, reconstructs a food assumption image using the food reconstruction neural network from the image obtained by the imaging means 4, calculates a difference between the image obtained by the imaging means 4 and the food assumption image, and identifies the inspection object A whose difference exceeds the preset threshold value as a foreign substance.SELECTED DRAWING: Figure 1

Description

The present invention relates to a food inspection device, a food inspection method, and a learning method of a food reconstruction neural network of the food inspection device.

Conventionally, various techniques such as metal detection and X-ray inspection have been proposed for food inspection devices that detect foreign substances mixed in inspection objects such as food. In recent years, there has been proposed an image processing of a photographed image to detect a foreign substance (Patent Document 1).

In Patent Document 1 (see claim 1), spectroscopic images of a rotten portion sample and a healthy portion sample of a perishable substance such as food are obtained, and principal component analysis is performed using the difference in absorption spectra of each sample. Create a calibration formula to determine whether or not it is a rotting part by a statistical method such as regression analysis or regression analysis, and apply a spectral image of an unknown sample of the same substance to the calibration formula to determine whether or not the unknown sample is a rotting part. It is stated that.

However, in the determination method of Patent Document 1, there is a possibility that the detection accuracy of the foreign matter may be lowered for the foreign matter different from the putrefactive part sample. Therefore, in order to maintain a certain detection accuracy, it is necessary to perform machine learning for all possible foreign substances, but it may be difficult to implement due to restrictions such as time, labor, and computer processing power. ..

Japanese Unexamined Patent Publication No. 2005-201636

The present invention has been made in consideration of such circumstances, and an object of the present invention is a food inspection device and a food inspection method capable of improving the robustness of a food inspection for confirming that no foreign matter is mixed. And to provide a learning method of a food reconstruction neural network of a food inspection device.

In order to achieve such an object, the present invention is grasped by the following configuration.
(1) The present invention distinguishes between a light irradiating means for irradiating the inspection object with light using a food for confirming that no foreign matter is mixed as an inspection object and an imaging means for capturing an image of the inspection object. The identification processing device includes a processing device, and the identification processing device includes a food reconstruction neural network in which only non-defective food products are learned in advance, and the food reconstruction neural network is used from an image obtained by the imaging means. , The food assumption image is reconstructed, the difference between the image obtained by the imaging means and the food assumption image is calculated, and the inspection object whose difference exceeds a preset threshold is identified as a foreign substance. Is.

(2) The present invention is characterized in that, in the configuration of (1) above, a reconstructed convolutional neural network using a self-encoder is used as the food reconstructed neural network.

(3) The present invention is characterized in that, in the configuration of (1) above, color reconstruction using only brightness is used as the food reconstruction neural network.

(4) The present invention is a food inspection method for confirming that the food to be inspected is free of foreign substances by using the food inspection apparatus having any one of the above (1) to (3).

(5) The present invention is a method for learning a food reconstruction neural network of the food inspection device having the above configuration (1), and uses the imaging means of the food inspection device to confirm that no foreign matter is mixed in the food. The food assumption image generated by the convolutional neural network from the good product image in which the good product of the food is shown and no foreign matter is shown based on the original image obtained by the imaging means of the food inspection device, and the good product image. Learning is performed so that the difference between the two is minimized.

(6) The present invention is the learning method of the configuration of the above (5), characterized in that the good product image is obtained by imaging only the good product of the food with the imaging means of the food inspection device. Is.

(7) The present invention is the learning method of the configuration of the above (5), and is characterized in that the good product image is obtained by performing a process of removing an image of a foreign substance from the original image.

According to the present invention, there is provided a food inspection apparatus, a food inspection method, and a learning method of a food reconstruction neural network of a food inspection apparatus, which can improve the robustness of a food inspection for confirming that no foreign matter is mixed. be able to.

It is the schematic which shows the food inspection apparatus which concerns on embodiment of this invention. It is the schematic which shows the identification state of the inspection object. It is a flow chart which shows the process of the model learning stage of the identification processing method of Example 1 of the identification processing apparatus which concerns on embodiment of this invention. It is explanatory drawing of the preparation stage (S110) of the learning data which concerns on Example 1. FIG. It is explanatory drawing of the calculation step (S120) of the average luminance which concerns on Example 1. FIG. It is explanatory drawing of the pretreatment step (S210) which concerns on Example 1. FIG. It is explanatory drawing of the learning stage (S220) of the model which concerns on Example 1. FIG. It is a flow chart which shows the processing procedure of the detection step of the identification processing method of Example 1 of the identification processing apparatus which concerns on embodiment of this invention. It is explanatory drawing of each step (S310, S320, S330) of the detection step (S300) which concerns on Example 1. FIG. It is a graph which shows the example of the categorization (RGB color gamut) of ab space. It is explanatory drawing which shows the expression example of the categorization of the ab value of a pixel. It is a conceptual diagram which shows the example of the structure of the neural network "model Net (f (θ))" which concerns on Example 1. FIG. It is explanatory drawing of the pretreatment step (S210) which concerns on Example 2. FIG. It is explanatory drawing of the learning stage (S220) of the model which concerns on Example 2. FIG. It is explanatory drawing of each stage (S310, S320, S330) of the detection stage (S300) which concerns on Example 2. FIG.

Hereinafter, embodiments for carrying out the present invention (hereinafter, referred to as “embodiments”) will be described in detail with reference to the accompanying drawings. The same elements are designated by the same reference numerals throughout the description of the embodiments.

First, a food inspection device 1 for in-line identification of foreign matter F from food B (inspection object A) for which it is desired to confirm that foreign matter F is not mixed will be described with reference to FIG. FIG. 1 is a schematic view showing a food inspection device according to an embodiment of the present invention. The food inspection device 1 shown in FIG. 1 includes a transport means 2, a light irradiation means 3, an image pickup means 4, and an identification processing device 6.

The transport means 2 transports the inspection object A from the upstream process to the downstream process through the inspection process at the inspection location C, and is composed of a belt conveyor or the like. The transport means 2 transports the inspection object A at a transport speed of about 2 m / min to 20 m / min. The inspection object A includes a non-defective product S of food B and a foreign substance F.

The light irradiating means 3 irradiates the inspection object A at the inspection place C with light. The light irradiation means 3 may include a light enhancer 32.

The imaging means 4 captures an image of the inspection object A being transported at the inspection location C, and is composed of a CCD camera, a hyperspectral camera, and the like.

The identification processing device 6 identifies the foreign matter F of the food B in the inspection object A. The identification processing device 6 identifies the foreign matter F of the food B in the inspection object A from the image D1 captured by the imaging means 4.

The identification processing device 6 identifies the foreign matter F in-line from the inspection object A being transported by the transport means 2 based on the image D1. The identification processing device 6 has learned in advance the process of identifying the foreign matter F from the inspection object A by deep learning.

FIG. 2 is a schematic view showing an identification state of an inspection object. In FIG. 2, foreign substances F1 and F2 are identified in-line from the inspection object A being transported by the transport means 2. The foreign matter F2 is mixed with the portion of the non-defective product S in the food B.

Next, the identification processing device 6 according to the present embodiment will be described with reference to Examples 1 and 2.

[Example 1]
The first embodiment of the identification processing device 6 according to the present embodiment will be described with reference to FIGS. 3 to 12. 3 to 12 are diagrams showing the identification processing method of the first embodiment of the identification processing device 6 according to the present embodiment. This identification processing method includes a model learning stage and a detection stage. The model learning stage is a stage in which the neural network used in the detection stage is trained. In the first embodiment, the neural network is trained so as to perform color reconstruction using only the brightness. The detection step is a step of identifying the foreign matter F of the food B in the inspection object A being transported by the transport means 2 using the neural network learned in the model learning step.

<Model learning stage>
The model learning stage according to the first embodiment will be described with reference to FIGS. 3 to 7. FIG. 3 is a flow chart showing a processing procedure in the model learning stage of the identification processing method of the first embodiment of the identification processing apparatus according to the present embodiment. As shown in FIG. 3, in the model learning stage, the data preparation stage (S100) is first performed, and then the learning stage (S200) is performed.

(Data preparation stage)
The data preparation stage (S100) will be described. The data preparation stage (S100) is composed of a training data preparation stage (S110) and an average luminance calculation stage (S120). Hereinafter, each stage (S110, S120) of the data preparation stage (S100) will be described with reference to FIGS. 4 and 5, respectively.

(Data preparation stage: Learning data preparation stage (S110))
FIG. 4 is an explanatory diagram of the learning data preparation stage (S110). In the learning data preparation stage (S110), the data used in the learning stage (S200) according to the first embodiment is prepared. For this preparation, first, a plurality of (Nall) learning images (RGB images) are prepared. The learning image (RGB image) is an image in which the good product S of the food B is shown and the foreign matter F is not shown. The size of the learning image (RGB image) is "P1 x P2" pixels (px). The learning image (RGB image) may be an image obtained by capturing only the non-defective product S of the food B by the imaging means 4. In this case, the image D1 of the non-defective product S captured by the imaging means 4 becomes a plurality of (Nall) learning images (RGB images). Alternatively, the learning image (RGB image) may be the image D1 after the process of removing the image of the foreign substance F from the image D1 of the non-defective product S and the foreign substance F of the food B captured by the imaging means 4. Good. In this case, the image D1 after the process of removing the image of the foreign matter F from the image D1 captured by the imaging means 4 becomes a plurality of (Nall) learning images (RGB images). The process of removing the image of the foreign matter F may be a process of removing the image itself in which the foreign matter F appears, or a process of erasing the image area of the foreign matter F from the image in which the foreign matter F appears (for example, in white). It may be a painting process, etc.).

Next, the data preparation step (S100) is performed using Nall learning images (RGB images). First, Nall's learning images (RGB images) are divided into Nall's X% number of learning images (RGB images) and Nall's (100-X)% number of learning images (RGB images). And (S111). The X% number of learning images (RGB images) of Nall are learning images (RGB images) of the training set (Train set). The number of (100-X)% of Nall learning images (RGB images) is the learning images (RGB images) of the validation set. Train sets are used to train neural networks. The validation set is used to validate the learning quality of neural networks.

Next, for the training set (Train set) and the verification set (Validation set), each training image (RGB image) is decomposed into sub-images (RGB images) of "P3 x P3" pixels (px). (S112, where "P1> P3, P2>P3"). In addition, there should be no overlap between a plurality of sub-images (RGB images) decomposed from one learning image (RGB image).
It is not necessary to execute step S112 (decomposition process into sub-images). When step S112 (decomposition processing into sub-images) is not executed, the training image (RGB image) of the training set (Train set) and the learning image (RGB image) of the verification set (Validation set) are used as they are in the subsequent processing. used. However, by decomposing into sub-images, the pixel combinations in the learning stage of step S200, which will be described later, can be exponentially reduced, so that the learning time in the learning stage can be shortened.
In the following description of the process, it is assumed that step S112 (decomposition process into sub-images) has been executed.

(Data preparation stage: Average brightness calculation stage (S120))
FIG. 5 is an explanatory diagram of the calculation stage (S120) of the average luminance according to the first embodiment. The identification processing device 6 performs a calculation step (S120) of the average brightness on the sub-images (RGB images) of the N trains of the training set as the calculation target. First, the identification processing device 6 extracts a luminance channel (luminance ch) from each of the N train sub-images (RGB images) of the training set (S121). Next, the identification processing device 6 calculates the average brightness (Lavg) using the brightness channel (luminance ch) of the extracted Ntrain sub-images (S122). The calculation formula of the average brightness (Lavg) is expressed by the following formula (1).

The average brightness (Lavg) calculated by the identification processing device 6 is used to normalize the input data used in the training of the neural network in the learning stage (S200).

(Learning stage)
The learning stage (S200) will be described. The learning stage (S200) is composed of a preprocessing stage (S210) and a model learning stage (S220). Hereinafter, each stage (S210, S220) of the learning stage (S200) will be described with reference to FIGS. 6 and 7, respectively.

(Learning stage: Preprocessing stage (S210))
FIG. 6 is an explanatory diagram of the pretreatment stage (S210) according to the first embodiment. A learning input image (RGB image, "Pin1 x Pin2" pixel) is input to the preprocessing stage (S210). When step S112 (decomposition process into sub-images) is executed in the training data preparation stage (S110), the learning input image is a sub-image (Pin1 = Pin2 = P3). On the other hand, when step S112 (decomposition process into sub-images) is not executed in the learning data preparation stage (S110), the learning input image is a learning image (Pin1 = P1, Pin2 = P2). In the preprocessing stage (S210), for the training set (Train set) and the verification set (Validation set), "input (x) and correct color (y)" are selected from the respective learning input images (RGB images). Generate a pair.

The identification processing device 6 performs color space conversion processing for converting the learning input image (RGB image, “Pin1 × Pin2” pixels) from RGB space to Lab space, and L value (luminance) of L channel (Lch). And the a value and the b value (color) of the ab channel (abch) are generated (S211).

Next, the identification processing device 6 normalizes the L value (luminance) of Lch (S212). The calculation formula for this normalization is expressed by the following formula (2). The average luminance (Lavg) is a value calculated in the above-mentioned average luminance calculation step (S120). Equation (2) is an example of the normalization method.

The value of the result of Lch normalization is an input (x = {xi, j}). “Xi, j” is a value obtained as a result of normalization of the L value (luminance) of the “i, j” th pixel.

In step S212 described above, the L value (luminance) is normalized using the average brightness (Lavg), but the L value (luminance) may be standardized instead of the normalization. To standardize the L value (luminance), the average brightness (Lavg) may be set to 0 in the above formula (2) (that is, the numerator of the above formula (2) is set to the L value "Li, j"). .. However, if the L value (luminance) is normalized rather than standardized, the learning efficiency in the learning stage of step S200, which will be described later, can be improved.

Further, the identification processing device 6 categorizes and standardizes the a value and the b value (color) of the abch (S213). The value of the result of categorization of ach is the correct color (y).

Here, a method of categorizing and standardizing the a value and the b value (color) of ach will be described with reference to FIGS. 10 and 11. FIG. 10 is a graph showing an example of categorization (RGB color gamut) of ab space. FIG. 11 is an explanatory diagram showing an example of representation of categorization and normalization of pixel ab values.

First, as shown in FIG. 10, the ab space is discretized in advance evenly to the category bases (circles in FIG. 10) of an arbitrary number of categories (Q). Next, for each pixel of the image to be categorized, a predetermined number of categorical bases (vk) closest to the pixel value (p) (5 in the example of FIG. 10) is determined. Next, a vector (y') weighted by the distance between each category basis (vk) of the determination result and the pixel value (p) is obtained. Finally, the weights between the classes are standardized and the vector (y) is calculated. The calculation formula of this vector (y = {yk}) is represented by the following formula (3) (in the example of the formula (3), the number of category bases (vk) is five).

The vector (y) is an expression of categorization and normalization of the pixel value (p). In an example of the representation of categorization and normalization shown in FIG. 11, the number of categories is Q (that is, k is an integer from 1 to Q in the equation (3)), and the pixel value (p: a). The expression "vector (y = {yk})" of categorization and normalization of the value "-34" and the b value "65") is shown.

(Learning stage: Model learning stage (S220))
FIG. 7 is an explanatory diagram of the learning stage (S220) of the model according to the first embodiment. In the learning stage (S220) of the model, the pair of "input (x) and correct answer color (y)" of the training set (Train set) and the "input (x) and correct answer color (y)" of the validation set (Validation set) The neural network "model Net (f (θ))" is trained using the pair of "".

Here, the structure of the neural network “model Net (f (θ))” will be described with reference to FIG. FIG. 12 is a conceptual diagram showing an example of the structure of the neural network “model Net (f (θ))” according to the first embodiment.

The neural network "model Net (f (θ))" shown in FIG. 12 is a convolutional neural network composed of, for example, eight layers (Layer). The first layer is an input layer. The second to fourth layers are convolutional layers. The fifth to seventh layers are deconvolution layers, and the eighth layer is an output layer. The input image is "Pin1 x Pin2" pixels in size and is composed of na channels. In the first embodiment, the input image is only Lch (luminance channel) (that is, na = 1). The output size of the second layer is "P4 x P4 x 16". The output size of the third layer, the fourth layer, and the fifth layer is "P5 x P5 x 16". The output size of the sixth layer is "P4 x P4 x 16". The output size of the seventh layer is "Pin1 x Pin2 x 16". The output size of the eighth layer is "Pin1 x Pin2 x nb". In the first embodiment, nb is the number of categories (Q) of abch (that is, nb = Q). The magnitude relationship of Pin1, Pin2, P3, P4, P5 is "Pin1 or Pin2 ≥ Pin2 or Pin1> P4> P5".

Hereinafter, the learning stage (S220) of the model will be described with reference to FIG. 7.
The identification processing device 6 uses a pair of “input (x) and correct color (y)” of the training set to minimize the loss of “Categorical Cross-entropy”, which is a neural network “Model Net”. The parameter (θ) of "f (θ))" is learned (S221, S222). In step S221, the input (x) is input to the neural network “model Net (f (θ))”, and the output value of the neural network “model Net (f (θ))” is predicted to be the predicted color (y ^ = fθ (y ^ = fθ)). x)). Next, in step S222, the loss (E (y, y ^) of "Categorical Cross-entropy" is calculated using the predicted color (y ^ = fθ (x)) and the correct color (y). The calculation formula for the loss (E (y, y ^)) of "Categorical Cross-entropy" is represented by the following formula (4). Q is the number of categories in the categorization of ach in step S213 described above. Correct answer. The “yi, j, k” of the color (y) is the value of the kth category of the “i, j” th pixel (that is, yk according to the equation (3)). The “y” of the predicted color (y ^). “^ I, j, k” is a value in the k-th category of the “i, j” th output value of the neural network “model Net (f (θ))”.

As a method of learning the parameter (θ) of the neural network "model Net (f (θ))" that minimizes the loss of "Categorical Cross-entropy" (E (y, y ^)), the error back propagation method and stochastic descent The gradient descent method (Stochastic Gradient Descent: SGD) is used.

The identification processing device 6 repeatedly learns the parameter (θ) of the neural network “model Net (f (θ))” by a fixed number of iterations (epochs). The identification processing device 6 uses the pair of "input (x) and correct color (y)" of the validation set to set the parameter (θ) every time the learning of the parameter (θ) is repeated. Verify. In the verification of this parameter (θ), the above-mentioned steps S221 and S222 are performed using the pair of “input (x) and correct color (y)” of the validation set, and “Categorical Cross-entropy” is performed. The loss (E (y, y ^)) of is calculated. The latest parameter (θ) when this loss (E (y, y ^) decreases is saved as the best parameter. The parameter of this saved best parameter. (Θ) is a parameter (θ) applied to the neural network “model Net (f (θ))” used in the detection stage.

The above is the description of the model learning stage according to the first embodiment. By this model learning stage, a neural network "model Net (f (θ))" that learns only the non-defective product S of food B is generated.

<Detection stage>
The detection stage according to the first embodiment will be described with reference to FIGS. 8 and 9. FIG. 8 is a flow chart showing a processing procedure in the detection stage of the identification processing method of the first embodiment of the identification processing apparatus according to the present embodiment. As shown in FIG. 8, in the detection stage (S300), the pretreatment stage (S310) is first performed, then the prediction stage (S320) is performed, and then the post-processing stage (S330) is performed. FIG. 9 is an explanatory diagram of each stage (S310, S320, S330) of the detection stage (S300) according to the first embodiment. Hereinafter, each stage (S310, S320, S330) of the detection stage (S300) will be described with reference to FIG.

(Detection stage: Preprocessing stage (S310))
The input image is an RGB image of the image D1 in which the inspection object A being transported is captured by the imaging means 4. The size of the input image (RGB image) is "P1 x P2" pixels. In the preprocessing step (S310), a pair of "input (x) and correct color (y)" is generated from the input image (RGB image).

The identification processing device 6 performs color space conversion processing for converting the input image (RGB image) from the RGB space to the Lab space, and the L value (luminance) of Lch) and the a value and b value (color) of abch. ) And (S311). Next, the identification processing device 6 normalizes the L value (luminance) of Lch (S312). The calculation formula for this normalization is represented by the above formula (2). The average luminance (Lavg) is a value calculated in the above-mentioned average luminance calculation step (S120). The value of the result of Lch normalization is the input (x). Further, the identification processing device 6 categorizes and standardizes the a value and the b value (color) of the abch (S313). The method of categorization and normalization is the same as that of step S213 described above. The value of the result of categorization of ach is the correct color (y).

(Detection stage: Prediction stage (S320))
In the prediction stage (S320), the "input (x)" of the input image (RGB image) is used by using the neural network "model Net (f (θ))" in which only the non-defective product S of food B is learned in the model learning stage described above. ) And the correct color (y) ”to generate a heat map (h).

First, the identification processing device 6 inputs the input (x) to the neural network "model Net (f (θ))" and predicts the output value of the neural network "model Net (f (θ))" (y). ^ = Fθ (x)) (S321). Next, the identification processing device 6 calculates the reconstruction error (heat map (h)) between the predicted color (y ^ = fθ (x)) and the correct color (y) for each pixel (S322). The calculation formula of this heat map (h) is expressed by the following formula (5). Nk is the number of categories (Q). “Hi, j” is a heat map value of the “i, j” th pixel.

(Detection stage: Post-processing stage (S330))
In the post-processing stage (S330), the heat map (h) of the input image (RGB image) is used to obtain the detection positions (i, j) of the foreign matter F.

First, in the heat map (h), the identification processing device 6 sets a pixel (i, j) in which the heat map value “hi, j” exceeds a predetermined threshold value Th1 as a defective “1”, and sets the other pixels as a good “0”. , The heat map (h) is binarized (S331). Next, the identification processing device 6 calculates the distance between the pixels with the defect “1”, and classifies the pixels with the defect “1” whose distance is equal to or less than the predetermined threshold Th2 into the same cluster (S332). Next, the identification processing device 6 calculates the center of gravity of each cluster and outputs the pixels (i, j) of the center of gravity as the detection positions (i, j) of the foreign matter F (S333).

The above is the description of the detection stage according to the first embodiment. In this detection step, the detection positions (i, j) at which the foreign matter F of the food B in the inspection object A being transported by the transport means 2 is identified are output.

In Example 1 described above, the neural network "model Net (f (θ))" is a color reconstruction convolutional neural network using an L value (luminance), and is an example of a food reconstruction neural network. The predicted color (y ^ = fθ (x)) is an example of a food assumption image.

In the imaging of the learning image (RGB image) used for learning the neural network "model Net (f (θ))", the lowest illuminance among the four corners of the angle of view of the imaging target location imaged by the imaging means 4. Is preferably 80% or more, more preferably 90% or more of the illuminance at the center of the angle of view. In the convolutional neural network of color reconstruction using the L value, since it is greatly affected by the L value input to the neural network, it becomes easy to perform reproducible learning by stabilizing the illuminance.

Further, in the learning of a convolutional neural network for color reconstruction using the L value (brightness), features are extracted by obtaining a plurality of feature maps via the neural network by convolutional processing by the convolutional layer. Next, the number of pixels of the original image size is reproduced via the neural network by the deconvolution process by the deconvolution layer. By repeating these in a nested manner (second, third, ..., nth layer) a plurality of times, more conceptual feature extraction becomes possible.
It is preferable to use a sub-image (RGB image) for learning color reconstruction using this brightness. The sub image (RGB image) is obtained by cutting out an image size of less than about 10 times the average pixel length of the food B reflected in the learning image. The average pixel length of food B is when the maximum number of pixels in either the vertical or horizontal direction when the target food B is imaged is measured and a predetermined number of pixels (10 as an example here) are imaged. Is the average value of.
In the learning of color reconstruction using this brightness, the L value (input (x)) of the learning image or sub-image is first input, and the color-reconstructed reconstruction category obtained from the nth deconvolution layer is obtained. Cal expression is obtained. The loss function (Categorical Cross) between the categorical representation (correct color (y)) of the learning image or sub-image that is the original image and the reconstructed categorical representation (predicted color (y ^ = fθ (x))). -entropy) is learned to be as low as possible.

Further, in the detection stage using the convolutional neural network of color reconstruction using the L value (brightness), the L value of the input image is input to the first convolutional layer and the categorical representation obtained from the nth layer ( The difference between the food assumption image) and the categorical expression obtained from the input image is obtained, and pixels exceeding a predetermined threshold value for the difference are detected as "non-good objects (that is, foreign substances)".

The above is the description of the first embodiment.

[Example 2]
The second embodiment of the identification processing device 6 according to the present embodiment will be described with reference to FIGS. 13 to 15. 13 to 15 are diagrams showing the identification processing method of the second embodiment of the identification processing device 6 according to the present embodiment. The present identification processing method is composed of a model learning stage and a detection stage, as in the first embodiment described above. The model learning stage is a stage in which the neural network used in the detection stage is trained. In the second embodiment, a reconstructed convolutional neural network using a self-encoder is used. The detection step is a step of identifying the foreign matter F of the food B in the inspection object A being transported by the transport means 2 using the neural network learned in the model learning step.

<Model learning stage>
The model learning stage according to the second embodiment will be described with reference to FIGS. 13 and 14. In the model learning stage according to the second embodiment, the data preparation stage (S100) is first performed, and then the learning stage (S200) is performed, similarly to the procedure shown in FIG. 3 of the first embodiment. However, in the second embodiment, the average brightness calculation step (S120) is not executed in the data preparation step (S100).

(Data preparation stage)
The data preparation stage (S100) according to the second embodiment is composed of a training data preparation stage (S110). Since the learning data preparation stage (S110) according to the second embodiment is the same as that of the first embodiment (FIG. 4) described above, the description thereof will be omitted. As in the first embodiment, step S112 (decomposition process into sub-images) may or may not be executed.

(Learning stage)
The learning stage (S200) according to the second embodiment will be described. The learning stage (S200) according to the second embodiment is composed of a preprocessing stage (S210) and a model learning stage (S220). Hereinafter, each stage (S210, S220) of the learning stage (S200) according to the second embodiment will be described with reference to FIGS. 13 and 14, respectively.

(Learning stage: Preprocessing stage (S210))
FIG. 13 is an explanatory diagram of the pretreatment stage (S210) according to the second embodiment. A learning input image (RGB image, "Pin1 x Pin2" pixel) is input to the preprocessing stage (S210). When step S112 (decomposition process into sub-images) is executed in the learning data preparation stage (S110) according to the second embodiment, the learning input image is a sub-image (Pin1 = Pin2 = P3). On the other hand, when step S112 (decomposition process into sub-images) is not executed in the learning data preparation stage (S110) according to the second embodiment, the learning input image is a learning image (Pin1 = P1, Pin2 = P2). ). In the preprocessing stage (S210) according to the second embodiment, the training set (Train set) and the verification set (Validation set) are subjected to "input (x) and correct answer image" from the respective learning input images (RGB images). (Y) ”pair is generated.

The identification processing device 6 standardizes each RGB channel (ch) with respect to the learning input image (RGB image, “Pin1 × Pin2” pixel) (S2110). The calculation formula for this standardization is expressed by the following formula (6). Equation (6) is an example of a standardization method.

The value as a result of standardization of each RGB channel is an input (x = {xi, j, k}). “Xi, j, k” is a value obtained as a result of normalization of the channel k of the “i, j” th pixel. Further, the input (x = {xi, j, k}) is used as the correct image (y) paired with the input (x = {xi, j, k}). That is, in one pair, the input (x) and the correct image (y) are the same.

In the pretreatment step (S210) according to the second embodiment described above, the RGB channels are standardized, but the RGB channels may be normalized using the average value of the RGB channels. The method of normalizing each RGB channel is the same as the method of normalizing the L value (luminance) according to the first embodiment described above.

(Learning stage: Model learning stage (S220))
FIG. 14 is an explanatory diagram of the learning stage (S220) of the model according to the second embodiment. In the learning stage (S220) of the model according to the second embodiment, the pair of “input (x) and correct image (y)” of the training set (Train set) and the “input (x)” of the validation set (Validation set) The neural network "CAE (f (θ))" is learned by using the pair of "correct image (y)".

The structure of the neural network “CAE (f (θ))” according to the second embodiment is the same as the structure of the neural network “model Net (f (θ))” of FIG. 12 according to the first embodiment described above. However, in FIG. 12, in the neural network “CAE (f (θ))” according to the second embodiment, the input image has a size of “Pin1 × Pin2” pixels and is composed of three RGB channels ( That is, na = 3). The output size of the eighth layer is "Pin1 x Pin2 x 3" (that is, nb = 3). In the second embodiment, na and nb are the number of RGB channels "3" (na = nb = 3).

Hereinafter, the learning stage (S220) of the model according to the second embodiment will be described with reference to FIG.
The identification processing device 6 uses a pair of "input (x) and correct image (y)" of the training set to minimize the loss of "Binary Cross-entropy", which is a neural network "CAE (f)". (Θ)) ”parameter (θ) is learned (S2210, S2220). In step S2210, the input (x) is input to the neural network “CAE (f (θ))”, and the output value of the neural network “CAE (f (θ))” is predicted image (y ^ = fθ (x)). ). Next, in step S2220, the loss (E (y, y ^) of "Binary Cross-entropy" is calculated using the predicted image (y ^ = fθ (x)) and the correct image (y). The calculation formula for the loss (E (y, y ^)) of "Binary Cross-entropy" is expressed by the following formula (7). The "yi, j, k" in the correct image (y) is "i, j". It is the value of the channel k (each of RGB channels) of the third pixel. The “y ^ i, j, k” of the predicted image (y ^) is the “i, j” of the neural network “CAE (f (θ))”. This is the value of the channel k (each RGB channel) of the third output value.

The error back propagation method and stochastic gradient descent are methods for learning the parameter (θ) of the neural network “CAE (f (θ))” that minimizes the loss (E (y, y ^)) of the “Binary Cross-entropy”. The descent method (Stochastic Gradient Descent: SGD) is used.

The identification processing device 6 repeatedly learns the parameter (θ) of the neural network “CAE (f (θ))” by a fixed number of iterations (epochs). The identification processing device 6 uses the pair of "input (x) and correct image (y)" of the validation set to set the parameter (θ) every time the learning of the parameter (θ) is repeated. Verify. In the verification of this parameter (θ), the above-mentioned steps S2210 and S2220 are performed using the pair of “input (x) and correct image (y)” of the validation set, and “Binary Cross-entropy” is performed. The loss (E (y, y ^)) of is calculated. The latest parameter (θ) when this loss (E (y, y ^) decreases is saved as the best parameter. The parameter of this saved best parameter. (Θ) is a parameter (θ) applied to the neural network “CAE (f (θ))” used in the detection stage.

The above is the description of the model learning stage according to the second embodiment. By this model learning stage, a neural network "CAE (f (θ))" that learns only the non-defective product S of food B is generated.

<Detection stage>
The detection step according to the second embodiment will be described with reference to FIG. As for the detection step according to the second embodiment, the pretreatment step (S310) is first performed, and then the prediction step (S320) is performed in the same manner as the procedure of the detection step (S300) shown in FIG. 8 of the first embodiment. Then, the post-treatment step (S330) is performed. FIG. 15 is an explanatory diagram of each stage (S310, S320, S330) of the detection stage (S300) according to the second embodiment. Hereinafter, each stage (S310, S320, S330) of the detection stage (S300) according to the second embodiment will be described with reference to FIG.

(Detection stage: Preprocessing stage (S310))
The input image is an RGB image of the image D1 in which the inspection object A being transported is captured by the imaging means 4. The size of the input image (RGB image) is "P1 x P2" pixels. In the preprocessing step (S310), a pair of "input (x) and correct answer image (y)" is generated from the input image (RGB image).

The identification processing device 6 standardizes each RGB channel (ch) with respect to the input image (RGB image) (S3110). The calculation formula for this standardization is represented by the above formula (6). The value as a result of standardization of each RGB channel is an input (x = {xi, j, k}). “Xi, j, k” is a value obtained as a result of normalization of the channel k of the “i, j” th pixel. Further, the input (x = {xi, j, k}) is used as the correct image (y) paired with the input (x = {xi, j, k}). That is, in one pair, the input (x) and the correct image (y) are the same.

(Detection stage: Prediction stage (S320))
In the prediction stage (S320), the input image (RGB image) is obtained by using the neural network “CAE (f (θ))” in which only the non-defective product S of the food B is learned in the model learning stage according to the second embodiment described above. A heat map (h) is generated from a pair of "input (x) and correct image (y)".

First, the identification processing device 6 inputs the input (x) to the neural network “CAE (f (θ))” and predicts the output value of the neural network “CAE (f (θ))” (y ^ = Let fθ (x)) (S3210). Next, the identification processing device 6 calculates the reconstruction error (heat map (h)) between the predicted image (y ^ = fθ (x)) and the correct image (y) for each pixel (S3220). The calculation formula of this heat map (h) is expressed by the following formula (8). “Hi, j” is a heat map value of the “i, j” th pixel.

As another calculation method of the heat map (h), the difference between the correct image (y) and the predicted image (y ^ = fθ (x)) is grayscale-converted, and the value after the grayscale conversion is heat-mapped. May be used for the value.

(Detection stage: Post-processing stage (S330))
In the post-processing stage (S330), the heat map (h) of the input image (RGB image) is used to obtain the detection positions (i, j) of the foreign matter F. Since the post-treatment step (S330) according to the second embodiment is the same as the post-treatment step (S330) of the above-described first embodiment (S330), the description thereof will be omitted.

The above is the description of the detection stage according to the second embodiment. In this detection step, the detection positions (i, j) at which the foreign matter F of the food B in the inspection object A being transported by the transport means 2 is identified are output.

In Example 2 described above, the neural network "CAE (f (θ))" is a reconstructed convolutional neural network using a self-encoder, and is an example of a food reconstructed neural network. The predicted image (y ^ = fθ (x)) is an example of a food assumption image.

In the learning of the reconstructed convolutional neural network using the self-encoder, the learning image or sub-image (input (x)) obtains a plurality of feature maps via the neural network by the convolution processing by the convolutional layer. The features are extracted with, and then reconstructed via the neural network by the reverse convolution processing by the reverse convolution layer. Further, by repeating the convolution process and the deconvolution process a plurality of times in a nested manner, more conceptual feature extraction becomes possible.
In the learning of the reconstructed convolutional neural network using this self-encoder, the output from the final deconvolutional layer should match the original input learning image or sub-image (correct image (y)) as much as possible. That is, the loss function representing the difference is learned to be as low as possible. As the loss function at this time, a loss function such as BCE (Binary Cross-entropy) can be used.

The above is the description of the second embodiment.

Although the food inspection device 1 shown in FIG. 1 described above includes the transport means 2, the transport means 2 may not be provided. For example, the inspection object A may be configured to be inspected by taking an image while dropping it. Further, the inspection object A may be configured to be inspected by taking an image while rotating and dropping it. By imaging and inspecting the inspection object A while rotating and dropping it, it is possible to image and inspect a plurality of directions and all directions of the inspection object A.

The effects of the described embodiments will be described above.
In the food inspection device 1 of the embodiment, the food B for confirming that the foreign matter F is not mixed is set as the inspection target A, and the light irradiation means 3 for irradiating the inspection target A with light and the image of the inspection target A are displayed. An image pickup means 4 for taking an image and an identification processing device 6 are provided, and the identification processing device 6 is provided with a food reconstruction neural network in which only the non-defective product S of the food B is learned in advance, and the image obtained by the image pickup means 4 is provided. From, the food assumption image is reconstructed using the food reconstruction neural network, the difference between the image obtained by the imaging means 4 and the food assumption image is calculated, and the difference is a preset threshold value. The inspection object A exceeding the above is identified as a foreign substance. According to this food inspection apparatus 1, the food assumption image is "an image predicted when all the captured foods B are non-defective products S", and the food assumption image and the image obtained by the imaging means 4 It becomes possible to identify the inspection object A whose difference exceeds the preset threshold value as a foreign substance. As a result, since it is not necessary to learn foreign substances, it is possible to deal with arbitrary foreign substances, and it is possible to improve the robustness of food inspection to confirm that there is no foreign matter mixed, that is, the versatility in abnormality discrimination. it can. In particular, if a color reconstruction convolutional neural network that uses only the L value (brightness) is used as the food reconstruction neural network, the detailed design of the neural network will be compared with the reconstruction convolutional neural network that uses a self-encoder. Detailed design of a convolutional neural network for color reconstruction using only the L value, which is less dependent on the inspection object of the model. It can be confirmed that there is no mixture of.

In addition, by using color reconstruction using only brightness as the food reconstruction neural network, the difficult problem of reconstructing colors from only brightness is learned in the learning of the food reconstruction neural network, and as a result, a self-explanatory rule. It is expected that the learning of the neural network will be avoided and the learning efficiency will be improved.

Further, the identification processing device 6 of the embodiment may be realized by dedicated hardware, or may be configured by a computer system such as a personal computer to realize the function of the identification processing device 6 of the embodiment. The function may be realized by executing the program of.

Further, an input device, a display device, or the like may be connected to the identification processing device 6 as peripheral devices. Here, the input device refers to an input device such as a keyboard and a mouse. The display device refers to a CRT (Cathode Ray Tube), a liquid crystal display device, or the like.
Further, the peripheral device may be directly connected to the identification processing device 6, or may be connected via a communication line.

Although the present invention has been described above using the embodiments, it goes without saying that the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. Further, it is clear from the description of the scope of claims that such a modified or improved form may be included in the technical scope of the present invention.

1 Food inspection device 2 Transport means 3 Light irradiation means 32 Auxiliary device 4 Imaging means 6 Identification processing device A Inspection object B Food S Good product F Foreign matter

Claims (7)

A light irradiation means for irradiating the inspection object with light using a food for which it is confirmed that there is no foreign matter as an inspection object, and a light irradiation means.
An imaging means for capturing an image of the inspection object and
Equipped with an identification processing device
The identification processing device is
It is equipped with a food reconstruction neural network that learns only good food products in advance.
From the image obtained by the imaging means, the food assumption image is reconstructed using the food reconstruction neural network.
The difference between the image obtained by the imaging means and the food assumption image is calculated.
An inspection object whose difference exceeds a preset threshold value is identified as a foreign substance.
Food inspection equipment. The food inspection device according to claim 1.
A food inspection apparatus characterized in that color reconstruction using only luminance is used as the food reconstruction neural network. The food inspection device according to claim 1.
A food inspection apparatus characterized in that a reconstructed convolutional neural network using a self-encoder is used as the food reconstructed neural network. By the food inspection apparatus according to any one of claims 1 to 3.
A food inspection method that confirms that the food to be inspected is free of foreign substances. The method for learning the food reconstruction neural network of the food inspection apparatus according to claim 1.
By the imaging means of the food inspection device, the food to be confirmed to be free of foreign substances is imaged,
The difference between the food assumption image generated by the convolutional neural network from the good product image in which the good product of the food is shown and no foreign matter is shown based on the original image obtained by the imaging means of the food inspection device is different from the good product image. Learn to be minimized,
Learning method of food reconstruction neural network of food inspection equipment. The learning method according to claim 5.
The image of the non-defective product is obtained by imaging only the non-defective product of the food product by the imaging means of the food inspection device.
Learning method of food reconstruction neural network of food inspection equipment. The learning method according to claim 5.
It is characterized in that a non-defective image is obtained by performing a process of removing an image of a foreign substance from the original image.
Learning method of food reconstruction neural network of food inspection equipment.

Images