TW202028722A - Method, program and device for determining quality of sample using fluorescence image - Google Patents

Abstract

An object of the present invention is to conduct quality determination of samples with high-accuracy and high-efficiency in consideration of fluorescence photobleaching. Provided is a method for determining quality of sample using fluorescence image, which includes a fluorescence image acquiring step (S01), a machine-learning step (S02), and a quality class determination step (S03), wherein in the step S01, the acquiring time-point for the fluorescence image is set to be within a predetermined period P (P>0) from a time-point T (T>=0) which is set in consideration of fluorescence photobleaching with irradiation initiation of excitation light as the start point.

Description

Quality judging method, program and device of sample using fluorescent image

The present invention relates to a method, a program, and a device for determining the quality of a sample using a fluorescent image, and more particularly to a method, program, and device for determining the quality of a component suitable for use in the sample.

Conventionally, in order to evaluate the quality of samples, etc., hyperspectral imaging technology using continuous spectrum or multispectral imaging technology using discrete spectrum has been used. In addition, a "fluorescence fingerprint imaging technique" using a "fluorescence fingerprint" as the "spectrum" of the above-mentioned spectral image has been proposed (see, for example, Patent Document 1 and Non-Patent Document 1).

Fluorescent fingerprint is also called Excitation Emission Matrix (EEM), which changes the excitation wavelength of the excitation light step by step, and irradiates the test sample containing the fluorescent substance with the excitation light, which will emit from the test sample The intensity of light (fluorescence) at the predetermined fluorescence wavelength is determined by excitation wavelength (λEx), fluorescence wavelength (measurement wavelength) (λEm), fluorescence intensity (IEx, Em) The point is drawn in the three-dimensional space of three orthogonal axes to obtain a collection of points and then visualize the collection (refer to Figure 2).

Fluorescent fingerprints can be expressed as a three-dimensional graph (refer to Fig. 3) or a two-dimensional graph (refer to Fig. 4) by expressing the fluorescent intensity of each point with contour shapes and color distribution. The fluorescent fingerprint exhibits a pattern inherent to the test sample with three-dimensional huge information, so it is very suitable for various identification and quantification.

In the hyperspectral image of the fluorescent fingerprint, as shown in Figure 5, in addition to the fluorescent fingerprint information mentioned above, it also contains location information. Therefore, the fluorescent fingerprint imaging technology can not only identify the test The components in the sample can also be known for their distribution. The "wavelength condition" in Figure 5 corresponds to the combination of excitation wavelength (λEx) and fluorescence wavelength (λEm), and the number of wavelength conditions is equal to the combination of excitation wavelength (λEx) and fluorescence wavelength (λEm). number.

Non-Patent Document 1 discloses the use of methods such as quadratic programming in order to obtain a visual image of specific components in a sample from fluorescent fingerprint images. Such a method can be said to be suitable for hyperspectral images. A solution to the classification problem.

Generally speaking, regarding the classification of hyperspectral images, the knowledge obtained is that it is effective if it is solved by deep learning, which is a type of machine learning (see, for example, Non-Patent Document 2).

Regarding deep learning, which is a kind of machine learning, various methods are known (see, for example, Non-Patent Document 3). Among them, the insights obtained regarding deep learning, which is suitable for image recognition in particular, are: using convolutional neural networks ( Convolutional Neural Network: CNN) is very effective (see, for example, Non-Patent Documents 3 and 4).

[Prior Technical Literature]

[Patent Literature]

[Patent Document 1] JP 2012-98244 A

[Non-Patent Literature]

技術

開発(中譯：螢光指紋影像化技術的開發)」；「日本食品科學工學會誌」VOL.62 NO.10, P477-483 (2015) [Non-Patent Document 1] Fanchuan Meita et al. "Xiaoguang Fingerprint

technology

Development (Chinese translation: Development of fluorescent fingerprint imaging technology)";"Journal of the Japanese Society of Food Science and Technology" VOL.62 NO.10, P477-483 (2015)

[Non-Patent Document 2] Li, Y, et al. "Spectral-Spatial Classification of Hyperspectral Imagery with 3D Convolutional Neural Network" Remote Sens. 2017, 9(1), 67; (http://dx.doi.org/ 10.3390/rs9010067)

[Non-Patent Document 3] Okaya Takayuki "Deep Learning (Chinese Translation: Deep Learning)" 2015-4-7, Kodansha

[Non-Patent Document 4] Harada Tatsuya "Knowing the Face (Chinese Translation: Knowing the Portrait)" 2017-5-24, Kodansha

As mentioned above, by using the fluorescent fingerprint imaging technology using the hyperspectral image of fluorescent fingerprints, not only the components in the sample can be identified, but also the distribution, etc., but when dealing with fluorescent fingerprints, it must be considered The so-called fluorescence photobleaching is a unique problem of fluorescence.

Fluorescence fading refers to the instability of the structure of the fluorescent pigment molecules after being excited by light, which causes the structure of the fluorescent pigment molecules to change, becoming unable to maintain the excited state and stop emitting fluorescence. Depending on the degree of fading, it may become difficult to observe the fluorescence.

However, the previous fluorescent fingerprint imaging technology did not fully consider how to deal with such fluorescent fading.

The present invention was proposed in order to solve such a problem. It is hoped that the method of machine learning to solve the classification problem in consideration of the phenomenon of fluorescent fading in the processing of fluorescent images can be applied with high precision and efficiency. Carry out the quality judgment of the sample.

To illustrate the implementation aspects of the present invention, there are the following various aspects.

(Aspect 1)

A method that includes:

The fluorescence image data acquisition process is to obtain the fluorescence image as the fluorescence image data corresponding to the combination of the excitation wavelength and the fluorescence wavelength. The fluorescence image is for sample irradiation with known quality distinction. The excitation light of the predetermined excitation wavelength is based on the predetermined period P(P>0) starting from the time T(T≧0) set with the start of the excitation light irradiation as the starting point and considering the fading of the fluorescence Fluorescent portrait derived from the intensity of the reflected light of the fluorescent wavelength;

The machine learning program is to create input image data with a predetermined number of channels from the aforementioned fluorescent image data, and use the input image data as training data to perform machine learning using a computer to construct a reference to the components contained in the aforementioned sample In terms of classifiers that give the most suitable quality distinction; and

The quality classification judgment procedure is to apply the aforementioned classifier to the input image data obtained from the fluorescent image data of the sample containing the unknown quality classification to determine the quality classification contained in the sample.

(Aspect 2)

In the method described in aspect 1, machine learning is performed that also considers the shape of the object existing in the fluorescent image data, so that it is reflected in the judgment of the quality classification included in the sample.

(Aspect 3)

In the method described in aspect 1 or aspect 2, a plurality of the aforementioned times T (0≦T ₁ <...<T _n ) are set, and the fluorescence obtained from each time T _i (0≦i≦n) The input image data with the predetermined number of channels created by the image data is used as the training data for each of the above-mentioned plural times, and the computer is used for machine learning, and the components contained in the above-mentioned sample Each of these moments is assigned a classifier of the most suitable quality classification, and then when determining the quality classification, a classifier suitable for the quality classification is selected to determine the quality classification contained in the sample.

(Aspect 4)

In the method described in aspect 1 or aspect 2, a plurality of the aforementioned times T (0≦T ₁ <...<T _n ) are set, and the fluorescent image obtained at each time T _i (0≦i≦n) The data is accumulated in time series to create cumulative fluorescent image data. The input image data with a predetermined number of channels created from the above cumulative fluorescent image data is used as training data, and the computer is used for machine learning to construct A classifier that gives the most suitable quality classification to the components contained in it.

(Aspect 5)

In the methods described in aspects 1 to 4, the aforementioned time T is the time before the fading of the fluorescent light starts.

(Aspect 6)

In the method described in any one of aspects 1 to 5, the time T and the period P are set to be dependent on the quality classification to improve the accuracy of the quality classification.

(Aspect 7)

In the method described in aspects 1 to 6, it further includes a visual image data obtaining program for obtaining visual image data, and the input image data with a predetermined number of channels made from the aforementioned fluorescent image data and the visual image data, to The input image data is used as training data for machine learning using a computer.

(Aspect 8)

In the method described in aspect 7, the aforementioned visual image data is composed of R (wavelength: value near 680 nm), G (wavelength: value near 560 nm), and B (wavelength: value near 450 nm).

(Aspect 9)

In the methods described in aspects 1 to 8, the excitation wavelength and the fluorescence wavelength are suitable for the detection of polyphenol and chlorophyll.

(Aspect 10)

In the method described in aspects 1 to 9, the sample is a tobacco raw material, and the quality classification is the mesophyll (lamina) of Burley species, the mesophyll of yellow species, the mesophyll of oriental (orient) species, and the Bone (leaf vein) (stem), tobacco sheet (sheet), puffed dry body (puff).

(Aspect 11)

A program for a computer to execute the methods described in aspects 1 to 10.

(Aspect 12)

A device with:

The machine learning method is to obtain the fluorescence image data corresponding to the combination of the excitation wavelength and the fluorescence wavelength related to the sample containing the known quality classification, and create a predetermined channel from the fluorescence image data ( Channel) number of input image data, use the input image data as training data to perform machine learning to construct a classifier that gives the most suitable quality distinction to the components contained in the aforementioned sample, based on the fluorescent image data The intensity of the reflected light of the predetermined fluorescent wavelength obtained in the predetermined period P (P>0) starting from the time T (T≧0) set in consideration of the fading of the fluorescent light starting from the start of the irradiation of the excitation light Fluorescent portrait data obtained; and

The quality classification judging means is to apply the classifier to the input image data obtained from the fluorescent image data of the sample containing the unknown quality classification to determine the quality classification contained in the sample.

(Aspect 13)

In the device described in aspect 12, machine learning that also considers the shape of an object existing in the fluorescent image data is performed, and this is reflected in the judgment of the quality classification included in the sample.

(Aspect 14)

In the device described in aspect 12 or aspect 13, input image data with a predetermined number of channels created from the fluorescent image data at the respective acquisition time T is used as the training data at each acquisition time. The computer performs machine learning to construct a classifier that gives the most suitable quality classification to the components contained in the aforementioned sample. Then, when the quality classification is judged, the classifier suitable for the quality classification is selected to determine the sample Included quality distinction.

(Aspect 15)

In the device described in aspect 12 or aspect 13, the fluorescent image data that are different from each other at the acquisition time T are accumulated in time series to form accumulated fluorescent image data, which is created from the aforementioned accumulated fluorescent image data The input image data with a predetermined number of channels is used as training data and machine learning is performed using a computer to construct a classifier that gives the most suitable quality classification to the components contained in the aforementioned sample.

(Aspect 16)

In the device described in aspects 12 to 15, the visual image data is also obtained, and input image data having a predetermined number of channels is formed from the aforementioned fluorescent image data and the visual image data, and the input image data is used as training data. Use computers for machine learning.

(Aspect 17)

In the device of aspect 16, the aforementioned visual image data is composed of R (wavelength: value near 680 nm), G (wavelength: value near 560 nm), and B (wavelength: value near 450 nm).

In the above aspect, the so-called "nearby value" refers to a value within a predetermined range above and below the center wavelength. The predetermined amplitude can be set to, for example, 5 to 10 nm.

In addition, the so-called "program" is a data processing method described in any language or description method, and can be in the form of source code, binary code, or the like. The "program" can be a single integrated structure, or it can be a structure dispersed into multiple modules or libraries, or a structure that cooperates with other existing programs to achieve its functions.

In addition, the "device" can be constituted by hardware, but it can also be constituted by a combination of means for realizing various functions using computer software. The function realization means may include, for example, a program module.

In addition, the quality classification of tobacco of aspect 10 will be described later.

According to the present invention, it is possible to achieve a high-precision and efficient sample quality judgment in consideration of the phenomenon of fluorescence fading.

600: Fluorescence imaging system

610: Spectroscopic lighting device

612: Xenon light source

614, 622: band pass filter

620: Spectrophotographic device

624: Photography installation

630: sample

710: Portrait of the sth floor

720: filter

730: Portrait of the s+1 layer

1110: Machine learning methods

1115: Classifier that has been learned

1120: Quality discrimination means

S01~S03: Program

Fig. 1 is a flow chart for explaining an outline of the implementation related to the method of the present invention.

FIG. 2 is an explanatory diagram showing the outline of the spectrum of the fluorescence emitted from the measurement object when the measurement object is irradiated with excitation light.

Figure 3 is a graph showing three-dimensional contour shapes of an example of fluorescent fingerprints.

Figure 4 is a graph showing the contour shape of an example of fluorescent fingerprints in two dimensions.

Figure 5 is an explanatory diagram showing an example of the hyperspectral image and multispectral image of fluorescent fingerprints in the form of a schematic diagram.

Figure 6 is a diagram showing an example of a fluorescent imaging system.

Fig. 7 is an explanatory diagram showing a processing example of the convolutional layer.

Figure 8 is an explanatory diagram showing the structure of AlexNet.

Figure 9 is an explanatory diagram showing the structure of GoogLeNet.

Figure 10 is an explanatory diagram showing the structure of SegNet.

Fig. 11 is a block diagram for explaining an outline of the implementation related to the device of the present invention.

Figure 12A is a visual image of a sample of tobacco raw material taken under white light.

Figure 12B is the fluorescent image obtained by irradiating the tobacco raw material sample with excitation light without passing through the filter.

Figure 13 is a fluorescence profile obtained with excitation-fluorescence wavelength combinations I to VI for a tobacco raw material sample.

Figure 14 is a schematic diagram used to illustrate the outline of the application of the semantic segmentation method to tobacco raw material samples.

Figure 15 is a table showing an example of the selection of input image data in the case of machine learning for a sample of tobacco raw materials that combines fluorescent image data and visual image data.

Figure 16 is a table showing the judgment results obtained by applying the learned convolutional neural network to test specimens of known tobacco raw materials.

Hereinafter, an example of an embodiment related to the method of the present invention will be explained, an example of an embodiment related to the device of the present invention will be explained, and an example of applying the method of the present invention to the judgment of the quality of tobacco raw materials will be explained.

Please note that the present invention is not limited to the implementation examples described below.

I. Examples of implementation aspects related to the method of the present invention

I-1. One aspect of implementation

Fig. 1 is a flowchart for explaining the outline of an implementation aspect (implementation aspect I-1) related to the method of the present invention. As shown in Figure 1, the implementation pattern I-1 contains a fluorescent image Data acquisition program (S01), machine learning program (S02), and quality classification judgment program (S03). Hereinafter, the above-mentioned procedures will be described.

I-1-1. Procedure for obtaining fluorescent image data (S01)

This procedure is a procedure to obtain a fluorescent image as a number of fluorescent image data corresponding to the combination of excitation wavelength and fluorescent wavelength. The fluorescent image has a predetermined value for sample irradiation with a known quality distinction The excitation light of the excitation wavelength is based on the predetermined fluorescent light obtained in the predetermined period P (P>0) starting at the time T (T≧0) set in consideration of the fading of the fluorescent light with the start of the excitation light irradiation as the starting point Fluorescent image created by the intensity of the reflected light of the wavelength.

Here, we will briefly explain how to obtain fluorescent image data.

For the acquisition of fluorescent image data, the fluorescent imaging system 600 shown in Figure 6 is used. The fluorescent imaging system 600 is composed of a spectroscopic illumination device 610 and a spectroscopic photography device 620. The spectroscopic lighting device 610 is, for example, a xenon lamp light source 612 (MAX-303 (trade name), Asahi Spectroscopy (company name)) through a band-pass filter 614 that can be adjusted in the range of 340 to 420 nm with a scale of 10 nm. The emitted excitation light of a specific wavelength is irradiated to the sample 630. The spectroscopic imaging device 620 allows the reflected light from the sample 630 to pass through the band-pass filter 622 that can be adjusted in the range of 420 to 700 nm with a scale of 10 nm, and the imaging device 624 only captures fluorescent images of specific wavelengths. By separately changing the band-pass filter 614 of the light source and the band-pass filter 622 on the side of the imaging device to photograph the sample, a number of copies corresponding to the combination of the excitation wavelength and the fluorescence wavelength can be obtained as shown in Figure 5. Fluorescent portrait information. In Fig. 6, the size of the sample is 4×4 cm, but it is not particularly limited to this value, and the size can be appropriately set. In addition, it is also possible to synthesize multiple images obtained as fluorescent image data.

Furthermore, the time T (T≧0) set in consideration of the fluorescence fading from the start of the excitation light irradiation as the acquisition time of the fluorescent image data is a configuration that takes into account the aforementioned fluorescent fading problem. Time T can be the time before the start of fluorescence fading, but because the degree of fluorescence fading has a non-linear characteristic with time, such fluorescence fading characteristics can be used to obtain fluorescent image data suitable for classification The required time and period are set as the time T and the period P suitable for each quality classification.

As described above, the time T and the period P can be set to be dependent on the quality classification in the sample to improve the judgment accuracy of the quality classification, and they can be reflected in the learning result described later to further improve the judgment accuracy.

I-1-2. Machine Learning Program (S02)

This program is to create input image data with a predetermined number of channels from the fluorescent image data obtained in the aforementioned fluorescent image data acquisition program, and use the input image data as training data to perform machine learning on a computer to construct a test For the components contained in the sample, it is a program that gives the most suitable quality classification classifier.

First, if necessary, perform pre-processing on the fluorescent image data acquired in the aforementioned fluorescent image data acquisition program (S01) to create input image data. In the case where the number of channels of the acquired fluorescent image data is K ₁ , it is not necessary that the number of channels K of the input image data is also K ₁ , and may be a value smaller than K ₁ (implementation of reduction). In the case of implementing such a reduction, how to organize and integrate the data can be appropriately determined in consideration of preliminary investigations conducted in advance and known results.

In addition to pre-processing such as data removal, response to outliers, size changes, and data volume adjustments, there are also: data conversion to converge to a certain range. Regularization of data, standardization of data processed so that the average of the data is 0, and the standard deviation of 1, the non-correlation of the data that removes the correlation between the data, the standardization of the data, and the non-correlation of the data. The processing of whitening and other related data can be appropriately selected. If the training data has been pre-processed, the test data and unknown data must be processed in the same way.

Next, the method of machine learning suitable for classification is explained.

The method of machine learning suitable for classification, except for example: training of tree structure, decision tree (decision tree) that classifies data by branching; combining multiple decision trees, by taking the output of each decision tree The majority decision is used to perform random forest (Random Forest: RF) for classification; to perform multi-dimensional hyperplane training to perform support vector machine (SVM) for data classification; to use the nearest K points K Nearest Neighbor method (KNN method), which is mostly used for classification; In addition to combining a plurality of simple classifiers to construct a nonlinear classifier, ensemble learning can also be used A neural network constructed by modeling a network of brain nerve cells. Regarding neural networks, it has been known that the use of multilayer neural networks, especially deep learning using Convolutional Neural Networks (CNN), is effective for image classification. The outline of the convolutional neural network method will be described later. In addition, for example, Bag of Visual Words (BoVW) can also be used. BoVW is a method derived from the analogy of the Bag of Words (BoW; also known as Bag of Features) that is subordinate to the model for calculating article features. It is a method of applying text classification technology to portrait classification.

In addition to the above-mentioned methods, for example, partial least squares discriminant analysis (Partial Least Squares-Discriminant Analysis: PLS-DA), which is effective for the classification of high-dimensional data, can be used, which belongs to the statistics of variables following the Bernoulli distribution. A regression model is a statistical method such as Logistic Regression (LR).

Below, the outline of deep learning using convolutional neural networks suitable for image classification will be explained.

<Outline of Convolutional Neural Network Structure>

Convolutional neural network is composed of three layers: convolution layer, pooling layer, and fully connected layer. It is mainly used in image recognition feedforward neural network (feedforward neural network). network), which is a neural network that uses back propagation and stochastic gradient descent (Stochastic Gradient Descent: SGD) to optimize learning. Also, because the purpose is classification, there are cases where a softmax layer is used to normalize the output of the fully connected layer, and a classification layer that uses the output of the softmax layer as the input and outputs the probability of classification.

In a convolutional neural network, the convolutional layer and the pooling layer are repeated multiple times and then connected to the fully connected layer. The fully connected layer is repeated multiple times and the output of the final fully connected layer is used as the input of the softmax layer.

The convolutional layer performs filtering processing that applies a plurality of filters to the input image, and uses the filter processing to convert the input image into a plurality of images representing the characteristics of the image. Then, the pooling layer reduces the size of the image in a way that does not damage the characteristics of the image, and the fully connected layer connects all the neurons between the layers.

In the following, the features and behaviors of the convolutional layer, the pooling layer, and the fully connected layer, as well as the learning in the convolutional neural network, are briefly described.

<Convolutional layer>

Convolution (convolution) calculation can strengthen or weaken a certain feature of the image, so the convolution layer has the function of converting the input image into a more emphasized image through such a convolution process.

In addition, the image has the so-called locality, and the convolutional layer also has the function of detecting the characteristics of the image by using the locality of such an image. Therefore, the convolutional layer uses multiple filters that detect different features to detect features.

Figure 7 is a diagram showing a processing example of the convolutional layer. Figure 7 shows the application size for the image (input image) 710 of the sth layer whose size (width × height × number of channels) is M ^s × N ^s × K ^s The calculation result for the case of processing K ^s+1 filters 720 of P ^s+1 ×Q ^s+1 ×K ^s .

Generally speaking, in a neural network, the propagation of information from input to output is called forward propagation, and the backtracking of information from output to input is called back propagation.

For the example shown in Figure 7, regarding forward propagation, if the width and height (p, q) of the channel k on the k'th filter are weighted as w ^s+1 _{p, q, k, k '} , The input of channel k'at the position (m,n) of the portrait (output portrait) of the s+1 layer is a ^s+1 _m,n,k' , then a ^s+1 _{m,n,k 'Is} expressed as the following formula (1).

a ^s+1 _m,n,k' =Σ _p,q,k w ^s+1 _p,q,k,k' ‧z ^s _m+p,n+q,k +b ^s+1 _k' (1 )

Among them, z ^s _m+p,n+q,k is the pixel value of channel k at the position (m+p,n+q) of the portrait (input portrait) 710 in the s- _th layer, b ^s+1 _k' It is bias.

Therefore, the pixel value z ^s+1 _m,n,k' of the channel k'at the position (m,n) of the image (output image) 730 of the s+1 layer can be activated using the activation function h(‧) and It is expressed as the following formula (2).

z ^s+1 _m,n,k' = h(a ^s+1 _m,n,k' ) (2)

According to the neuron model, the weight system is equivalent to the transmission efficiency of the synapse, and the deviation system is equivalent to the neuron's sensitivity (the degree of easy excitement). In addition, the activation function can be said to be a function to excite neurons.

Next, the back propagation in the example shown in Figure 7 will be explained. The error back propagation system will trace back the output obtained by forward propagation and the error of the forward solution layer by layer and update the weights and deviations to minimize the error. And the progresser. Hereinafter, the function that defines the error between the output and the positive solution is referred to as a loss function.

In order to find the back propagation of the error in the convolutional layer, the gradient of the weight must be obtained.

Therefore, if the loss function is expressed as J to obtain the partial differential with respect to the weight, it can be expressed as the following equation (3).

In addition, the following equation (4) is derived from equation (1),

的話，可表示成如下的式(5)。 Further export

If it is, it can be expressed as the following formula (5).

The partial differential for the deviation can also be expressed as the following equation (6) by the same calculation.

Therefore, δ ^s _m,n,k can be expressed as the following formula (7),

Among them, h'(‧) is the differential value of the activation function h(‧).

Equation (7) indicates that the error of the input layer (lower layer) can be calculated using the value obtained by convolution processing the error of the output layer (upper layer) with weight.

Regarding the activation function, because the network needs more than three layers to function effectively, the activation function must be non-linear. Moreover, as described above, in order to make machine learning by backpropagation, the activation function must be differentiated.

A typical activation function has a logistic sigmoid function (logistic sigmoid function) shown in the following formula (8).

h(x)=σ(x)=1/(1+exp(-x)) (8)

In the case of back propagation, because the product of the error and the differential coefficient is used to calculate the error of the lower layer, the value of the error δ ^s will decrease exponentially as the lower layer goes, so the value of x will be input If it is large (or small), the value of the differential coefficient h'(x) becomes small, and there is almost no problem of updating the parameters. In order to alleviate this problem, some people have proposed Rectified Linear Unit (ReLU) represented by the following equations (9) and (10) as the activation function.

h(x)=max(0,x) (9)

In addition, some people have proposed LReLU (Leaky ReLU), RReLU (Random ReLU), PReLU (Parametric ReLU), etc., which belong to the improved version of ReLU, and ELU (Exponential Linear Unit) that contributes to the standardization of data. As the activation function, the detailed description is omitted here.

<Pooling layer>

Next, the pooling layer will be described. The pooling layer performs a process called pooling to divide the portrait into regions, and then extract the values representing the regions and arrange them to create a new portrait. Pooling is also called the process of blurring the image, which can reduce the sensitivity of the object's position. In this way, even if the position of the object changes slightly, the output can be kept unchanged. Therefore, the pooling layer provides robustness to changes in position. Moreover, because pooling will reduce the size of the image, it also has the effect of reducing the amount of calculation.

In image processing, the maximum pooling method in which the maximum value of each area is used as the value representing each area is mostly used.

<Fully connected layer>

The fully connected layer is a layer used in normal neural networks, and is usually arranged after the convolutional layer and the pooling layer that are repeated several times.

<Learning in Convolutional Neural Networks>

Convolutional neural network is the same as normal neural network for back propagation learning.

In the convolutional layer, the gradient of each value constituting the filter is calculated based on the value propagated from the error between the output and the correct solution, and the filter is updated. Moreover, the deviation is also updated similarly. The error is propagated to the upper layer through the convolutional layer.

In the pooling layer, no learning is performed, and errors are propagated to higher layers through the pooling layer.

In the fully connected layer, errors are propagated in the same way as a normal neural network.

Through such adjustment of weights and deviations, the neural network is optimized in a way that minimizes the error. The algorithm used for such optimization is known as the stochastic gradient descent method.

The stochastic gradient descent method randomly selects the algorithm of the sample every time it is updated, and the update formula of the weight w and the deviation b is the following formula.

In addition to the optimization algorithm, it is known that the inertia term is added to the Momentum in the stochastic gradient descent method, the AdaGrad that automatically adjusts the update amount, and the overcoming AdaGrad will cause the learning to stagnate due to the decrease in the update amount. Methods such as RMSProp of weakness, Adam (Adaptive moment estimation), which has the feature of integrating Momentum and AdaGrad, are omitted here in detail.

I-1-3. Quality classification judgment procedure (3)

Finally, the learned classifier is applied to the input image data obtained from the fluorescent image data of the sample containing the unknown quality classification, and the quality classification contained in the sample is determined (quality classification judgment procedure: S03). As mentioned above, in the case of pre-processing the training data, the unknown data must also be processed in the same way. Furthermore, it is preferable to adapt the acquisition time and period of the fluorescent image data of the sample including the unknown quality classification to the acquisition time and period of the fluorescent image data during learning.

With such a configuration, it is possible to accurately reflect the time change of the fluorescence fading, and further improve the classification accuracy.

<Example of actual configuration of convolutional neural network>

As an example of the actual structure of a convolutional neural network, the following examples are AlexNet and GoogLeNet to illustrate the outline of the two.

As shown in Figure 8, AlexNet is composed of five convolutional layers and three fully connected layers. After the first and second convolutional layers, the normalization layer is used. After each normalization layer and the fifth After each convolutional layer, the maximum pooling layer is used. The AlexNet system has 60 million parameters and 650,000 neurons. Because it contains very large parameters, a lot of effort has been made in learning to avoid getting local solutions through dropout, data expansion, etc.

GoogLeNet, as shown in Figure 9, uses a module called Inception that also has a network in the layer to construct a deep network that connects 9 such modules. In the Inception module, there are three convolutional layers of 1×1, 3×3, and 5×5 in parallel with the maximum pooling layer. By arranging multiple convolutional layers in parallel, a partial image with a complex number of widths can be obtained. Correlation. In addition, the 1×1 convolutional layer provided in the front stage of the 3×3, 5×5 convolutional layer has the function of dimensional reduction. Although GoogLeNet has a deeper structure than AlexNet, because these are all convolutional layers, the number of parameters is only about 1/12.

I-2. Another aspect of implementation

Next, the outline of another implementation aspect (Implementation aspect I-2) related to the method of the present invention will be described, but the description of the same parts as the implementation aspect I-1 will not be repeated and will be omitted.

The implementation mode I-2 is to perform machine learning that also considers the shape of the object existing in the fluorescent image data, so that it is reflected in the judgment of the quality classification included in the aforementioned sample.

In order to perform such machine learning that also considers the shape of the object, a program called semantic segmentation can be used to accurately estimate the class and contour of the object in the image. Semantic segmentation uses pixel level to identify objects, which can be classified (classified) by pixel level. Someone once proposed an encoder-decoder network as a semantic segmentation method using deep learning.

Figure 10 shows a network called SegNet for semantic segmentation. The outline of its structure and operation is as follows.

The encoder of SegNet is made by removing the full connection layer of the network for class recognition called VGG16. The decoder is a network that reverses the input and output of the encoder, and a softmax layer is added to the final layer. The image input to the decoder receives upsampling in the unpooling layer. The inverse pooling layer corresponds to the pooling layer of the encoder one-to-one, and the unit that outputs the maximum value during encoding is memorized, and the decoder is up-sampled according to the memory. Using the information during coding not only supplements the spatial information, but also does not require learning of up-sampling methods.

The image enlarged by inverse pooling will become a lot of loose images with a value of 0, so the transposed convolution layer connected to the inverse pooling layer is used to convert the loose images into dense images. The output image of the final decoder is input to the softmax layer, and each pixel is converted to the probability of the object category one by one, and then each pixel is represented by the category with the highest probability, and the final segmentation result is obtained in this way. SegNet is a small batch (mini- The total cross-entropy loss of all pixels in the batch is used as a loss function, and end-to-end learning is performed by the stochastic gradient descent method. The so-called small batch refers to a part of the data (a small sample set) randomly selected from all the data (samples).

With such a configuration, even in the case where a plurality of quality classifications share a part of the material, since the shape is taken into consideration, the accuracy of the quality classification judgment can be further improved.

I-3. Another aspect of implementation

Next, the outline of another implementation aspect (Implementation aspect I-3) related to the method of the present invention will be explained, but the description of the same parts as the implementation aspects I-1 and I-2 will not be repeated. It is omitted.

Implementation mode I-3 is to set a plurality of times (0≦T ₁ <...<T _n ) at the time T set in consideration of the fading of the fluorescent light starting from the start of the excitation light irradiation, and from each time T _i (0 ≦i≦n) The input image data with a predetermined number of channels created by the fluorescent image data obtained is used as the training data at each of the above-mentioned plural times. In other words, the most suitable quality classification classifier is assigned to each of the aforementioned plural times, and then when the quality classification is judged, the classifier suitable for the quality classification is selected to determine the quality classification contained in the sample.

Regarding the selection of a suitable classifier for a sample containing components of unknown quality, for example, it can be selected to be the same as the acquisition time of the fluorescent image data starting from the start of the excitation light for the unknown sample. The corresponding classifier.

With such a configuration, it is possible to accurately reflect the time change of the fluorescence fading, and further improve the classification accuracy.

I-4. Another aspect of implementation

Next, the outline of yet another implementation aspect (Implementation aspect I-4) related to the method of the present invention will be described, but the same parts as the implementation aspects I-1, I-2, and I-3 will not be repeated. The description is omitted.

The implementation mode I-4 is to set a plurality of times (0≦T ₁ <...<T _n ) at the time T set in consideration of the fading of the fluorescent light starting from the start of the excitation light irradiation, and set each time T _i (0≦ i≦n) The fluorescent image data obtained are accumulated in time series to form fluorescent image data, and the input image data made from the fluorescent image data is used as training data for machine learning.

In the implementation aspect I-4, the convolutional layer may be a three-dimensional convolutional layer that also reflects the dimension of time, instead of a two-dimensional convolutional layer.

With such a configuration, it is possible to use the fluorescent image data accumulated in the time series to more precisely grasp the time change of the fluorescent fading, and the classification accuracy can be further improved.

In addition, the acquisition time of the fluorescent image data of the unknown sample is best adapted to the acquisition time of the fluorescent image data at the time of machine learning. However, in the implementation mode I-4, the machine is time-series The fluorescence image data during learning is accumulated, and it can be thought that it will retain the characteristics of each acquisition time. Therefore, even if the fluorescence image data of an unknown sample is acquired at any time, it can be expected to ensure the necessary classification accuracy.

I-5. Another aspect of implementation

Next, the outline of yet another implementation aspect (implementation aspect I-5) related to the method of the present invention will be described, but the implementation aspects I-1, I-2, I-3, and I- will not be repeated. 4 The description of the same parts will be omitted.

The implementation mode I-5 also includes a visual image acquisition procedure for obtaining visual image data, and from the aforementioned fluorescent image data and the visual image data, an input image data with a predetermined number of channels is created, and the input image data is used as training data And for machine learning.

Visual image data refers to hyperspectral images or multispectral images with wavelengths within the visible wavelength range.

Here, the acquisition of visual image data will be briefly explained.

To obtain visual image data, for example, in the fluorescent imaging system 600 shown in FIG. 6, which is used to obtain fluorescent image data, the band-pass filter 614 of the spectroscopic illumination device 610 is not used, but The sample 530 is illuminated with white light emitted from a xenon light source. The spectroscopic imaging device 620 allows the reflected light reflected from the sample 630 to pass through the band-pass filter 622 corresponding to the specific visible wavelength, and the imaging device 624 captures only the visible image of the specific wavelength. By changing the band-pass filter 622 on the side of the imaging device to photograph the sample, visual image data corresponding to multiple visible wavelengths can be obtained. The specific visible wavelength can be R (wavelength: near 680nm), G (wavelength: near 560nm), B (wavelength: near 450nm), but it is not necessarily limited to these wavelengths, and other wavelengths can also be used. Or use one or two of RGB.

The feature of the implementation mode I-5 is that it uses both the acquired fluorescent image data and the visual image data to form the input image data. Moreover, when the number of channels of the acquired fluorescent image data and the visual image data are K ₁ and K ₂ respectively, it is not necessary that the number of channels K of the input image data is K ₁ +K ₂ , and K can be less than The value of K ₁ + K ₂ (implementation of reduction). In the case of implementing such a reduction, how to organize and integrate the data can be appropriately determined in consideration of preliminary investigations conducted in advance and known results.

By using both the fluorescent image data and the visual image data, the accuracy of the quality classification can be improved.

II. An aspect of implementation related to the device of the present invention

FIG. 11 is a block diagram for explaining the outline of an implementation aspect (implementation aspect II) related to the device of the present invention.

The implementation aspect II is equipped with: a machine learning means 1110 for constructing a classifier and a quality discrimination determination means 1120, which roughly correspond to the aforementioned implementation aspects I-1 and I-5.

The machine learning means 1110 is to input the fluorescent image data of the sample containing the known quality classification, or the fluorescent image data and the visual image data, and form the input image data with a predetermined number of channels from the fluorescent image data , Use the input image data as training data to perform machine learning, and construct a classifier that gives the most suitable quality classification to the aforementioned samples. In addition, the fluorescence image data contains the number of sheets (the number of channels) corresponding to the combination of the excitation wavelength and the fluorescence wavelength related to the sample, and is based on the consideration of fluorescence from the start of the excitation light irradiation. The intensity of the reflected light of the predetermined fluorescent wavelength obtained during the predetermined period from the time T (T≧0) set for the fading.

The quality classification judging means 1120 applies the learned classifier constructed by the machine learning means 1110 to the input image data obtained from the fluorescent image data of the sample whose quality classification is unknown, and judges that the sample contains The quality of distinction.

The details of the learning method, etc., are as explained in Implementation Mode I-1.

In addition, the modified example of the implementation aspect II can be derived from the aspects corresponding to the aforementioned implementation aspects I-2, I-3, and I-4, but there are substantial overlaps in the content, so the description will be omitted.

III. Examples of states applied to the judgment of quality distinction of tobacco raw materials

Hereinafter, an implementation mode of applying the aforementioned method to the judgment of the quality classification of tobacco raw materials will be described.

First, an example of the quality classification of tobacco raw materials is roughly as follows.

<Quality Classification of Tobacco Raw Materials>

‧BLY: the mesophyll of the tobacco leaf

‧Mesophyll of yellow species (FCV): the mesophyll of tobacco leaves of yellow species

‧The mesophyll of oriental species (ORI): the mesophyll of oriental tobacco leaves

‧Stem: the veins of tobacco leaves

‧Tobacco sheet: The main raw material of mesophyll, middle bone, etc. is formed by adding fiber, additives, etc. into a sheet (see, for example, Japanese Patent No. 3872341)

‧Puff: It is made by moistening and swelling the shredded middle bone and then drying it (refer to Japanese Patent No. 5948316, for example)

Next, an aspect of the implementation related to the quality classification judgment of tobacco raw materials will be explained. Please note that the present invention is not limited to this aspect.

<Preparation for using sample>

Tobacco raw materials belonging to the above-mentioned quality classification are used as samples for use, but the mesophyll is prepared by cutting the mesophyll of tobacco from different countries of production into a size of about 2mm in width and then mixing them. The middle bone system varies the country of origin. The middle bones of tobacco are mixed and prepared.

<Preliminary Discussion and Verification>

In order to confirm whether the aforementioned method can be applied to the determination of the quality of tobacco raw materials, preliminary investigations and verifications as described below are carried out.

(1) Discussion and verification of the combination of excitation wavelength and fluorescence wavelength

In order to obtain fluorescence image data, the combination of excitation wavelength and fluorescence wavelength must be determined. Therefore, considering the knowledge and insights related to tobacco raw materials obtained in the past, the excitation wavelengths of polyphenols and chlorophyll are selected as candidates for excitation and fluorescence wavelengths, and the appropriateness of the selection is verified.

First, the visible image obtained under white light is irradiated with excitation light on the tobacco raw material sample shown in Figure 12A without passing through the filter to obtain a fluorescent image as shown in Figure 12B. From the fluorescent image in Figure 12B, it was confirmed that the spatial resolution of a piece of shredded tobacco can be detected without a filter. In addition, it was confirmed that the above-mentioned quality classification can be roughly discriminated.

Next, for the combination of excitation and fluorescence wavelength (nm) as I (340,460), II (360,440), III (380,680), IV (400,660), V (400,680), VI (420,660), use the combination shown in Figure 6 The fluorescent imaging system shown, obtains the fluorescent image shown in Figure 13. The operation of the fluorescent imaging system shown in Fig. 6 has been described in detail in "I-1-1.", so the description will be omitted.

It can be judged from Figure 13 that the mesophyll part (BLY) of the Belle species (BLY), the mesophyll part of the yellow species (FCV), the mesophyll part (ORI) of the oriental species (ORI), the stem, the tobacco sheet (sheet), and the puffing The appropriateness of the distinction between the various qualities of the puff, so it is confirmed that it is effective to use the combination of excitation and fluorescence wavelengths I to VI as described above in the judgment of the quality distinction.

(2) Verification of the effectiveness of semantic segmentation

As mentioned above, semantic segmentation uses the pixel level to identify objects, which can achieve the classification (classification) of the pixel level.

For example, with regard to tobacco raw materials, there is a possibility that a part of the material is shared, such as stem, sheet, and puff. However, even in this case, it is also possible to consider the shape It is expected that the accuracy of the quality classification can be further improved.

Therefore, in order to verify this point, an experiment to confirm the effectiveness of applying semantic segmentation as described below was performed.

Figure 14 is a schematic diagram used to illustrate the outline of the application of semantic segmentation to tobacco raw material samples. The outline of the program shown in Figure 14 is as follows.

For the fluorescent image (A) obtained by using the combination of excitation and fluorescence wavelengths I to VI as described above, the image (B) is obtained by applying the above-mentioned SegNet, which is one of the semantic segmentation methods.

Then, the portrait (B) is contrasted and emphasized and then binarized to obtain the portrait (C).

Then, useless objects are removed from the portrait (C) to obtain the portrait (D).

Finally, each object in the portrait (D) is labeled.

As a result, it was confirmed that the appropriateness of the shape was considered using such a procedure.

(3) Verification of the effectiveness of the combined use of both fluorescent images and visual images

Only use three-channel visible light data, namely R (wavelength: near 680nm), G (wavelength: near 560nm), B (wavelength: near 450nm) multispectral data (visual image data), for Cypress Mesophyll of Lai species (BLY), Mesophyll of yellow species (FCV), Mesophyll of Oriental species Department (ORI), using convolutional neural networks (AlexNet, GoogLeNet) for deep learning. However, the recognition rate in this case is about 68%.

Therefore, it is predicted that if both the fluorescent image data and the visual image data are used to create the input image data, and machine learning is performed with the input image data, it is expected that the accuracy of the quality classification of tobacco raw materials can be further improved.

In order to verify this point, an experiment on the effectiveness of both fluorescent images and visual images was carried out.

First of all, for the mesophyll part (BLY) of the Bailey species, the mesophyll part (FCV) of the yellow species, the mesophyll part (ORI) of the oriental species, the middle bone (stem), the tobacco sheet (sheet), and the puff Cut into shreds, prepare multi-spectral data of R (wavelength: near 680nm), G (wavelength: near 560nm), and B (wavelength: near 450nm) from the visual image data. Fluorescence image data are excitation and fluorescence The 6-channel image data of tobacco raw material samples composed of multispectral data with wavelengths (nm) of (300,450), (340,450), (380,650) are about 500.

Next, the 6-channel image obtained in this way is reduced to 3 channels to create an input image for learning. Figure 15 is a table showing an example of 3-channel input image data using fluorescent image data and visual image data for machine learning about tobacco raw material samples. This table shows, for example, that the data of channel 1 is R and the contribution of spectral data is 100, and the contribution of excitation and fluorescence wavelength (nm) is (340,450), (380,650) is 50 respectively.

Then, using the 3-channel input image data obtained in this way as training data, apply the aforementioned AlexNet for deep learning.

As a result, it was confirmed that it is appropriate to use both the fluorescent image and the visual image in this way.

The convolutional neural network used is not limited to AlexNet, and other known networks (such as GoogLeNet, etc.) can also be used for deep learning.

<Examples related to the judgment of the quality of tobacco raw materials>

Based on the preliminary discussion and verification as described above, a system for deep learning of tobacco raw material samples is installed, and the deep learning described below is performed.

The installed system uses SegNet, a network for semantic segmentation.

In addition, multi-spectral data with visible image data as R (wavelength: near 680 nm), G (wavelength: near 560 nm), and B (wavelength: near 450 nm) are used. The fluorescent image data is excitation and fluorescence wavelength ( nm) is the image data of the 9-channel tobacco raw material sample composed of the multispectral data of I (340,460), II (360,440), III (380,680), IV (400,660), V (400,680), VI (420,660).

The visual image data and fluorescent image data are acquired using the fluorescent imaging system shown in Figure 6, but the acquisition time and period (T, P) (unit: second) starting from the start of excitation light irradiation are considered Fluorescence faded and set as follows.

For the visual image, it is set to R(0,1), G(1,1), B(2,10). For the fluorescent image, it is set to be acquired every 10 seconds after the acquisition of the visual image. I, II, III, IV, V, VI.

According to such a setting regarding the acquisition of fluorescent images, it is possible to perform in-depth learning that utilizes the characteristics of fluorescent fading, and as a result, further improvement in the accuracy of determination can be expected.

Then, for the tobacco raw material samples with known quality distinctions, deep learning using convolutional neural networks (weight: 0.5, number of training epochs: 400) is performed. The number of training cycles 1 means to learn all the training materials once.

Figure 16 is a table showing the determination results obtained by applying the thus-learned convolutional neural network to test samples of tobacco raw materials whose quality is known.

A, B, D, O, R, and S in the table are the symbols representing FCV, BLY, Puff, ORI, Sheet, Stem.

For the description of the table, for example, the first row of the table indicates: in terms of the actual number of tobacco raw material A (FCV) 162, it is estimated that the numbers of A, B, D, O, R, and S are respectively 99 ,5,11,33,4,10. In addition, the "precision" in the table indicates, for example, the ratio of the estimated number of "A" to the actual number of "A".

From Table (B), calculate the overall judgment accuracy rate in the following way:

Total number of correct judgments (99+115+137+115+139+129) ÷ total number (162+140+164+159+142+159)=79%

In this way, although the judgment accuracy rate varies slightly depending on the type of raw material, as a whole, a high judgment accuracy rate that can ensure practical effectiveness is achieved.

Please note that in addition to the above-mentioned implementation aspects, the present invention can also adopt various different implementation aspects within the scope of the technical ideas described in the scope of the patent application.

S01~S03: Program

Claims (17)

A method that includes: The fluorescence image data acquisition process is to obtain the fluorescence image as the fluorescence image data corresponding to the combination of the excitation wavelength and the fluorescence wavelength. The fluorescence image is for sample irradiation with known quality distinction. The excitation light of the predetermined excitation wavelength is based on the predetermined period P(P>0) starting from the time T(T≧0) set with the start of the excitation light irradiation as the starting point and considering the fading of the fluorescence Fluorescent portrait derived from the intensity of the reflected light of the fluorescent wavelength; The machine learning program is to create input image data with a predetermined number of channels from the aforementioned fluorescent image data, use the input image data as training data and use a computer to perform machine learning, and construct an assignment to the components contained in the aforementioned sample The most suitable quality classification classifier; and The quality classification judgment procedure is to apply the aforementioned classifier to the input image data obtained from the fluorescent image data of the sample containing the unknown quality classification to determine the quality classification contained in the sample. Such as the method described in item 1 of the scope of patent application, wherein: The aforementioned method is to perform machine learning that also considers the shape of the object existing in the fluorescent image data, so that it is reflected in the judgment of the quality classification included in the aforementioned sample. Such as the method described in item 1 or 2 of the scope of patent application, wherein: Set a plurality of the aforementioned times T (0≦T ₁ <...<T _n ), and create input image data with a predetermined number of channels from the fluorescent image data obtained at each time T _i (0≦i≦n) , Using a computer to perform machine learning as training data at each of the aforementioned plural times, and construct a classifier that gives the most suitable quality distinction to the aforementioned sample at each of the aforementioned plural times, and then When determining the quality classification, a classifier suitable for the quality classification is selected to determine the quality classification contained in the sample. Such as the method described in item 1 or 2 of the scope of patent application, wherein: Set a plurality of the aforementioned times T (0≦T ₁ <...<T _n ), and accumulate the fluorescent image data obtained at each time T _i (0≦i≦n) in a time series to create cumulative fluorescent image data Using the input image data with a predetermined number of channels created from the above-mentioned accumulated fluorescent image data as training data, machine learning is performed using a computer, and a classifier that gives the most suitable quality classification to the above-mentioned sample is constructed. Such as the method described in any one of items 1 to 4 in the scope of patent application, wherein: The aforementioned time T is the time before the fading of the fluorescent light starts. Such as the method described in any one of items 1 to 5 in the scope of patent application, wherein: The aforementioned time T and period P are set to be dependent on the quality classification to improve the accuracy of the quality classification. For example, the method described in any one of items 1 to 6 in the scope of the patent application further includes: a visual image acquisition procedure for obtaining visual image data, And from the aforementioned fluorescent image data and the visual image data, an input image data with a predetermined number of channels is created, and the input image data is used as training data to perform machine learning using a computer. As the method described in item 7 of the scope of patent application, in which, The aforementioned visual image data is composed of R (wavelength: value near 680 nm), G (wavelength: value near 560 nm), and B (wavelength: value near 450 nm). Such as the method described in any one of items 1 to 8 in the scope of patent application, wherein: The excitation wavelength and fluorescence wavelength are suitable for the detection of polyphenols and chlorophyll. Such as the method described in any one of items 1 to 9 in the scope of patent application, wherein: The sample is a tobacco raw material, and the quality is divided into the mesophyll part of the Belle species, the mesophyll part of the yellow species, the mesophyll part of the Orient species, the middle bone, the tobacco slice, and the puffed dried body. A program that enables a computer to execute the method described in any one of items 1 to 10 in the scope of the patent application. A device with: The machine learning method is to obtain the fluorescence image data corresponding to the combination of the excitation wavelength and the fluorescence wavelength related to the sample containing the known quality classification, and make a predetermined number of channels from the fluorescence image data The input image data is used as training data for machine learning to construct a classifier that gives the most suitable quality distinction to the aforementioned sample. The fluorescent image data is based on the start of excitation light irradiation Fluorescent image data obtained from the intensity of the reflected light of the predetermined fluorescent wavelength obtained during the predetermined period P (P>0) starting from the time T (T≧0) set for the starting point and considering the fluorescent fading; as well as The quality classification judging means is to apply the aforementioned classifier to the input image data obtained from the fluorescent image data of the sample containing the unknown quality classification to determine the quality classification contained in the sample. As the device described in item 12 of the scope of patent application, in which, The aforementioned device performs machine learning that also considers the shape of the object existing in the aforementioned fluorescent image data, so that it is reflected in the judgment of the quality classification included in the aforementioned sample. Such as the device described in item 12 or 13 of the scope of patent application, wherein: The input image data with a predetermined number of channels created from the fluorescent image data at the respective acquisition time T is used as the training data at each acquisition time. The computer is used for machine learning to construct Each of the aforementioned acquisition moments is assigned the most suitable classifier for quality classification, and then a suitable classification is selected for samples containing unknown quality classifications When the quality classification is determined, a classifier suitable for the quality classification is selected to determine the quality classification included in the sample. Such as the device described in item 12 or 13 of the scope of patent application, wherein: Accumulate the fluorescent image data that are different from each other at the acquisition time T in time series to create cumulative fluorescent image data, and use the input image data with a predetermined number of channels created from the cumulative fluorescent image data as training data The computer performs machine learning to construct a classifier that gives the most suitable quality distinction to the aforementioned samples. The device described in any one of items 12 to 15 in the scope of patent application, wherein: The aforementioned device also obtains visual image data, and forms input image data with a predetermined number of channels from the aforementioned fluorescent image data and the visual image data, and uses the input image data as training data to perform machine learning using a computer. As the device described in item 16 of the scope of patent application, in which, The aforementioned visual image data is composed of R (wavelength: value near 680 nm), G (wavelength: value near 560 nm), and B (wavelength: value near 450 nm).

Images