JP7293644B2 - Image identification device, classifier learning method, image identification method and program - Google Patents

Description

The present invention relates to an image identification device, a classifier learning method, an image identification method, and a program.

As a technique for identifying an image, a technique for identifying an image by using image data whose identification result is known as learning data to train a deep neural network (DNN) and using the trained DNN is known. It is In the image identification technique, there is no particular restriction on the image to be identified, and arbitrary image data can be used as the identification target. For example, when an affected area such as a skin disease is photographed, a scale or the like is often photographed together in order to clarify the size of the affected area. In addition to images of affected areas, for example, in the case of images such as images of flowers where the size of the object is desired to be clarified, it is easier to understand the size by photographing the scale together. In many cases, scales and the like are also included in the image data for the camera.

In this case, it is conceivable that scales and the like appearing together have an adverse effect on image identification. A technique to do so has been proposed.

N. C. F. Codella; Q. B. Nguyen; S. Pankanti; D. A. Gutman; B. Helba; A. C. Halpern; July-September 2017, Pages:5:1-5:15

According to the technique described in Non-Patent Document 1, by cropping (cutting) and resizing the diseased area, it is possible to reduce the influence of scales and the like reflected in the image, but the aspect ratio of the original image Since information such as size is lost, there is a problem that the accuracy of image identification is lowered.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image identification device, a classifier learning method, an image identification method, and a program capable of improving the accuracy of image identification. .

In order to achieve the above object, a determination device according to one aspect of the present invention includes:
a cropped image generation unit that generates a cropped image by performing cropping processing, which is processing for cropping an image including an identification target region, which is a lesion region, from an input image;
a non-cropped image generation unit that generates a non-cropped image that is not subjected to the crop processing from the input image;
a crop image identifier that identifies the crop image;
a non-cropped image identifier that identifies the non-cropped image;
an identification result determination unit that acquires a final identification result using both the identification result of the cropped image classifier and the identification result of the non-cropped image classifier;
Prepare.

According to the present invention, it is possible to improve the accuracy of image identification.

It is a figure which shows the functional structure of the image identification apparatus which concerns on Embodiment 1 of this invention. 5 is a flowchart of crop image learning processing according to the first embodiment; 5 is a flowchart of cropped image generation processing according to the first embodiment; FIG. 10 is a diagram for explaining cropped image generation processing; 5 is a flowchart of non-cropped image learning processing according to the first embodiment; 4 is a flowchart of non-cropped image generation processing according to the first embodiment; FIG. 10 is a diagram for explaining non-cropped image generation processing; 5 is a flowchart of identification processing according to the first embodiment; FIG. 10 is a diagram for explaining activation states of a cropped image classifier and a non-cropped image classifier when a lesion area is input to the image classifier according to the first embodiment; It is a figure explaining the identification result determination part based on the modification 2 of this invention. FIG. 11 is a flowchart of a general learning process according to Modification 2; FIG. 10 is a flowchart of identification processing according to modification 2;

DESCRIPTION OF THE PREFERRED EMBODIMENTS Image identification devices and the like according to embodiments of the present invention will be described below with reference to the drawings. The same reference numerals are given to the same or corresponding parts in the drawings.

(Embodiment 1)
The image identification apparatus 100 according to the first embodiment of the present invention uses a DNN classifier trained with an image obtained by cropping a region to be identified and a DNN classifier trained with an uncropped image to identify an unknown image. identify. The image identification device 100 can improve the accuracy of image identification by using both the outputs of the classifiers based on these two types of DNNs. Such an image identification device 100 will be described below.

The image identification device 100 according to the first embodiment includes a control unit 10, a storage unit 20, an image input unit 31, an output unit 32, a communication unit 33, and an operation input unit 34, as shown in FIG.

The control unit 10 is composed of a CPU (Central Processing Unit) or the like, and by executing a program stored in the storage unit 20, each unit (crop image generation unit 11, non-crop image generation unit 12, crop image identification 13, the non-cropped image discriminator 14, and the discrimination result determination unit 15).

The storage unit 20 includes a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and stores programs executed by the CPU of the control unit 10 and necessary data.

The image input unit 31 is a device for inputting image data for learning or (unknown) image data to be identified. The control unit 10 acquires image data via the image input unit 31 . Any device can be used as the image input unit 31 as long as the control unit 10 can acquire image data. For example, when image data is stored in the storage unit 20 and the control unit 10 acquires the image data by reading the storage unit 20 , the storage unit 20 also serves as the image input unit 31 . Also, when the control unit 10 acquires image data from an external server or the like via the communication unit 33 , the communication unit 33 also serves as the image input unit 31 .

The output unit 32 is a device for the control unit 10 to output the result of identifying the image input from the image input unit 31 and the like. For example, the output unit 32 is a liquid crystal display or an organic EL (Electro-Luminescence) display. In this case, the output section 32 functions as a display section. However, the image identification device 100 may include such a display (display unit) as the output unit 32, or may include the output unit 32 as an interface for connecting an external display. When the image identification device 100 includes the output unit 32 as an interface, the identification result and the like are displayed on an external display connected via the output unit 32 . The output unit 32 functions as output means.

The communication unit 33 is a device (network interface, etc.) for transmitting and receiving data to and from another external device (eg, a server storing a database of image data). The control section 10 can acquire image data via the communication section 33 .

The operation input unit 34 is a device that receives a user's operation input to the image identification device 100, and is, for example, a keyboard, mouse, touch panel, or the like. The image identification device 100 receives an instruction or the like from the user via the operation input unit 34 . The operation input unit 34 functions as operation input means.

Next, functions of the control unit 10 will be described. The control unit 10 implements functions of a cropped image generation unit 11 , a non-cropped image generation unit 12 , a cropped image classifier 13 , a non-cropped image classifier 14 and a classification result determination unit 15 .

The cropped image generation unit 11 acquires an identification target area from an input image input via the image input unit 31, cuts (crops) a square image obtained by adding a margin area to the acquired identification target area, and divides it into a square image. An image (cropped image) resized to the input size of the cropped image classifier 13 is generated. A process in which the cropped image generation unit 11 cuts out an image including an identification target area from an input image is called cropping process.

The non-cropped image generation unit 12 generates an edge-enhanced image (non-cropped image) by performing HPF (High Pass Filter) processing on the input image input via the image input unit 31 . Note that the non-cropped image generation unit 12 may generate an image in which edges are emphasized by processing other than the HPF processing.

The cropped image classifier 13 and the non-cropped image classifier 14 are both image classifiers based on a convolutional neural network (CNN), which is a type of DNN. The control unit 10 functions as a cropped image classifier 13 and also as a non-cropped image classifier 14 by executing a program that implements a CNN-based classifier.

The cropped image identifier 13 has an input layer to which the cropped image generated by the cropped image generator 11 is input, an output layer to which the identification result of the input cropped image is output, and layers other than the input layer and the output layer. and an intermediate layer for outputting the result of identifying the cropped image from the output layer.

The non-cropped image identifier 14 has an input layer to which the non-cropped image generated by the non-cropped image generator 12 is input, an output layer to which the identification result of the input non-cropped image is output, an input layer and an output layer. and an intermediate layer, which is a layer other than the intermediate layer, and outputs the result of identifying the non-cropped image from the output layer.

The identification result determination unit 15 uses both the output from the cropped image identifier 13 (cropped image identification result) and the output from the non-cropped image identifier 14 (non-cropped image identification result) to determine the final obtain a good identification result. Basically, the identification result determination unit 15 simply adds and averages the output from the cropped image identifier 13 and the output from the non-cropped image identifier 14 to obtain the final identification result. However, it is not limited to this. When obtaining the final identification result, the identification result determination unit 15 multiplies the output from the cropped image classifier 13 and the output from the non-cropped image classifier 14 by predetermined weights, and obtains a weighted average may be taken. The identification result determination unit 15 functions as identification result determination means.

The functional configuration of the image identification device 100 has been described above. Next, processing for learning a CNN (cropped image classifier 13 and non-cropped image classifier 14) using cropped images and non-cropped images will be described.

First, crop image learning processing for learning the crop image classifier 13 using crop images will be described with reference to FIG. This processing is executed when the cropped image classifier 13 is trained. Also, before executing this process, it is necessary to prepare learning image data with correct labels. The correct label indicates what the image with the correct label is. must be attached to each learning image data as As another example, if you want to prepare a CNN that outputs the name of a flower when an image of a flower is input, it is necessary to attach the "name of the flower" as a correct label to each learning image data.

First, the control unit 10 acquires learning image data via the image input unit 31 (step S101). Then, the cropped image generation unit 11 performs cropped image generation processing, which will be described later, to generate a cropped image from the learning image data acquired in step S101 (step S102). Step S102 is also called a crop image generation step.

Next, the control unit 10 inputs the cropped image generated by the cropped image generation unit 11 to the cropped image classifier 13, and learns the cropped image classifier 13 based on the correct label attached to the learning image data. (step S103). Step S103 is also called a crop image learning step.

Then, the control unit 10 determines whether or not to end learning (step S104). For example, learning is completed after learning a predetermined number of image data for learning (for example, M). If the learning is not to be ended (step S104; No), such as when there remains image data for learning that has not been learned, the process returns to step S101. If learning is finished (step S104; Yes), the crop image learning process is finished.

Next, the cropped image generation processing performed in step S102 will be described with reference to the flowchart of FIG. 3 and the specific example of FIG. This process is a process of generating a cropped image from a given input image.

First, the cropped image generation unit 11 acquires a region to be identified from a given input image (step S111). As an example of a given input image, the upper left diagram of FIG. 4 shows an input image 1010 in which an identification target 1011 (for example, a disease image) and scales 1012 are shown. In the upper right diagram of FIG. 4, an identification target area 1020 acquired by the cropped image generation unit 11 is shown. Any method can be used to acquire the region to be identified, but for example, the cropped image generation unit 11 automatically extracts (and acquires) the region 1020 using a discrimination target region determiner obtained in advance by machine learning. Such an identification target area determiner prepares, for example, a large amount of image data and a foreground map corresponding to the image (correct data of the identification target area created by a human) as teacher data for learning. , can be created by inputting this teacher data to the CNN for learning.

Acquisition of a region to be identified is not limited to the machine learning method as described above. The user may specify a region 1020 to be identified in the input image 1010 via the operation input unit 34, and the cropped image generation unit 11 may acquire the region 1020 specified by the user. Alternatively, the region 1020 automatically extracted by machine learning or the like may be made modifiable by the user via the operation input unit 34, and the cropped image generation unit 11 may acquire the identification target region corrected by the user. .

Next, the cropped image generation unit 11 extracts a rectangular area that circumscribes the identification target area acquired in step S111 (step S112). The middle left diagram of FIG. 4 shows how a rectangular area 1030 circumscribing the identification target 1011 is extracted. This rectangular area 1030 need not be square.

Then, the cropped image generator 11 crops (cuts out) a square area obtained by adding a margin area to the rectangular area extracted in step S112 (step S113). If the rectangular area 1030 described above is rectangular, the area to be cropped is made square by adjusting the vertical and horizontal amounts of the margins. The middle right diagram of FIG. 4 shows a square area 1040 obtained by adding a hatched margin area 1041 to the rectangular area 1030, and this square area 1040 is to be cropped.

The length of the margin 1042 that defines the size of the margin area 1041 may be changed according to the size of the rectangular area 1030, or may be set to a constant length. When changing according to the size of the rectangular area 1030, for example, "10% of the length of one side of the rectangular area 1030" may be set. When the rectangular area 1030 is a rectangular area having long sides and short sides, for example, "the long side adds a margin of 10% of the length of the long side of the rectangular area 1030 to both sides, and the short side is added to both sides so as to match the length of the long side with the added margin."

Then, the cropped image generator 11 resizes the square area cropped in step S113 according to the input size of the CNN (cropped image classifier 13) (step S114), and ends the process. The image of this resized square area is the clipped image. The lower left diagram of FIG. 4 shows a resized square area 1050, which is a resized and enlarged square area 1040 cut out in step S113. Since the identification object 1011 included in the square area 1050 is also resized at the same magnification as the square area 1050, it is in an enlarged state.

Through the cropped image generation processing described above, a cropped image in which the identification target included in the input image is a close-up is generated. Then, using the generated crop image, the crop image classifier 13 is learned by the above-described crop image learning process.

Next, non-cropped image learning processing for learning the non-cropped image classifier 14 using non-cropped images will be described with reference to FIG. This processing is executed when the non-cropped image classifier 14 is trained. As in the cropped image learning process described above, it is necessary to prepare learning image data labeled with correct answers before executing the non-cropped image learning process. The same data as the learning data used in .

First, the control unit 10 acquires learning image data via the image input unit 31 (step S201). Then, the non-cropped image generation unit 12 performs non-cropped image generation processing, which will be described later, to generate a non-cropped image from the learning image data acquired in step S201 (step S202). Step S202 is also called a non-cropped image generation step.

Next, the control unit 10 inputs the non-cropped image generated by the non-cropped image generating unit 12 to the non-cropped image classifier 14, and identifies the non-cropped image based on the correct label attached to the learning image data. learning the device 14 (step S203). Step S203 is also called a non-crop image learning step.

Then, the control unit 10 determines whether or not to end learning (step S204). For example, learning is completed after learning a predetermined number of image data for learning (for example, M). If the learning is not to be terminated (step S204; No), such as when there is image data for learning that has not been learned, the process returns to step S201. If learning is finished (step S204; Yes), the non-crop image learning process is finished.

Next, the non-cropped image generation processing performed in step S202 will be described with reference to the flowchart of FIG. 6 and the specific example of FIG. This process is a process of generating a non-cropped image from a given input image.

First, the non-cropped image generation unit 12 performs HPF processing (edge enhancement processing) on a given input image (step S211). As an example of a given input image, the upper diagram in FIG. 7 shows an input image 1010 in which an identification target 1011 (for example, a disease image) and scales 1012 are shown. The middle diagram of FIG. 7 shows that the input image 1010 has undergone HPF processing, resulting in an image 1060 including an edge-enhanced identification target 1061 and an edge-enhanced scale 1062. there is

Next, the non-cropped image generating unit 12 cuts both sides of the HPF-processed input image (this is referred to as side-cut processing), cuts out a square area (step S212), and ends the process. In the bottom view of FIG. 7, a square area 1070 is shown, which is cut by dashed line 1063 on both sides of image 1060 in the middle view of FIG.

Through the non-cropped image generation processing described above, a non-cropped image in which the edges of the input image are emphasized is generated. Then, using the generated non-cropped image, the non-cropped image classifier 14 is learned by the above-described non-cropped image learning process.

By having the cropped image classifier 13 and the non-cropped image classifier 14 learn as described above, the image identifying device 100 can identify an unknown input image. Next, identification processing for identifying an unknown input image will be described with reference to FIG. This process is performed in identifying unknown images.

First, the control unit 10 acquires an unknown image to be identified by the image identification device 100 via the image input unit 31 (step S301).

Next, the cropped image generation unit 11 generates a cropped image from the unknown image acquired in step S301 by the above-described cropped image generation processing (FIG. 3) (step S302). Then, the control unit 10 inputs the generated cropped image to the cropped image classifier 13 and acquires the output value of the cropped image classifier 13 (step S303). Step S303 is also called a cropped image identification step.

Next, the non-cropped image generation unit 12 generates a non-cropped image from the unknown image acquired in step S301 by the above-described non-cropped image generation processing (FIG. 6) (step S304). Then, the control unit 10 inputs the generated non-cropped image to the non-cropped image classifier 14 and acquires the output value of the non-cropped image classifier 14 (step S305). Step S305 is also called a non-cropped image identification step. Note that the processing from step S302 to step S303 and the processing from step S304 to step S305 may be performed in parallel. It may be performed prior to the processing of S303.

Then, the identification result determining unit 15 adds and averages the output value of the cropped image classifier 13 obtained in step S303 and the output value of the non-cropped image classifier 14 obtained in step S305, and obtains the final identification result. Determine (step S306). Step S306 is also called an identification result determination step.

Then, the control unit 10 outputs the final identification result determined by the identification result determining unit 15 to the output unit 32 (step S307), and ends the process. In step S307, the control unit 10 may output not only the final identification result but also the output of the cropped image classifier 13 and the output of the non-cropped image classifier 14 to the output unit 32.

With the identification processing described above, the image identification apparatus 100 can improve identification accuracy by using both cropped images and non-cropped images. In addition, since the cropped image generation unit 11 cuts out the identification target area by adding a margin area, the risk of losing the end of the identification target area (for example, the edge of the lesion area boundary) can be greatly reduced. . Moreover, since the cropped image generator 11 cuts out the cropped image in a square shape, the aspect ratio of the image is maintained, and the cropped image identifier 13 can use the aspect ratio as information for identification.

As a specific example, as shown in FIG. 9, an input image 1100 including a lesion area 1101 of seborrheic keratosis is input to each of the cropped image classifier 13 and the non-cropped image classifier 14, and CAM (Class Activation) is performed. mapping), it can be confirmed that the cropped image classifier 13 shows an active reaction centering on the central region, and the non-cropped image classifier 14 shows an active reaction over the entire lesion region. This is because, in the case of the cropped image classifier 13, a lesion area is always expected to exist in the central area, whereas in the non-cropped image classifier 14 it is not necessarily expected that a lesion area exists in the central area. Therefore, it is considered that the non-cropped image discriminator 14 must perform processing including lesion area determination.

The activation map 1110 shown in FIG. 9 corresponds to each case (melanoma (MM), seborrheic keratosis (SK), nevus pigmentosum (NCN)) when the input image 1100 is input to the cropped image classifier 13. )) is a diagram showing an activated region by CAM. An activation map 1120 shown in FIG. 9 is a diagram showing an activation region by CAM for each case when the input image 1100 is input to the non-cropped image classifier 14 . In both figures, regions with higher activity are shown darker. FIG. 9 also shows the score (output value) of each case by each discriminator.

As shown in FIG. 9, the score of each case by the crop image classifier 13 is 0.0890 for melanoma (MM), 0.8681 for seborrheic keratosis (SK), and pigmented nevi ( NCN)) is 0.0429, and the scores for each case by the non-cropped image classifier 14 are 0.3145 for melanoma (MM), 0.5713 for seborrheic keratosis (SK), and nevus pigmentosum. (NCN)) is 0.1142.

In the case of a case image with a large lesion area, such as the case shown in FIG. , and the melanoma (MM) score (0.3145) obtained by the non-cropped image classifier 14 is relatively high in FIG. 9 as well. However, in the cropped image classifier 13, since the size of the lesion area is normalized, discrimination is performed with emphasis on the structure inside the lesion area without depending on the size of the lesion area. Therefore, in the example shown in FIG. 9, the cropped image discriminator 13 detects white dots (milia-like tumors) and black dots (comedone-like dilation), which are findings of seborrheic keratosis (SK). As a result of the detection, it is presumed that the score (0.8681) of seborrheic keratosis (SK) increased.

In this way, in the image identification apparatus 100, the cropped image classifier 13 performs classification with an emphasis on the internal structure of the identification target region, and the non-cropped image classifier 14 performs classification based on features captured from the entire input image. Since both of these identification results are used to determine the final identification result, the identification accuracy can be improved.

(Modification 1)
In the first embodiment, the cropped image generation unit 11 automatically extracts the region to be identified by the identification target region determiner that performs machine learning using a manually created foreground map. However, as a method for automatically extracting a region to be identified, there is also a method using an activation map as shown in FIG. This modified example 1 will be described.

The functional configuration of the image identification device 100 according to Modification 1 is the same as that of Embodiment 1, as shown in FIG. However, in Modification 1, before performing the cropped image learning process, the control unit 10 first performs the non-cropped image learning process (FIG. 5) so that the non-cropped image classifier 14 has been trained. Then, in step S111 of the cropped image generation process (FIG. 3), the control unit 10 first inputs the input image to the non-cropped image classifier 14 and acquires the activation map of the non-cropped image classifier 14 . Then, in the activation map, regions that are activated to a predetermined reference value or more are extracted as identification target regions.

There are several methods for generating the activation map. In the case of the activation map by CAM as shown in FIG. 9 and the activation map by Gradient-weighted Class Activation Mapping (Grad-CAM), identification Since the activation map for each class is obtained, the activation map obtained by adding all the activation maps obtained for each identification class and averaging them (herein called the "total activation map") is obtained. is used to extract the identification target region. For example, the average value of all the elements of the general activation map is obtained, and the portion of the general activation map whose value is equal to or greater than the average value is defined as the identification target area.

In generating the activation map, instead of using CAM or Grad-CAM, simply average each element (feature map) of the intermediate layer immediately before the fully connected layer immediately before the CNN output layer in the channel direction. There is also an activation map generation method that uses an activation map. When using such an activation map, this activation map can be treated in the same manner as the general activation map described above. For example, the average value of all the elements of this activation map is obtained, and the portion of the activation map whose value is equal to or greater than the average value is defined as the identification target area.

The image identification device 100 according to Modification 1 is the same as the image identification device 100 according to Embodiment 1, except that in step S111 of the cropped image generation process (FIG. 3), an activation map is used to extract a region to be identified. are the same. The image identification device 100 according to Modification 1 has the advantages of the image identification device 100 according to Embodiment 1. In addition, the image identification device 100 does not need to separately learn the determination device for extracting the identification target region (the non-cropped image identification device 14, the non-cropped image discriminator 14 can also be used as a discriminator for extracting a discrimination target region.

(Modification 2)
In the first embodiment, the identification result determination unit 15 obtains the final identification result by averaging (additional average or weighted average) the output from the cropped image classifier 13 and the output from the non-cropped image classifier 14. I was getting However, it is not limited to this. The identification result determination unit 15 connects the output layer of the cropped image classifier 13 and the output layer of the non-cropped image classifier 14, inputs them to the fully connected layer, and generates a new output layer via the fully connected layer. may obtain the final identification result. Such a modification 2 will be described.

The functional configuration of the image identification device 100 according to Modification 2 is the same as that of Embodiment 1, as shown in FIG. As shown in FIG. 10, the identification result determination unit 15 according to Modification 2 connects the output layer 1301 of the cropped image classifier 13 and the output layer 1401 of the non-cropped image classifier 14 as a fully connected layer. , to a new output layer 1501 to obtain the final identification result.

In modification 2, it is necessary to perform neural network learning for the output layer 1501 of the identification result determination unit 15 as well. The overall learning process for this purpose will be described with reference to FIG. If this process is performed after the cropped image learning process (FIG. 2) and the non-cropped image learning process (FIG. 5) are completed, the learning time can be shortened. However, it is also possible to perform the overall learning process before performing the cropped image learning process (FIG. 2) or the non-cropped image learning process (FIG. 5). Since the learning of the discriminator 13 and the non-cropped image discriminator 14 is also completed, there is no need to separately perform the cropped image learning process (FIG. 2) and the non-cropped image learning process (FIG. 5).

First, the control unit 10 acquires learning image data via the image input unit 31 (step S401). Then, the cropped image generation unit 11 performs cropped image generation processing (FIG. 3) to generate a cropped image from the learning image data acquired in step S401 (step S402). The non-cropped image generation unit 12 also performs non-cropped image generation processing (FIG. 6) to generate a non-cropped image from the learning image data acquired in step S401 (step S403). Steps S402 and S403 may be performed in parallel, or the process of step S403 may be performed prior to the process of step S402, contrary to FIG.

Then, the control unit 10 converts the cropped image (square area 1050) generated by the cropped image generation unit 11 into a cropped image in the CNN connecting the cropped image classifier 13 and the non-cropped image classifier 14 as shown in FIG. The non-cropped image (square area 1070) generated by the non-cropped image generation unit 12 is input to the classifier 13 and input to the non-cropped image classifier 14. Based on the correct label attached to the learning image data, The weight of the fully-connected connection between the fully-connected layer connecting the output layer 1301 and the output layer 1401 and the output layer 1501 of the identification result determination unit 15 is learned (step S404).

Then, the control unit 10 determines whether or not to end learning (step S405). For example, learning is completed after learning a predetermined number of image data for learning (for example, M). If the learning is not to be terminated (step S405; No), such as when there is still image data for learning that has not been learned, the process returns to step S401. If learning is to be ended (step S405; Yes), the overall learning process is ended.

By making the neural network for the output layer 1501 of the identification result determination unit 15 learn as described above, the image identification device 100 according to the modification 2 can identify an unknown input image. Next, identification processing for identifying an unknown input image by the image identification device 100 according to Modification 2 will be described with reference to FIG. This process is performed in identifying unknown images.

First, the control unit 10 acquires an unknown image to be identified by the image identification device 100 via the image input unit 31 (step S501).

Then, the cropped image generation unit 11 performs cropped image generation processing (FIG. 3) to generate a cropped image from the unknown image acquired in step S501 (step S502). The non-cropped image generation unit 12 also performs non-cropped image generation processing (FIG. 6) to generate a non-cropped image from the unknown image acquired in step S501 (step S503). Note that steps S502 and S503 may be performed in parallel, or the process of step S503 may be performed prior to the process of step S502, contrary to FIG.

Then, the identification result determining unit 15 assigns the cropped image generated by the cropped image generating unit 11 to the cropped image classifier 13 in the CNN as shown in FIG. , and the non-cropped image generated by the non-cropped image generator 12 is input to the non-cropped image classifier 14 . Then, the final identification result is determined by the output value output to the output layer 1501 (step S504).

Then, the control unit 10 outputs the final identification result determined by the identification result determination unit 15 to the output unit 32 (step S505), and ends the process. Note that in step S505, the control unit 10 outputs not only the final identification result, but also the output of the cropped image classifier 13 (the value of each element in the output layer 1301 in FIG. 10) and the output of the non-cropped image classifier 14 ( 10) may also be output to the output unit 32.

With the identification processing described above, the image identification apparatus 100 according to Modification 2 can improve the identification accuracy by using both the cropped image and the non-cropped image.

In the above-described embodiment and modification, the control unit 10 also functions as the cropped image classifier 13 and the non-cropped image classifier 14 by executing a program that implements a CNN-based classifier. However, it is not limited to this. The image identification device 100 realizes the functions of a cropped image classifier 13 and a non-cropped image classifier 14 (for example, a GPU (Graphics Processing Unit), a dedicated IC (Integrated Circuit), etc.) separately from the control unit 10. device.

In addition, although the first embodiment has been described mainly taking skin diseases as an example, the present invention is not limited to the field of dermatology, and can be widely applied in the general field of image identification. For example, it can be applied to identification of flowers, identification of microscopic photographs of bacteria, and the like.

Also, the above-described embodiments and modifications can be combined as appropriate. For example, by combining Modification 1 and Modification 2, an activation map automatically extracts a region to be identified, and as shown in FIG. may be configured to obtain the final identification result from the output layer 1501 by connecting .

Further, in the above-described embodiment and modified example, the non-cropped image generation unit 12 generates an image in which the edges of the input image are emphasized, but the present invention is not limited to this. It is important that the non-cropped image generator 12 does not crop the input image, and the input image may be used as it is as the non-cropped image.

Each function of the image identification device 100 can also be implemented by a computer such as a normal PC (Personal Computer). Specifically, in the above embodiment, the program for image identification processing performed by the image identification device 100 is pre-stored in the ROM of the storage unit 20 . However, the program may be stored in a computer-readable storage medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), a MO (Magneto-Optical Disc), a memory card, or a USB (Universal Serial Bus) memory. By storing and distributing the program in a recording medium, and reading and installing the program in the computer, a computer capable of realizing each of the functions described above may be configured.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to such specific embodiments, and the present invention includes the invention described in the claims and their equivalents. be The invention described in the original claims of the present application is appended below.

(Appendix 1)
a cropped image generation unit that generates a cropped image by performing cropping processing, which is processing for cutting out an image including an identification target region from an input image;
a non-cropped image generation unit that generates a non-cropped image that is not subjected to the crop processing from the input image;
a crop image identifier that identifies the crop image;
a non-cropped image identifier that identifies the non-cropped image;
an identification result determination unit that acquires a final identification result using both the identification result of the cropped image classifier and the identification result of the non-cropped image classifier;
An image identification device comprising:

(Appendix 2)
The cropped image generating unit acquires the identification target area from the input image, and generates the cropped image as an image including a predetermined margin area in the identification target area.
The image identification device according to appendix 1.

(Appendix 3)
The cropped image generation unit acquires the identification target area by an identification target area determiner obtained by machine learning.
The image identification device according to appendix 2.

(Appendix 4)
The cropped image generation unit acquires the identification target area by extracting an area activated by an activation map of the non-cropped image classifier.
The image identification device according to appendix 2.

(Appendix 5)
The non-cropped image generation unit generates the non-cropped image as an image obtained by subjecting the input image to edge enhancement processing and sidecut processing.
5. The image identification device according to any one of appendices 1 to 4.

(Appendix 6)
The non-cropped image generation unit uses the input image as the non-cropped image,
5. The image identification device according to any one of appendices 1 to 4.

(Appendix 7)
The identification result determination unit acquires a final identification result by averaging the output value of the cropped image classifier and the output value of the non-cropped image classifier.
7. The image identification device according to any one of appendices 1 to 6.

(Appendix 8)
The identification result determination unit connects the output layer of the cropped image classifier and the output layer of the non-cropped image classifier to form a fully connected layer, and the output layer via the fully connected layer finally to get a good identification result,
7. The image identification device according to any one of appendices 1 to 6.

(Appendix 9)
Furthermore, a display unit for displaying the output value of the cropped image classifier and the output value of the non-cropped image classifier,
9. The image identification device according to any one of appendices 1 to 8.

(Appendix 10)
wherein the identification target area is a lesion area;
10. The image identification device according to any one of appendices 1 to 9.

(Appendix 11)
a cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cutting out an image including an identification target region from an input image;
a non-cropped image generating step of generating a non-cropped image that is not subjected to the crop processing from the input image;
a cropped image learning step of training a cropped image classifier with the cropped image;
a non-cropped image learning step of training a non-cropped image classifier on the non-cropped image;
A discriminator training method that includes

(Appendix 12)
a cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cutting out an image including an identification target region from an input image;
a non-cropped image generating step of generating a non-cropped image that is not subjected to the crop processing from the input image;
a crop image identification step of identifying the crop image;
a non-cropped image identification step of identifying the non-cropped image;
an identification result determination step of obtaining a final identification result using both the identification result of the cropped image identification step and the identification result of the non-cropped image identification step;
Image identification method including.

(Appendix 13)
In the computer of the image identification device,
A cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cutting out an image including an identification target region from an input image;
a non-cropped image generating step of generating a non-cropped image not subjected to the crop processing from the input image;
a cropped image identification step for identifying the cropped image;
a non-cropped image identification step of identifying the non-cropped image; and
an identification result determination step of obtaining a final identification result using both the identification result of the cropped image identification step and the identification result of the non-cropped image identification step;
program to run the

DESCRIPTION OF SYMBOLS 10... Control part 11... Crop image generation part 12... Non-crop image generation part 13... Crop image classifier 14... Non-crop image classifier 15... Identification result determination part 20... Storage part 31... Image Input unit 32 Output unit 33 Communication unit 34 Operation input unit 100 Image identification device 1010, 1100 Input image 1011, 1061 Identification object 1012, 1062 Scale 1020 Area 1030 Rectangular area 1040, 1050, 1070 Square area 1041 Margin area 1042 Margin 1060 Image 1063 Dotted line 1101 Lesion area 1110, 1120 Activation map 1301, 1401, 1501 Output layer

Claims (17)

a cropped image generation unit that generates a cropped image by performing cropping processing, which is processing for cropping an image including an identification target region, which is a lesion region, from an input image;
a non-cropped image generation unit that generates a non-cropped image that is not subjected to the crop processing from the input image;
a crop image identifier that identifies the crop image;
a non-cropped image identifier that identifies the non-cropped image;
an identification result determination unit that acquires a final identification result using both the identification result of the cropped image classifier and the identification result of the non-cropped image classifier;
An image identification device comprising: a cropped image generator that generates a cropped image by performing a cropping process, which is a process of cutting out an image including an identification target area from an input image;
a non-cropped image generation unit that generates a non-cropped image that is not subjected to the crop processing from the input image;
a cropped image identifier that identifies the identification target in the cropped image;
a non-cropped image classifier that identifies the identification target in the non-cropped image;
an identification result determination unit that acquires a final identification result using both the identification result of the cropped image classifier and the identification result of the non-cropped image classifier;
An image identification device comprising: The cropped image generation unit acquires the identification target area from the input image, and generates the cropped image as an image including a predetermined margin area in the identification target area.
3. The image identification device according to claim 1 or 2 . The cropped image generation unit acquires the identification target area by an identification target area determiner obtained by machine learning.
The image identification device according to claim 3 . The cropped image generation unit acquires the region to be identified by extracting the region activated by the activation map of the non-cropped image classifier.
The image identification device according to claim 3 . The non-cropped image generation unit generates the non-cropped image as an image obtained by subjecting the input image to edge enhancement processing and sidecut processing.
The image identification device according to any one of claims 1 to 5 . The non-cropped image generation unit uses the input image as the non-cropped image,
The image identification device according to any one of claims 1 to 5 . The identification result determination unit acquires a final identification result by averaging the output value of the cropped image classifier and the output value of the non-cropped image classifier.
The image identification device according to any one of claims 1 to 7 . The identification result determination unit connects the output layer of the cropped image classifier and the output layer of the non-cropped image classifier to form a fully connected layer, and the output layer via the fully connected layer finally to get a good identification result,
The image identification device according to any one of claims 1 to 7 . Furthermore, a display unit for displaying the output value of the cropped image classifier and the output value of the non-cropped image classifier,
The image identification device according to any one of claims 1 to 9 . The area to be identified is a lesion area,
The image identification device according to claim 2 . A cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cropping an image including an identification target region, which is a lesion region, from an input image;
a non-cropped image generating step of generating a non-cropped image that is not subjected to the crop processing from the input image;
a cropped image learning step of training a cropped image classifier with the cropped image;
a non-cropped image learning step of training a non-cropped image classifier on the non-cropped image;
A discriminator training method that includes a cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cropping an image including an identification target region from an input image;
a non-cropped image generating step of generating a non-cropped image from the input image without performing the crop processing;
a cropped image learning step of training a cropped image classifier with the cropped image to identify the identification target;
a non-cropped image learning step of training a non-cropped image classifier with the non-cropped image to identify the identification target;
A discriminator training method that includes A cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cropping an image including an identification target region, which is a lesion region, from an input image;
a non-cropped image generating step of generating a non-cropped image that is not subjected to the crop processing from the input image;
a crop image identification step of identifying the crop image;
a non-cropped image identification step of identifying the non-cropped image;
an identification result determination step of obtaining a final identification result using both the identification result of the cropped image identification step and the identification result of the non-cropped image identification step;
Image identification method including. a cropped image generation step of generating a cropped image by performing cropping processing, which is processing for cropping an image including an identification target region from an input image;
a non-cropped image generating step of generating a non-cropped image from the input image without performing the crop processing;
a cropped image identification step of identifying the identification target in the cropped image;
a non-cropped image identification step of identifying the identification target in the non-cropped image;
an identification result determination step of obtaining a final identification result using both the identification result of the cropped image identification step and the identification result of the non-cropped image identification step;
Image identification method including. to the computer ,
A cropped image generation step of generating a cropped image obtained by performing cropping processing, which is processing for cropping an image including an identification target region, which is a lesion region, from an input image;
a non-cropped image generating step of generating a non-cropped image not subjected to the crop processing from the input image;
a cropped image identification step for identifying the cropped image;
a non-cropped image identification step of identifying the non-cropped image; and
an identification result determination step of obtaining a final identification result using both the identification result of the cropped image identification step and the identification result of the non-cropped image identification step;
program to run the to the computer,
A cropped image generation step of generating a cropped image obtained by performing cropping processing, which is processing for cutting out an image including an identification target region from an input image;
a non-cropped image generating step of generating a non-cropped image not subjected to the crop processing from the input image;
a cropped image identification step of identifying the identification target in the cropped image;
a non-cropped image identification step of identifying the identification target in the non-cropped image; and
an identification result determination step of obtaining a final identification result using both the identification result of the cropped image identification step and the identification result of the non-cropped image identification step;
program to run the

Images