JP2020194446A - Program, information processing method, and information processor - Google Patents

Abstract

To provide a program or the like capable of reducing a load for creating teacher data used for semantic segmentation.SOLUTION: A computer acquires an image and label information indicating an object in the image. The computer inputs the acquired image to a learned model which is learned to output object information indicating the object in the image when the image is input. Then, the computer generates a heat map indicating an area in the input image used as a basis when the learned model outputs the object information to the input image. The computer associates the label information with each pixel in the area indicated by the heat map in the input image.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to programs, information processing methods and information processing devices.

When inspecting the occurrence of cracks on the surface of structures such as roads, buildings, tunnels, and dams, a neural network is used to extract linear figures such as cracks from the surface image of the structure. (See, for example, Patent Document 1). Further, in recent years, a technique called semantic segmentation has been used in which each pixel in an image is classified into a region of a specific object by using a neural network. In semantic segmentation, for example, using a neural network trained with a dog image and a cat image, each pixel in the image including the dog and the cat is classified into a dog area, a cat area, and other areas, respectively. ..

Japanese Unexamined Patent Publication No. 2018-195001

In semantic segmentation, when training a neural network, an image in which each pixel in the image is manually classified into each region by a human in advance is used as teacher data. Therefore, in the field of machine learning these days, a lot of labor and budget are spent on the work of creating such teacher data.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide a program or the like capable of reducing the burden of creating teacher data used for semantic segmentation.

The program according to one aspect of the present disclosure learns to acquire an image and label information indicating an object in the image, and output object information indicating the object in the image when the image is input. The acquired image is input to the trained model, and a heat map showing a region in the image as a basis when the trained model into which the image is input outputs object information is generated. Then, the computer is made to execute the process of associating the label information with each pixel in the region indicated by the heat map in the image.

In the present disclosure, it is possible to reduce the burden of creating teacher data used for semantic segmentation.

It is a block diagram which shows the configuration example of an information processing apparatus. It is a schematic diagram which shows the structural example of the classification model. It is a schematic diagram for demonstrating the operation of the heat map generation application. It is a schematic diagram for demonstrating the segmentation DNN. It is a flowchart which shows an example of the teacher data generation processing procedure by an information processing apparatus. It is a schematic diagram which shows the screen example. It is a schematic diagram which shows the screen example. It is a flowchart which shows an example of the teacher data generation processing procedure by the information processing apparatus of Embodiment 2.

Below, the program, information processing method, and information processing apparatus of the present disclosure are used to generate teacher data for training the DNN used for semantic segmentation by using a deep neural network (DNN) trained to perform classification. The details will be described with reference to the drawings showing the embodiments applied to the device.

(Embodiment 1)
An information processing device that generates teacher data for training a segmentation DNN used for semantic segmentation by using a classification DNN will be described. In the present embodiment, road markings such as cracks, dents (rutting, potholes, etc.), white lines, yellow lines, etc. on the road surface based on the road surface image obtained by photographing the surface (road surface) of the asphalt-paved road. The classification DNN trained to determine the presence or absence of a defect or the like is used. Further, in the present embodiment, teacher data used for learning the segmentation DNN for classifying each pixel in the road surface image into a cracked region, a dented region, a road marking defect region, and the like is generated.

FIG. 1 is a block diagram showing a configuration example of an information processing device. The information processing device 10 is configured by using a personal computer, a server computer, or the like, and includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and the like, and each of these units. Are interconnected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). By appropriately executing the control program 12P stored in the storage unit 12, the control unit 11 causes the information processing device 10 to perform various information processing, control processing, and the like that should be performed by the information processing device of the present disclosure.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores in advance various data and the like necessary for executing the control program 12P and the control program 12P executed by the control unit 11. Further, the storage unit 12 temporarily stores data or the like generated when the control unit 11 executes the control program 12P. Further, the storage unit 12 stores the classification model 12a, which is a DNN constructed by, for example, a deep learning process. In the classification model 12a, when the image data (road surface image) obtained by photographing the road surface is input, the road surface photographed in the image data has cracks, dents, missing road markings, etc. (object). It is a trained model trained to output information (object information) indicating whether or not it exists. The DNN (learned model) performs a predetermined operation on an input value and outputs the operation result, and data such as a coefficient and a threshold of a function that defines this operation is stored in the storage unit 12 as a DNN (learned model). It is stored as a classification model 12a). Further, the storage unit 12 stores the teacher data DB (database) 12b in which the teacher data used for learning the classification model 12a is stored. The teacher data includes the road surface image obtained by photographing the road surface, information indicating cracks, dents, and lack of road surface signs existing on the road surface in the road surface image, or information indicating that the road surface is in a state other than these (correct answer). The data is a set of labels and label information), and a large number of such teacher data are stored in the teacher data DB 12b. The class classification model 12a uses such teacher data to indicate whether or not there are cracks, dents, missing road markings, etc. on the road surface in the road surface image when the road surface image is input. I have learned to output. Further, the storage unit 12 stores a heat map generation application 12c (heat map generation application), which is an application program that visualizes a part (a part that contributed to the classification result) that the DNN paid attention to when classifying the input data. The teacher data DB 12b may be stored in an external storage device connected to the information processing device 10, or may be stored in a storage device capable of communicating with the information processing device 10 via the network.

The communication unit 13 is an interface for connecting to a network such as the Internet or a LAN (Local Area Network) by wired communication or wireless communication, and transmits / receives information to / from an external device via the network. The input unit 14 includes a mouse, a keyboard, and the like, receives an operation input by the user, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel integrally configured.

The reading unit 16 reads information stored in a portable storage medium 1a including a CD (Compact Disc) -ROM, a DVD (Digital Versatile Disc) -ROM, and a USB (Universal Serial Bus) memory. The programs and data stored in the storage unit 12 may be read from the portable storage medium 1a by the control unit 11 via the reading unit 16 and stored in the storage unit 12, for example. Further, the programs and data stored in the storage unit 12 may be downloaded by the control unit 11 from an external device via the network via the communication unit 13 and stored in the storage unit 12. Further, the program and data may be stored in the semiconductor memory 1b, and the control unit 11 may read the program and data from the semiconductor memory 1b.

FIG. 2 is a schematic diagram showing a configuration example of the classification model 12a. The classification model 12a of the present embodiment is composed of a CNN (Convolution Neural Network) model, for example, as shown in FIG. In addition to the CNN model, the classification model 12a may be composed of an RNN (Recurrent Neural Network) model, an RSTM (Long Short-Term Memory) model, or the like. The classification model 12a shown in FIG. 2 is composed of an input layer, an intermediate layer, and an output layer. The intermediate layer includes a convolution layer, a pooling layer and a fully connected layer. In the classification model 12a of the present embodiment, the road surface image (image data) is input via the input layer. Each pixel in the road surface image is input to each node of the input layer, and the road surface image input via each node of the input layer is input to the intermediate layer. The road surface image input to the intermediate layer is extracted by filtering or the like in the convolutional layer to generate a feature map, which is compressed by the pooling layer to reduce the amount of information. The convolution layer and the pooling layer are repeatedly provided in a plurality of layers, and the feature map generated by the plurality of convolution layers and the pooling layer is input to the fully connected layer. A plurality of fully connected layers (two layers in FIG. 2) are provided, and the output values of the nodes of each layer are calculated using various functions and threshold values based on the input feature map, and the calculated output values are calculated. Is sequentially input to the node of the next layer. The fully connected layer finally gives each node of the output layer its own output value by sequentially inputting the output value of the node of each layer to the node of the subsequent layer. The number of each of the convolution layer, the pooling layer and the fully connected layer is not limited to the example shown in FIG.

In the classification model 12a of the present embodiment, the output layer has four nodes. For example, node 0 outputs the probability that it should be determined that a crack exists on the road surface in the input road surface image, and node 1 Outputs the probability that it should be determined that there is a dent, node 2 outputs the probability that it should be determined that there is a defect in the road marking, and node 3 outputs the probability that it should be determined that there is another state. .. The other states include a state in which there are no cracks, dents, or defects in road markings on the road surface, and a state in which damages other than cracks, dents, and defects in road markings are present on the road surface. The output value of each node of the output layer is, for example, a value of 0 to 1.0, and the total of the probabilities output from each of the four nodes is 1.0 (100%).

The classification model 12a is a teacher including a road surface image and information indicating cracks, dents, and lack of road markings existing on the road surface in the road surface image or information indicating that the road surface is in a state other than these (correct label). Learn using data. In the classification model 12a, when the road surface image included in the teacher data is input, the output value from the output node corresponding to the correct label included in the teacher data approaches 1.0, and the output from other output nodes is output. Learn so that the value approaches 0. In the learning process, the classification model 12a optimizes data such as coefficients and thresholds of various functions that define predetermined operations performed on input values. As a result, when a road surface image is input, information indicating whether the road surface in the road surface image has cracks, dents, or missing road markings, or whether the road surface is in another state is output. The trained trained classification model 12a is obtained. The learning of the classification model 12a may be performed by the information processing device 10 or another learning device, but the learning process is already completed. When the classification model 12a is learned by another learning device, the information processing device 10 may acquire the learned classification model 12a from the learning device via, for example, a network or a portable storage medium 1a. Since the information processing device 10 of the present embodiment stores the teacher data (teacher data DB 12b) in the storage unit 12, when the learned class classification model 12a is acquired from the learning device, the class classification model 12a The teacher data used for learning is also acquired and stored in the teacher data DB 12b of the storage unit 12.

FIG. 3 is a schematic diagram for explaining the operation of the heat map generation application 12c. The heat map generation application 12c provides a heat map showing which part of the input data the DNN calculated the output data based on when the input data is input to the DNN and the output data is output from the DNN. Generate. In the information processing device 10 of the present embodiment, when the heat map generation application 12c inputs a road surface image to the classification model 12a and the classification model 12a outputs the respective output values from each node of the output layer, the heat map generation application 12c outputs the respective output values. The classification model 12a generates a heat map showing which region in the road surface image the output value is calculated based on. Specifically, when a road surface image is input to the class classification model 12a and a value close to 1.0 is output from node 0 of the output layer of the class classification model 12a, the heat map generation application 12c is a class classification model. 12a generates a heat map showing a region in the road surface image as a basis when it is determined that the road surface in the road surface image has cracks (class corresponding to node 0). The heat map not only shows the region that affects the output value from the DNN, but also shows the degree of influence that each pixel has on the output value for each pixel in the region. That is, the heat map has a value indicating the degree (level) of the influence that each pixel has on the output value corresponding to the pixel position of each pixel of the input image. Each value in the heat map indicates the degree of influence that each pixel has on the output value by a predetermined number of numerical values such as a numerical value in five stages. In the heat map shown in FIG. 3, the degree of influence of each region (each pixel) on the output value in the road surface image is shown by different colors (different shades). The DNN has an intermediate layer composed of a plurality of layers, and the heat map generation application 12c can generate a heat map for each layer of the intermediate layer.

The heat map generated by the heat map generation application 12c shows the area of the object of the class classified by DNN (classification model 12a) in the input image. Therefore, the information processing apparatus 10 of the present embodiment uses the heat map generated by the heat map generation application 12c to generate teacher data for training the segmentation DNN (learning model) used for semantic segmentation. The heat map generation application 12c is, for example, Grad-CAM (Gradient weighted Class Activation Mapping), Guided Grad-CAM, CAM, Grad-CAM ++, Guided Grad-CAM ++, Salience Map, Guided Backpropagation, Layer-wise Relevance Propagation, Integrated. Gradients, Smooth Grad, DeepLIFT (Deep Learning Important FeaTures) and the like can be used. The heat map generation application 12c is not limited to the above-mentioned application, and a program that generates a heat map by various calculation methods can be used.

The segmentation DNN will be described. Here, a segmentation DNN, which is a trained model trained to classify each pixel of the input image into an apple region, a banana region, and other regions when an image is input, will be described as an example. To do. FIG. 4 is a schematic diagram for explaining the segmentation DNN, FIG. 4A shows a configuration example of the segmentation DNN, and FIG. 4B shows an example of teacher data used for learning the segmentation DNN. The segmentation DNN is composed of, for example, a SegNet model, an FCN (Fully Convolutional Network) model, a U-Net model, or the like as shown in FIG.

The segmentation DNN shown in FIG. 4 has an input layer, an intermediate layer and an output layer, and the intermediate layer includes a convolution layer and a pooling layer in the first half portion and an upsampling layer (reverse pooling layer) and a convolution layer in the second half portion. .. In the segmentation DNN having such a configuration, the input image input via the input layer is in the intermediate layer, the feature amount of the image is extracted in the convolution layer in the first half, a feature map is generated, and the feature map is compressed in the pooling layer. .. Also in the segmentation DNN, a plurality of convolution layers and pooling layers are provided, and the feature map generated by the plurality of convolution layers and pooling layers is enlarged (pixels interpolated) by the upsampling layer in the latter half and convolved. The image is smoothed at the layer. A plurality of upsampling layers and convolution layers are also provided, and the feature map enlarged by the plurality of upsampling layers and convolution layers is output from the output layer. In the feature map output from the output layer, as shown in FIG. 4, different colors are added to the input image in the apple-classified area, the banana-classified area, and the other areas. It is an image. In the segmentation DNN, the number of layers of the convolution layer and the pooling layer, the number of layers of the upsampling layer and the convolution layer, and the like are not limited to the examples shown in FIG.

In the segmentation DNN having the above-described configuration, the input image is associated with information (class label) indicating an object to be discriminated (here, an apple or a banana) for each pixel in the input image as shown in FIG. 4B. Learning is performed using teacher data including the label image. The segmentation DNN learns to output the label image included in the teacher data when the input image included in the teacher data is input. In the learning process, the segmentation DNN optimizes data such as coefficients and thresholds of various functions that define predetermined operations performed on input values. As a result, when an image is input, a segmentation DNN learned to classify each pixel in the input image into an apple region, a banana region, and other regions (regions of each class) can be obtained. .. Note that the segmentation DNN may be learned using teacher data in which label images are prepared for each class as shown in FIG. 4B, or class labels of a plurality of classes (objects) are associated with one input image. Learning may be performed using the teacher data of the label image.

The information processing apparatus 10 of the present embodiment generates a label image (teacher data) used for learning the segmentation DNN as shown in FIG. 4B by using the heat map generated by the heat map generation application 12c. In the present embodiment, when a road surface image is input, each pixel of the road surface image is classified into a cracked region, a dented region, a road surface marker defect region, and other regions, respectively. The teacher data for learning the above is generated by using the classification model 12a, the teacher data DB 12b, and the heat map generation application 12c.

Hereinafter, the process performed by the information processing apparatus 10 of the present embodiment when generating the teacher data used for learning the segmentation DNN will be described. FIG. 5 is a flowchart showing an example of a procedure for generating teacher data by the information processing apparatus 10, and FIGS. 6 and 7 are schematic views showing a screen example. The following processing is executed by the control unit 11 according to various programs stored in the storage unit 12 of the information processing device 10. In the present embodiment, the following processing is realized by the control unit 11 executing the program, but some processing may be realized by a dedicated hardware circuit.

When the information processing device 10 receives, for example, an execution instruction of the teacher data generation process used for learning the segmentation DNN via the input unit 14, the control unit 11 (acquisition unit) stores the teacher data DB 12b. Read the teacher data (S11). Each teacher data includes a road surface image and label information (object information) indicating cracks, dents, and road marking defects (objects) existing on the road surface in the road surface image. Based on the read teacher data, the control unit 11 generates an image designation screen for receiving the designation of the image (road surface image) used for generating the teacher data used for learning the segmentation DNN, and displays it on the display unit 15 ( S12).

FIG. 6 shows an example of an image designation screen. The image designation screen displays a road surface image included in each of a plurality of read teacher data as a selectable image A2, and one road surface image selected from the selectable image A2. Is displayed as the processed image A1. In the image designation screen, the display area of the selectable image A2 is configured to be scrollable, for example, so that all road surface images (teacher data) read from the teacher data DB 12b can be displayed. Further, the image designation screen has an input field A3 for selecting an application program to be used when generating a heat map from the processed image A1. In the input field A3, a pull-down menu for selecting an arbitrary one from the selectable heat map generation application 12c is set. In the pull-down menu shown in FIG. 6, a heat map generation application 12c is displayed so that Grad-CAM, Guided Grad-CAM, or CAM can be selected. However, the pull-down menu is all applications that can be executed by the information processing device 10. It is configured so that you can select. Further, the image designation screen is for instructing the execution of the process of generating the teacher data used for learning the segmentation DNN by using the application program selected via the input field A3 based on the selected processed image A1. Has a run button.

On the image designation screen, the control unit 11 accepts the designation of any road surface image of the selectable image A2 via the input unit 14 (S13), and when the designation is accepted, processes the designated (selected) road surface image. It is displayed on the image designation screen as image A1. Further, on the image designation screen, when the control unit 11 receives the selection of any of the heat map generation applications 12c via the input unit 14, the name of the selected application is displayed in the input field A3. Further, on the image designation screen, the control unit 11 receives an execution instruction of the teacher data generation process by operating the execution button via the input unit 14. Therefore, the control unit 11 determines whether or not the execution button is operated to accept the execution instruction of the teacher data generation process (S14), and determines that the execution instruction is not accepted (S14: NO). , Return to the process of step S13. When it is determined that the execution instruction of the teacher data generation process has been received (S14: YES), the control unit 11 inputs the selected road surface image to the process image A1 to the class classification model 12a. Then, the control unit 11 (generation unit) outputs based on which region in the processed image A1 when the classification model 12a outputs the label information of the processed image A1 based on the processed image A1 by the heat map generation application 12c. A heat map showing whether the value has been calculated is generated (S15). Specifically, when the classification model 12a outputs each output value from each node of the output layer based on the processed image A1, the control unit 11 uses the processing as the basis for calculating each output value. A heat map showing the degree of influence on the region and the output value in the image A1 is generated.

The heat map generation application 12c generates a heat map for each layer of the intermediate layer of the classification model 12a, and the control unit 11 synthesizes a plurality of generated heat maps to generate a synthetic heat map (S16). ). For example, the ratio for synthesizing each layer of the intermediate layer is set in advance, and the control unit 11 multiplies the value at the same position by the composition ratio of each layer in the heat map of each layer, and then calculates the total value. By calculating, it is combined into one heat map. The composition ratio of each layer is set so that, for example, the layer closer to the output layer has a larger value. When set in this way, the weight for the heat map generated in each layer can be increased as the layer is closer to the output layer.

The control unit 11 generates an edit screen for editing the generated synthetic heat map and displays it on the display unit 15 (S17). FIG. 7 shows an example of an edit screen, and the edit screen displays a processed image A1, a selectable image A2, an input field A3, and an execution button in the same manner as the image designation screen. Further, on the edit screen, the label information (class information) of the processed image A1 and the heat map M1 of each layer generated by the heat map generation application 12c are displayed below the processed image A1. In the example shown in FIG. 7, regarding the processed image A1 in which two label information is set, the classes of the processed image A1 are class 1 which is a defect of the road marking (pedestrian crossing) and a defect of the road marking (roadside zone). It is displayed that it is class 2 and the heat map generated for the 5 layers of Layer 1 to Layer 5 is displayed for each class. The edit screen displays the composition ratio of each layer on the right side of the heat map M1 of each layer, and the composition ratio can be changed via the input unit 14. A check box (radio button) for selecting one of the classes is displayed on the edit screen, and the composition ratio of each layer can be changed for the class selected by the check box. Further, the editing screen displays the synthetic heat map M2 in a large area, and a pencil button B1 and an eraser button B2 for editing the synthetic heat map M2 are displayed above the synthetic heat map M2. The pencil button B1 is a button for instructing the execution of the process of adding the area (pixel) to be classified into the displayed class to the composite heat map M2. The eraser button B2 is a button for instructing the execution of the process of erasing the area (pixel) that should not be classified into the displayed class from the composite heat map M2. Further, the edit screen has an input field B3 for designating a storage destination (folder name and file name) for storing the edited synthetic heat map M2 as teacher data, and an input field for the edited synthetic heat map M2. It has a save button for instructing the execution of the process of storing in the storage destination input to B3. On the edit screen having the above-described configuration, a part of the composite heat map M2 may be enlarged and displayed.

On the edit screen, the control unit 11 determines whether or not the instruction for changing the composition ratio with respect to the heat map M1 of any layer has been received via the input unit 14 (S18). When it is determined that the instruction to change the composition ratio has been accepted (S18: YES), the control unit 11 changes the composition ratio being displayed to the composition ratio instructed to change, updates the edit screen, and performs the process of step S16. Return. Then, the control unit 11 generates a composite heat map from the plurality of heat maps generated in step S15 based on the changed composite ratio of each layer (S16), and generates the composite heat map M2 being displayed. The edit screen is updated by changing to the synthetic heat map (S17). Each time the control unit 11 receives an instruction to change the composition ratio for each layer, the processes of steps S16 to S17 are performed, and a composition heat map synthesized at the composition ratio instructed to be changed is generated and displayed.

When it is determined that the instruction for changing the composition ratio is not accepted (S18: NO), the control unit 11 operates the pencil button B1 or the eraser button B2 via the input unit 14 to operate each pixel in the processed image A1. It is determined whether or not the change instruction for the classification class has been accepted (S19). When adding pixels to the displayed composite heat map M2, that is, when adding pixels not classified into the displayed class to the displayed class, the user operates the pencil button B1 via the input unit 14. After that, the part (pixel) to be added to the composite heat map M2 is operated (coloring operation). Further, when erasing the pixels from the composite heat map M2 being displayed, that is, when excluding the pixels classified into the displayed class from the displayed class, the user operates the eraser button B2 via the input unit 14. After that, the part (pixel) to be erased from the composite heat map M2 is operated (erased operation). As a result, the control unit 11 receives an instruction for adding or deleting pixels (instruction for changing the classification class) to the composite heat map M2 being displayed.

When it is determined that the change instruction of the classification class has been accepted (S19: YES), the control unit 11 adds or deletes the pixel for which the change instruction has been received to the composite heat map M2 being displayed, and the composite heat map M2. Is changed (S20), and the displayed edit screen is updated. When it is determined that the instruction to change the classification class is not accepted (S19: NO), or after the process of step S20, the control unit 11 inputs the storage destination information into the input field B3 via the input unit 14 and saves it. By operating the button, the instruction to save the teacher data is accepted. Therefore, the control unit 11 determines whether or not the save button has been operated on the edit screen to accept the save instruction (S21), and if it determines that the save instruction has not been accepted (S21: NO), step S18. Return to the processing of. When it is determined that the instruction to save the teacher data has been accepted (S21: YES), the control unit 11 generates the teacher data based on the synthetic heat map M2 (S22). Specifically, the control unit 11 (correspondence unit) associates the label information (class label) of the processed image A1 with each pixel in the region indicated by the composite heat map M2 in the processed image A1 to provide teacher data (correspondence unit). Label image) is generated. Thereby, when the road surface image is input, each pixel of the road surface image is used for learning the segmentation DNN for classifying each pixel into a cracked area, a dented area, a road marking defect area, and other areas. Teacher data is generated. Then, the control unit 11 stores the generated teacher data in the storage destination received via the edit screen (S23), and ends the process.

The information processing apparatus 10 of the present embodiment has a classification model 12a that outputs information indicating an object existing in the image when an image is input by the above-described processing, and a teacher used for learning the classification model 12a. Using the data, it is possible to generate teacher data for training the segmentation DNN that classifies each pixel in the image into a region of an object existing in the image when the image is input. That is, it is possible to generate the teacher data used for learning (generating) the segmentation DNN by using the class classification model 12a and the teacher data (teacher data DB 12b) necessary for generating the class classification model 12a. In addition, as described above, by automatically generating the teacher data used for learning the segmentation DNN using the heat map generated by the heat map generation application 12c, the teacher data previously created manually ( The burden of creating a label image) can be reduced. In addition, when teacher data (label image) is manually created, there is a risk of variation among workers, but when teacher data is automatically created by the above-mentioned processing, each object is objectively classified. It is possible to create teacher data.

In recent years, semantic segmentation is an important technology in various fields including the field of autonomous driving and the medical field, and research and development are being carried out in various industries and companies. At that time, by automatically creating the teacher data used for the learning of the segmentation DNN, it is possible to reduce the budget and labor required to create the teacher data, which will contribute to the further development of the semantic segmentation technology. You can expect it.

In the present embodiment, the synthetic heat map automatically generated from the heat map generated by the heat map generation application 12c can be edited (changed) via the edit screen. Therefore, if an error is included in the automatically generated synthetic heat map, it can be corrected, so that highly accurate teacher data can be generated. Further, on the edit screen shown in FIG. 7, it is possible to reselect the processed image A1. Therefore, the teacher data (label image) based on the plurality of processed images A1 can be sequentially created on the edit screen once displayed. Further, in the present embodiment, the number of heat maps used for creating the composite heat map may be changed (designated), for example, via an edit screen. That is, the heat map generated by the heat map generation application 12c may be configured so that the heat map used for generating the synthetic heat map can be changed among the heat maps generated for each layer of the intermediate layer of the classification model 12a. Specifically, the edit screen shown in FIG. 7 shows an example in which a composite heat map is generated from five heat maps, but for example, only heat maps of layers close to the output layer (for example, three layers of Layer 3 to Layer 5) are displayed. May be used to generate a synthetic heatmap.

Further, in the generated heat map of each layer, the combined heat map may be generated by using only the heat map in which the maximum value (maximum level) of each value in the heat map is equal to or more than a predetermined threshold value. That is, a heat map whose maximum value in the heat map is less than a predetermined threshold value may not be used for generating the composite heat map. The larger each value of the heat map is, the higher the contribution to the classification result is. Therefore, by generating the composite heat map using only the heat map showing the high contribution, the accuracy is higher. Can generate teacher data. The threshold value as a criterion for determining whether or not to use the synthetic heat map may be set so as to be set via the input unit 14 on the editing screen, for example. In this case, the edit screen may display only the heat map determined to be used for generating the synthetic heat map based on the set threshold value. Also, when switching the heat map used for the composite heat map according to the maximum value in the heat map, the heat map used for generating the composite heat map is automatically selected from the heat map generated by the heat map generation application 12c. The information processing device 10 may be configured.

(Embodiment 2)
The information processing apparatus 10 that generates a composite heat map after performing noise removal processing on the heat map generated by the heat map generation application 12c will be described. Since the information processing device of the present embodiment has the same configuration as the information processing device 10 of the first embodiment, the description of the configuration will be omitted.

FIG. 8 is a flowchart showing an example of a teacher data generation processing procedure by the information processing apparatus 10 of the second embodiment. The process shown in FIG. 8 is the process shown in FIG. 5 in which the process of step S31 is added between steps S15 and S16. The description of the same steps as in FIG. 5 will be omitted. Further, in FIG. 8, the illustration of steps S17 to S23 in FIG. 5 is omitted.

In the information processing apparatus 10 of the present embodiment, the control unit 11 performs the processes of steps S11 to S15 in the same manner as in the first embodiment. As a result, when the processed image A1 designated via the image designation screen is input to the classification model 12a and the label information of the processed image A1 is output from the classification model 12a, the classification model 12a outputs an output value. The heat map generation application 12c generates a heat map showing the region in the processed image A1 as the basis for the calculation.

In the information processing apparatus 10 of the present embodiment, the control unit 11 performs a process of removing noise from each of the generated heat maps (S31). For example, the control unit 11 erases (removes) pixels having a value less than a predetermined threshold value from the heat map in each heat map. By erasing the parts (pixels) having a high possibility of noise in the heat map in this way, it is possible to generate a heat map showing only the region having a high degree of contribution to the classification result by the classification model 12a. The control unit 11 generates a composite heat map based on each heat map from which noise has been removed (S16), and performs the processes after step S17. Therefore, in the present embodiment, a more accurate synthetic heat map (teacher data) can be generated by synthesizing a heat map showing only a region having a high degree of contribution to the classification result.

Also in this embodiment, the same effect as that of the first embodiment can be obtained. Further, in the present embodiment, since the synthetic heat map is generated after the noise contained in the heat map generated by the heat map generation application 12c is removed, a more accurate synthetic heat map (teacher data) is generated. Can be done. In the heat map, the predetermined threshold value, which is a criterion for determining whether or not the noise is removed, may be changed to a value specified via, for example, the input unit 14. Further, for example, the predetermined threshold value may be automatically changed according to the maximum value or the average value in the heat map.

In each of the above-described embodiments, the segmentation DNN can be trained by using the teacher data for learning the segmentation DNN generated based on the classification model 12a and the teacher data DB 12b. As a result, when an image is input, it is possible to realize a segmentation DNN that can accurately classify each pixel in the input image into a region (area of each class) of each object to be discriminated. In the present embodiment, a configuration for creating teacher data for learning segmentation DNN, which classifies each pixel in a road surface image into a cracked area, a dented area, a road marking defect area, and other areas, has been described. However, the present disclosure is also applicable to segmentation DNNs that make other classifications. For example, from a surface image of a structure such as a building, a tunnel, or a dam, it is possible to generate teacher data for learning segmentation DNN that classifies each pixel in the surface image into each of a defective region such as a crack. It can also be applied to segmentation DNNs used in various fields such as the field of autonomous driving and the medical field, and can be used to generate teacher data for learning various segmentation DNNs. Further, it can be applied not only to a segmentation DNN for discriminating an object or a state existing in an image, but also to a segmentation DNN for discriminating a photogenic region or an instagram-worthy region in an image. That is, by using the classification model that determines the presence / absence of a photogenic region or the presence / absence of an instagramable region with respect to the input image and the teacher data used for learning this classification model, each pixel in the image. It is possible to generate teacher data for learning the segmentation DNN that classifies the image into a photogenic area or an instagramable area.

The embodiments disclosed this time should be considered as exemplary in all respects and not restrictive. The scope of the present disclosure is expressed by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

10 Information processing device 11 Control unit 12 Storage unit 14 Input unit 15 Display unit 12a Class classification model 12b Teacher data DB
12c heat map generation app

Claims (8)

Acquire label information indicating an image and an object in the image, and obtain
The acquired image is input to the trained model trained to output the object information indicating the object in the image when the image is input.
A heat map showing a region in the image as a basis when the trained model to which the image is input outputs object information is generated.
A program that causes a computer to execute a process of associating the label information with each pixel in the region indicated by the heat map in the image. A claim for causing the computer to perform a process of removing noise based on the degree of the base when the object information is output based on the image in which the trained model is input to the generated heat map. Item 1. The program according to item 1. The trained model is a neural network having a plurality of layers.
Generate the heatmap corresponding to each of the plurality of layers,
The program according to claim 1 or 2, wherein the computer is made to execute a process of synthesizing a heat map generated corresponding to each of the plurality of layers. Accepting the change instruction for the generated heat map,
The program according to any one of claims 1 to 3, which causes the computer to execute a process of changing the heat map based on the received change instruction. The acquired image, the label information corresponding to the image, and the heat map according to the degree of the base when the trained model to which the image is input outputs the object information are associated with each other. The program according to any one of claims 1 to 4, which causes the computer to execute the process of displaying on the display unit. When an image is input, each pixel in the image is targeted by using the teacher data including each pixel in the region indicated by the heat map in the image and the label information associated with each pixel. The program according to any one of claims 1 to 5, wherein the computer executes a process of training a learning model that classifies a learning model into a plurality of areas including an object area. The computer
Acquire label information indicating an image and an object in the image, and obtain
The acquired image is input to the trained model trained to output the object information indicating the object in the image when the image is input.
A heat map showing a region in the image as a basis when the trained model to which the image is input outputs object information is generated.
An information processing method for executing a process of associating the label information with each pixel in the region indicated by the heat map in the image. An acquisition unit that acquires label information indicating an image and an object in the image, and
The acquired image is input to the trained model trained to output the object information indicating the object in the image when the image is input, and the trained model in which the image is input is input. A generator that generates a heat map showing the area in the image that is the basis when the model outputs the object information,
An information processing device including an associating unit for associating the label information with each pixel in the region indicated by the heat map in the image.