JP7301467B2 - Image Interpretation Support System and Image Interpretation Support Program - Google Patents

Description

TECHNICAL FIELD The present invention relates to a technique for discriminating the type of article appearing in an image.

2. Description of the Related Art In interpreting remote sensing images, for example, a system as disclosed in Japanese Unexamined Patent Application Publication No. 2002-100002 is used for the purpose of saving labor and eliminating differences in individual skills.
This system outputs the location and type of objects in the input image.
Here, the object is an article whose position or model name the user wants to identify.

As means for realizing such a system, a neural network as described in Non-Patent Document 1 is effective.
Neural networks can improve the accuracy of their output through supervised learning. However, it is necessary to secure a sufficient amount of learning data.
If a sufficient amount of learning data cannot be secured by actual measurement, the method disclosed in Non-Patent Document 2 can compensate for the learning data. This method generates a simulated image by simulation and supplements the learning data.

JP 2010-271845 A

"ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky, Ilya Sutskever, Geoffrey E.; Hinton, Advances in Neural Information Processing Systems 25 (NIPS), 2012, pp. 1106-1114 "Learning from Simulated and Unsupervised Images through Adversarial Training", Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, R. uss Webb, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2107-2116

For the purpose of correcting errors in the output of the neural network or guaranteeing the correctness of the output of the neural network, the user carries out work to confirm the validity of the output of the neural network.
In validation, the user recalls the image of the object based on the identification results output from the neural network. The identification result indicates the type of object in the image.
The appearance of an object differs depending on the observation direction. Therefore, it is necessary to recollect images repeatedly while changing the direction of observation in the brain. This repetition contributes to the high cognitive load of validation.
In this way, since the work load for checking the validity is high, it is difficult to perform the work for checking the validity many times with high accuracy.

SUMMARY OF THE INVENTION An object of the present invention is to reduce the work load of confirming the validity of the output of a neural network in image interpretation.

The image interpretation support system of the present invention is
an estimating unit that uses a neural network to estimate the type of an object appearing in an actual image and the imaging direction of the object;
Estimation for generating an estimated image obtained by imaging an article of the same type as the target from the same direction as the estimated direction by performing an imaging simulation based on the estimated type and the estimated direction an image generator;
a display unit for displaying the actual image and the estimated image on a display.

According to the present invention, an estimated image based on the output of the neural network (estimated type, estimated direction) is displayed. This reduces the work load of confirming the validity of the output of the neural network in image interpretation.

1 is a configuration diagram of an image interpretation support system 100 according to Embodiment 1. FIG. 1 is a configuration diagram of an image interpretation support device 200 according to Embodiment 1. FIG. 4 is a flowchart of an image interpretation support method according to Embodiment 1; 4 is a flowchart of learning processing (S110) according to the first embodiment; 4 is a flowchart of estimation processing (S120) according to the first embodiment; 4 is a flowchart of support processing (S130) according to the first embodiment;

The same or corresponding elements are denoted by the same reference numerals in the embodiments and drawings. Descriptions of elements having the same reference numerals as those described will be omitted or simplified as appropriate. Arrows in the figure mainly indicate the flow of data or the flow of processing.

Embodiment 1.
The image interpretation support system 100 will be described with reference to FIGS. 1 to 6. FIG.

*** Configuration description ***
The configuration of the image interpretation support system 100 will be described based on FIG.
The image interpretation support system 100 is a system that uses a neural network to interpret images, and reduces the user's load in the task of confirming the validity of interpretation results.

The image interpretation support system 100 includes a display 101 and an image interpretation support device 200 .
The image interpretation support device 200 displays a screen on the display 101 for supporting confirmation of the validity of the interpretation result. The displayed screen will be described later.

The configuration of the image interpretation support device 200 will be described based on FIG.
The image interpretation support device 200 is a computer having hardware such as a processor 201 , a memory 202 , an auxiliary storage device 203 , a communication device 204 and an input/output interface 205 . These pieces of hardware are connected to each other via signal lines.

A processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, processor 201 is a CPU, DSP or GPU.
IC is an abbreviation for Integrated Circuit.
CPU is an abbreviation for Central Processing Unit.
DSP is an abbreviation for Digital Signal Processor.
GPU is an abbreviation for Graphics Processing Unit.

Memory 202 is a volatile storage device. Memory 202 is also referred to as main storage or main memory. For example, memory 202 is RAM. The data stored in memory 202 is saved in auxiliary storage device 203 as needed.
RAM is an abbreviation for Random Access Memory.

Auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is ROM, HDD or flash memory. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed.
ROM is an abbreviation for Read Only Memory.
HDD is an abbreviation for Hard Disk Drive.

Communication device 204 is a receiver and transmitter. For example, communication device 204 is a communication chip or NIC.
NIC is an abbreviation for Network Interface Card.

The input/output interface 205 is a port to which an input device and an output device are connected. For example, the input/output interface 205 is a USB terminal, the input device is a keyboard and mouse, and the output device is the display 101 .
USB is an abbreviation for Universal Serial Bus.

The image interpretation support device 200 includes elements such as a learning unit 210 , an estimation unit 220 and a support unit 230 . These elements are implemented in software.
The learning section 210 includes a simulated image generation section 211 and a parameter adjustment section 212 .
The support unit 230 includes an estimated image generation unit 231 and a display unit 232 .

The auxiliary storage device 203 stores an image interpretation support program for causing the computer to function as a learning unit 210 , an estimation unit 220 and a support unit 230 . The image interpretation aid program is loaded into memory 202 and executed by processor 201 .
The auxiliary storage device 203 further stores an OS. At least part of the OS is loaded into memory 202 and executed by processor 201 .
The processor 201 executes the image interpretation support program while executing the OS.
OS is an abbreviation for Operating System.

The input/output data of the image interpretation support program are stored in the storage unit 290 . For example, the storage unit 290 stores an actual image 291, a three-dimensional model file 292, and the like.
Memory 202 functions as storage unit 290 . However, a storage device such as the auxiliary storage device 203 , a register within the processor 201 and a cache memory within the processor 201 may function as the storage unit 290 instead of or together with the memory 202 .

The image interpretation support device 200 may include multiple processors that substitute for the processor 201 . A plurality of processors share the role of processor 201 .

The image reading support program can be recorded (stored) in a non-volatile recording medium such as an optical disc or flash memory in a computer-readable manner.

***Description of operation***
The operation of the image interpretation support device 200 corresponds to the image interpretation support method. Also, the procedure of the image interpretation support method corresponds to the procedure of the image interpretation support program.

Based on FIG. 3, the outline of the image interpretation support method will be described.
In step S110, the learning unit 210 adjusts the internal parameters 121 of the neural network 120 by supervised learning. The neural network 120 is a neural network that is used for estimating information for assisting interpretation with an image as an input. An internal parameter is a numerical parameter that influences the estimation result for the input of the neural network. Specifically, the internal parameters are the biases of the units that make up the neural network and the weights of the connections that make up the neural network.
In step S120, the estimation unit 220 uses the neural network 120 to estimate the type of object appearing in the real image 291 and the imaging direction of the object. A target object is an article shown in the real image 291 .
In step S<b>130 , the support unit 230 displays the actual image 291 and the estimated image 132 based on the estimation result on the display 101 .

Based on FIG. 4, the details of the learning process (S110) will be described.
In step S<b>111 , the simulated image generation unit 211 determines the type of article and generates type information 111 .
The type information 111 is information indicating the determined type. The item type is identified by an identifier.
Specifically, the simulated image generator 211 randomly determines the type of article.

In step S<b>112 , the simulated image generation unit 211 determines the imaging direction of the article and generates direction information 112 .
The imaging direction with respect to the article corresponds to the observation direction of the article.
Direction information 112 is information indicating the determined direction. The direction of imaging is identified by an identifier.
Specifically, the simulated image generator 211 randomly determines the imaging direction.

In step S<b>113 , the simulated image generation unit 211 acquires the three-dimensional model 113 from the three-dimensional model file 292 by searching the three-dimensional model file 292 .
The 3D model file 292 is a file in which the type of article and the 3D model of the article are associated with each other, and is stored in advance in the storage unit 290 . In the three-dimensional model file 292, a plurality of types of information and a plurality of three-dimensional models are associated with each other. A three-dimensional model is data representing the three-dimensional shape of an article. Specifically, a three-dimensional model is a combination of vertex numbers, X coordinates, Y coordinates, and Z coordinates when the surface of an article is represented by a combination of a plurality of polygons, and the numbers of vertices that make up the polygons. The data consists of
The 3D model 113 is a 3D model associated with the same type as the type indicated in the type information 111 .

In step S<b>114 , the simulated image generator 211 performs imaging simulation based on the three-dimensional model 113 and the direction indicated by the direction information 112 . Thereby, a simulated image 114 is generated.
The simulated image 114 corresponds to an image obtained by imaging an article of the same type as that indicated by the type information 111 from the same direction as that indicated by the direction information 112 .
Imaging simulation is the process of generating an image by simulating imaging. Specifically, in the imaging simulation, a method such as a ray tracing method is used to obtain a measurement value of the electromagnetic waves reflected on the surface of the article in the imaging device.

In step S<b>115 , the parameter adjustment unit 212 performs supervised learning on the internal parameters 121 of the neural network 120 .
In supervised learning, the parameter adjustment unit 212 uses the simulated image 114 as input data, and uses the type indicated by the type information 111 and the direction indicated by the direction information 112 as teacher data.
This adjusts the internal parameters 121 of the neural network 120 .

Specifically, the parameter adjustment unit 212 determines that the type estimated by the neural network 120 matches the type indicated by the type information 111 and the direction estimated by the neural network 120 is the direction indicated by the direction information 112. Adjust internal parameters 121 to match.

Based on FIG. 5, the details of the estimation process (S120) will be described.
Neural network 120 is set with internal parameters 121 after adjustment.

In step S<b>121 , the estimation unit 220 uses the neural network 120 to estimate the type of object appearing in the real image 291 and the imaging direction of the object. As a result, a type estimation result 123 and a direction estimation result 124 are obtained.
A real image 291 is an image obtained by imaging an object, and is pre-stored in the storage unit 290 . Alternatively, the actual image 291 is input to the image interpretation support device 200 by the user.
The imaging direction with respect to the object corresponds to the direction in which the object is observed.
The type estimation result 123 is information indicating the estimated type.
The direction estimation result 124 is information indicating the estimated direction.

Based on FIG. 6, the details of the support process (S130) will be described.
In step S<b>131 , the estimated image generation unit 231 acquires the 3D model 131 from the 3D model file 292 by searching the 3D model file 292 .
The 3D model 131 is a 3D model associated with the same type as the type indicated in the type estimation result 123 .

In step S<b>132 , the estimated image generation unit 231 performs imaging simulation based on the three-dimensional model 131 and the direction indicated by the direction estimation result 124 . An estimated image 132 is thereby generated.
The imaging simulation is the same as the processing performed in step S114.
The estimated image 132 corresponds to an image obtained by imaging an article of the same kind as the kind indicated by the kind estimation result 123 from the same direction as the direction indicated by the direction estimation result 124 .

The estimated image 132 is generated based on the output of the neural network 120 (type estimation result 123, direction estimation result 124). Therefore, it can be said that the estimated image 132 is a representation of the output of the neural network 120 in the form of an image.

In step S<b>133 , the display unit 232 displays the actual image 291 and the estimated image 132 on the display 101 .
Specifically, display unit 232 displays a confirmation screen on display 101 . The confirmation screen is a screen in which the actual image 291 and the estimated image 132 are displayed side by side.

*** Effect of Embodiment 1 ***
The image interpretation support device 200 can image the output of the neural network 120 to generate an estimated image 132 and display the actual image 291 and the estimated image 132 side by side on the display 101 .
The user can confirm the validity of the output of the neural network 120 by comparing the actual image 291 and the estimated image 132 . In other words, the user can confirm the validity of the output of the neural network 120 without recalling the image of the object while changing the direction in which the object is observed.

*** Supplement to Embodiment 1 ***
The embodiments are examples of preferred modes and are not intended to limit the technical scope of the present invention. Embodiments may be implemented partially or in combination with other embodiments. The procedures described using flowcharts and the like may be changed as appropriate.

Image interpretation support device 200 may be realized by a plurality of devices. For example, the 3D model file 292 may be stored in an external server device. In this case, the image interpretation support device 200 accesses the three-dimensional model file 292 by communicating with an external server device.
Each element of the image interpretation support device 200 may be implemented by software, hardware, firmware, or a combination thereof.
The “unit”, which is an element of the image interpretation support device 200, may be read as “processing” or “step”.

100 image interpretation support system, 101 display, 111 type information, 112 direction information, 113 three-dimensional model, 114 simulated image, 120 neural network, 121 internal parameter, 123 type estimation result, 124 direction estimation result, 131 three-dimensional model, 132 Estimated image 200 Image interpretation support device 201 Processor 202 Memory 203 Auxiliary storage device 204 Communication device 205 Input/output interface 210 Learning unit 211 Simulated image generation unit 212 Parameter adjustment unit 220 Estimation unit 230 Support Section 231 Estimated Image Generation Section 232 Display Section 290 Storage Section 291 Real Image 292 3D Model File.

Claims (4)

a learning unit that adjusts internal parameters of the neural network by supervised learning;
an estimating unit for estimating the type of the object and the imaging direction for observing the object by using the neural network after the internal parameters have been adjusted;
From the three-dimensional model file in which the type of the article and the three-dimensional model representing the three-dimensional shape of the article are associated with each other, the same type as the type of the object shown in the actual image estimated by the estimation unit is associated. the real image estimated by the estimation unit by performing an imaging simulation based on the acquired three-dimensional model and the imaging direction in which the object is observed estimated by the estimation unit; an estimated image generation unit that generates an estimated image obtained by imaging an article of the same type as the type of the object shown in the image from the same direction as the imaging direction in which the object estimated by the estimation unit is observed;
The actual image and the estimated image are presented to a user who confirms the validity of the type of the object estimated by the estimation unit by matching the actual image and the estimated image and the imaging direction in which the object is observed. a display unit for displaying on the display;
An image interpretation support system. a learning unit that adjusts internal parameters of the neural network by supervised learning;
an estimating unit for estimating the type of the object and the imaging direction for observing the object by using the neural network after the internal parameters have been adjusted;
From the three-dimensional model file in which the type of the article and the three-dimensional model representing the three-dimensional shape of the article are associated with each other, the same type as the type of the object shown in the actual image estimated by the estimation unit is associated. the real image estimated by the estimation unit by performing an imaging simulation based on the acquired three-dimensional model and the imaging direction in which the object is observed estimated by the estimation unit; an estimated image generation unit that generates an estimated image obtained by imaging an article of the same type as the type of the object shown in the image from the same direction as the imaging direction in which the object estimated by the estimation unit is observed;
The actual image and the estimated image are presented to a user who confirms the validity of the type of the object estimated by the estimation unit by matching the actual image and the estimated image and the imaging direction in which the object is observed. A display unit that displays on a display for supporting confirmation of the validity of the interpretation result of the actual image,
with
The learning unit
Determine the type of article and the imaging direction for observing the article,
Obtaining from the three-dimensional model file a three-dimensional model of an article associated with the same type as the determined type, and performing the imaging simulation based on the obtained three-dimensional model and the determined imaging direction, generating a simulated image obtained by imaging an article of the same type as the determined type from the same direction as the determined imaging direction;
An image interpretation support system that performs the supervised learning by using the simulated image as input data and using the determined type and the determined imaging direction as teacher data. a learning unit that adjusts internal parameters of the neural network by supervised learning;
an estimating unit for estimating the type of an object, which is an article in an actual image, and the imaging direction in which the object is observed, using a neural network after the internal parameters have been adjusted;
The 3D model file in which the type of the article and the 3D model representing the shape of the article are associated with each other is associated with the same type as the type of the object appearing in the actual image estimated by the estimating unit. A three-dimensional model is obtained, and an imaging simulation is performed based on the obtained three-dimensional model and the imaging direction in which the object is observed , which is estimated by the estimation unit. an estimated image generation unit configured to generate an estimated image obtained by capturing an article of the same type as the type of the object being observed from the same direction as the imaging direction in which the object estimated by the estimation unit is observed;
The actual image and the estimated image are presented to a user who confirms the validity of the type of the object estimated by the estimation unit by matching the actual image and the estimated image and the imaging direction in which the object is observed. As a display unit that displays on the display,
An image interpretation support program that makes computers work. a learning unit that adjusts internal parameters of the neural network by supervised learning;
an estimating unit for estimating the type of an object, which is an article in an actual image, and the imaging direction in which the object is observed, using a neural network after the internal parameters have been adjusted;
From the three-dimensional model file in which the type of the article and the three-dimensional model representing the three-dimensional shape of the article are associated with each other, the same type as the type of the object shown in the actual image estimated by the estimation unit is associated. the real image estimated by the estimation unit by performing an imaging simulation based on the acquired three-dimensional model and the imaging direction in which the object is observed estimated by the estimation unit; an estimated image generation unit that generates an estimated image obtained by imaging an article of the same type as the type of the object shown in the image from the same direction as the imaging direction in which the object estimated by the estimation unit is observed;
The actual image and the estimated image are presented to a user who confirms the validity of the type of the object estimated by the estimation unit by matching the actual image and the estimated image and the imaging direction in which the object is observed. As a display unit that displays on the display to support the confirmation of the validity of the interpretation result of the actual image,
An image interpretation support program for functioning a computer,
The learning unit
Determine the type of article and the imaging direction for observing the article,
A three-dimensional model associated with the same type as the determined type is obtained from the three-dimensional model file, and the imaging simulation is performed based on the obtained three-dimensional model and the determined imaging direction. generating a simulated image obtained by imaging an article of the same type as the type from the same direction as the determined imaging direction;
An image interpretation support program that performs the supervised learning by using the simulated image as input data and using the determined type and the determined imaging direction as teacher data.

Images