JP6985977B2 - Output device, output method, output program and output system - Google Patents

Description

The present invention relates to an output device, an output method, an output program and an output system.

Information processing that utilizes a model learned using a neural network is being actively performed. For example, there is known a technique for realizing various classification processes such as language recognition and image recognition by using a DNN (Deep Neural Network) having neurons connected in multiple stages.

In addition, a technique is known to generate a composite image used by a neural network to recognize a predetermined target in an image by inputting an image to be processed into the neural network and extracting the intermediate image from the intermediate layer of the neural network. Has been done.

Japanese Patent No. 6214073

However, with the above-mentioned conventional technique, it is difficult to dynamically modify the result output from the model. Specifically, the above-mentioned conventional technique generates information used for recognizing a predetermined object in an image, and it is difficult to apply it to a process of modifying the output result itself of the neural network. be.

The present application has been made in view of the above, and an object of the present invention is to provide an output device, an output method, an output program, and an output system capable of dynamically modifying the result output from the model.

The output device according to the present application is an input unit for inputting a first image to be processed into a model which is a neural network for outputting an image, and an intermediate image which is an image in an intermediate layer of the model in which the first image is input. The second from the output layer of the model based on the intermediate output unit that outputs the image, the reception unit that receives the intervention information that reflects the intervention processing for the intermediate image, and the intervention information received by the reception unit. It is characterized by having a result output unit for outputting an image.

According to one aspect of the embodiment, there is an effect that the result output from the model can be dynamically modified.

FIG. 1 is a diagram showing an example of output processing according to an embodiment. FIG. 2 is a diagram showing an execution example of the output process according to the embodiment. FIG. 3 is a diagram showing a configuration example of an output system according to an embodiment. FIG. 4 is a diagram showing a configuration example of the output device according to the embodiment. FIG. 5 is a diagram showing an example of a model storage unit according to an embodiment. FIG. 6 is a diagram showing an example of an image storage unit according to an embodiment. FIG. 7 is a diagram showing a configuration example of the display control device according to the embodiment. FIG. 8 is a diagram showing an example of an intermediate image storage unit according to an embodiment. FIG. 9 is a diagram showing an example of an intervention information storage unit according to an embodiment. FIG. 10 is a diagram (1) showing an example of the intervention process according to the embodiment. FIG. 11 is a diagram (1) showing the procedure of the intervention process according to the embodiment. FIG. 12 is a diagram (2) showing an example of the intervention process according to the embodiment. FIG. 13 is a diagram (2) showing the procedure of the intervention process according to the embodiment. FIG. 14 is a schematic diagram showing a procedure of output processing according to the embodiment. FIG. 15 is a flowchart showing a processing procedure according to the embodiment. FIG. 16 is a diagram showing a configuration example of an output device according to a modified example. FIG. 17 is a hardware configuration diagram showing an example of a computer that realizes the function of the output device.

Hereinafter, embodiments (hereinafter, referred to as “embodiments”) for implementing the output device, output method, output program, and output system according to the present application will be described in detail with reference to the drawings. Note that this embodiment does not limit the output device, output method, output program, and output system according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate explanations are omitted.

[1. Example of output processing]
First, an example of output processing according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of output processing according to an embodiment. In FIG. 1, the output device 100 according to the embodiment inputs an image to be processed (hereinafter, may be referred to as a “first image” for distinction) into a model trained as a neural network that outputs an image. An example of outputting an image (hereinafter, may be referred to as a "second image" for distinction) from the output layer of the model by the output process according to the embodiment will be described.

The output device 100 shown in FIG. 1 is a server device that executes output processing according to the embodiment. The output device 100 executes the output process according to the embodiment by using the model M01 learned as a neural network that outputs an image.

_{The display control devices 30 1} and 30 ₂ shown in FIG. 1 are server devices that control the image output from the intermediate layer of the model M01 to be displayed on an output device such as a display. When it is not necessary to distinguish between the display control devices 30 ₁ and 30 _2, they are collectively referred to as the display control device 30.

In an embodiment, the output device 100 holds a model M01 learned as a neural network in which a plurality of nodes are connected in multiple stages. For example, the model M01 may be a DNN, an LSTM (Long Short-Term Memory), a convolutional neural network, a recurrent neural network, or the like. Further, the model M01 may be a combination of the functions of these convolutional neural networks and recurrent neural networks.

The model M01 is a model trained to output a second image from an output layer via a plurality of intermediate layers when the first image is input to the input layer. That is, the model M01 is learned to output other output information obtained by converting the input information when a certain input information is input. For example, the model M01 is a model in which the features of a plurality of human faces are learned in advance. In this case, when the output device 100 inputs a face image of a predetermined user's face into the model M01, the output device 100 outputs an image obtained by synthesizing a plurality of human facial features learned at the time of generation of the model M01 as a second image. do.

Generally, in a neural network, an input signal is multiplied by various connection coefficients set in each of a plurality of intermediate layers to calculate an output signal. In this case, although the user can confirm the output result, he / she cannot intervene in the arithmetic processing by the intermediate layer. Therefore, after confirming the output result, he / she may input a different input signal (here, the user's face image). , It is necessary to relearn the model. That is, in the past, it was difficult for the user to finely modify the output result of the neural network or to intervene in the calculation process of the neural network.

Therefore, the output device 100 according to the embodiment reflects the intervention process by the user in the output result by the output process shown below. Specifically, the output device 100 outputs an intermediate image which is an image in the intermediate layer of the model M01 to which the first image is input. Then, the output device 100 receives the intervention information which is the information reflecting the intervention process for the intermediate image. Further, the output device 100 outputs a second image from the output layer of the model M01 based on the received intervention information. In this way, the output device 100 provides a means for the user to intervene in the processing of the intermediate layer, and adds the intervention information generated by the user's intervention to the arithmetic processing, so that the arithmetic performed by the trained model can be performed. Allows the user to make fine adjustments. Hereinafter, the output processing according to the embodiment will be described along the flow with reference to FIG.

First, the user photographs his / her face using an input device such as a camera 50 and creates a face image. Then, the user transmits the captured face image to the output device 100 (step S11). For example, the user may transmit the face image via the camera 50 having a network function for outputting the image, or may transmit the face image via a terminal device or the like connected to the camera 50. The camera 50 may create a face image that has undergone a predetermined trimming process. For example, the camera 50 may transmit a face image to the output device 100 after performing a trimming process for deleting information such as a background while leaving only a range showing the user's face based on the existing face recognition technology. good.

The output device 100 inputs the face image acquired from the user into the model M01 (step S12). Specifically, the output device 100 inputs the information of each pixel forming the face image as an input signal to the input layer of the model M01. In this case, the input layer of the model M01 has a number of pixels corresponding to each pixel constituting the face image and a number of nodes corresponding to the color information of the pixels (for example, 3 channels of RGB).

In the model M01, the node of the input layer is multiplied by a predetermined connection coefficient (weight value), and the node of the first intermediate layer is calculated. The structure of the neural network, such as the value of the connection coefficient to be multiplied by each node of the input layer and the connection coefficient corresponding to which node and which edge, is determined at the learning stage. Since the structure of the neural network is an existing technique, the description thereof will be omitted.

Subsequently, the output device 100 outputs an intermediate image based on the information contained in the first intermediate layer of the model M01 (step S13). Specifically, the output device 100 generates an intermediate image 61 based on the value possessed by the node of the first intermediate layer, and outputs the generated intermediate image 61. The output device 100 transmits the output intermediate image 61 to the display control device 30 _1. The display control device 30 ₁ controls the intermediate image 61 to be displayed on the display 60, which is an example of the output device (step S14).

In the example of FIG. 1, the intermediate image 61 is illustrated so that a portion (eyes, nose, etc.) constituting the user's face can be recognized. In this case, it _{is assumed that the display control device 30 1} generates the intermediate image 61 by reproducing the same arrangement of pixels as the first image based on each node of the first intermediate layer. However, the intermediate image output from each intermediate layer may actually be an image in which the user cannot recognize the target. For example, when the model M01 is a model trained by a convolutional neural network, the intermediate image is an image that reflects the information of a plurality of filters for extracting features used in the convolution. In this case, the intermediate image is represented by a mosaic-like image in which the user's face is separated by the size of the image to which the individual filters are applied (eg, 16 pixels x 16 pixels).

The display 60 is installed at a position that can be visually recognized by the user. That is, the user can visually recognize the intermediate image 61, which is an image of the face image captured by itself processed by the first intermediate layer.

Further, in the example of FIG. 1, the display 60 is a display that employs a touch panel. That is, the user can intervene with the display 60 by a touch operation. For example, the user touches an arbitrary part of the intermediate image 61 displayed on the display 60 with the finger 65 to perform intervention by the touch operation (step S15). The display control device 30 ₁ controls the display so that when the user performs a touch operation, the portion where the touch operation is performed (display 66 shown in FIG. 1) is painted black.

Here, the display control device 30 ₁ acquires the intervention information based on the touch operation of the user (step S16). The intervention information is information that reflects the intervention process for the intermediate image. Specifically, the intervention information is information indicating the pixels of the portion touched by the user in the intermediate image 61. That is, the pixel shown in the intervention information is a pixel corresponding to a portion painted in black on the display 60 by the user. More specifically, the intervention information is represented by, for example, coordinate information pointing to a pixel touched by the user.

The display control device 30 ₁ transmits the acquired intervention information to the output device 100 (step S17). The output device 100 performs an intervention process on the calculation of the model M01 based on the received intervention information (step S18). Specifically, the output device 100 identifies the node corresponding to the intervention information among the nodes of the first intermediate layer. That is, the output device 100 identifies the node corresponding to the pixel to which the user's touch operation is performed on the intermediate image 61. Then, the output device 100 performs a process of masking the specified node. Specifically, the output device 100 performs an operation on the next intermediate layer (second intermediate layer) using only the nodes excluding the specified node.

Through step S18, the output device 100 determines the value of the node of the second intermediate layer of the model M01. The output device 100 outputs an intermediate image 71 corresponding to the second intermediate layer (step S19), and transmits to the display control unit 30 _2. The display control device 30 ₂ displays the intermediate image 71 transmitted from the output device 100 on the display 70 (step S20).

In the example of FIG. 1, the camera 75 is installed in the vicinity of the display 70. The camera 75 is installed so that, for example, a situation in which the user operates the camera 50 or a situation in which the display 60 is touched can be captured. The camera 75, the status of the real-time user continuously captured, transmits the captured image to the display control unit 30 _2. In other words, the user performs an intervention based on the image captured by the camera 75 (step S21).

The display control device 30 ₂ converts the image captured by the camera 75 into binarization information based on a predetermined threshold value. The binarized information has the same size (that is, the same number of pixels) as the intermediate image 71. Then, the display control device 30 ₂ acquires the obtained binarization information as intervention information (step S22). For example, the display control device 30 ₂ acquires the coordinate information corresponding to the pixel indicating the black side among the binarization information as the intervention information.

The display control device 30 ₂ transmits the acquired intervention information to the output device 100 (step S23). The output device 100 performs an intervention process on the calculation of the model M01 based on the received intervention information (step S24). Specifically, the output device 100 identifies the node corresponding to the intervention information among the nodes of the second intermediate layer. For example, the output device 100 superimposes the intermediate image 71 and the binarization information, and identifies a node corresponding to the pixel showing the black side in the binarization information. Then, the output device 100 performs a process of masking the specified node. Specifically, the output device 100 performs an operation on the next intermediate layer (third intermediate layer) using only the nodes excluding the specified node.

The output device 100 repeats the output of the intermediate image and the acquisition of the intervention information as shown above up to the output layer of the model M01. When the output device 100 finishes the intervention process for the last intermediate layer, the output device 100 performs an operation from the last intermediate layer to the output layer and outputs a second image from the output layer (step S25).

The output device 100 controls the display 80 to display the second image 81, which is the output result. The second image 81 is a composite of the facial image input by the user and the features of the plurality of facial images learned at the time of learning the model M01. Further, the second image 81 reflects the intervention process in each intermediate layer. The output device 100 presents the output result to the user by displaying the second image 81 on the display 80 (step S26).

The processes from step S12 to step S26 are continuously executed from the start of the input in step S11 until a predetermined time elapses. That is, the user can confirm the change in the second image 81 by touching the display 60 or changing the figure (shadow) imaged by the camera 75. As a result, the user can interactively intervene in the calculation of the model M01 while checking the result output from the model M01. That is, the user can dynamically modify the output result of the model.

Here, a situation in which the output process by the output device 100 shown in FIG. 1 is actually performed will be described with reference to FIG. FIG. 2 is a diagram showing an execution example of the output process according to the embodiment. As shown in FIG. 2, the user sits at a position where the camera 50 captures his / her face image. Then, the user confirms the second image 81 displayed on the display 80 while touching the intermediate image 61 displayed on the display 60 or changing the appearance captured by the camera 75. As shown in FIG. 2, in the execution example of the output process, even if another display for displaying an intermediate image output from an intermediate layer different from the first intermediate layer and the second intermediate layer is further installed. good.

As described above, the output device 100 according to the embodiment inputs the first image to be processed into the model M01 which is a neural network that outputs an image. Then, the output device 100 outputs an intermediate image 61 or an intermediate image 71 which is an image in the intermediate layer of the model M01 to which the first image is input. Subsequently, the output device 100 receives the intervention information which is the information reflecting the intervention process for the intermediate image 61 and the intermediate image 71. Further, the output device 100 outputs the second image 81 from the output layer of the model M01 based on the received intervention information.

As described above, the output device 100 according to the embodiment has a configuration in which the intervention information generated by the intervention process such as the touch operation on the intermediate image 61 or the like is incorporated into the calculation of the model M01, so that the user can dynamically control the neural network. Allows you to intervene in the process. As a result, the output device 100 can dynamically correct the result output from the model. Further, the output device 100 can provide the user with entertainment such as enjoying the change of the second image by accepting the intervention operation from the user while presenting the second image which is the output result changing in real time. Further, the output device 100 masks the input of the portion touching the intermediate image and reflects the result in the second image, so that what portion of the intermediate image changes and what kind of influence the output result has. It is possible to let the user experience the internal processing of the model that cannot be normally recognized, such as whether to receive it.

In the examples of FIGS. 1 and 2, an example of using a touch operation, image information captured by a camera, or the like as an intervention process from a user is shown, but the intervention process is not limited to this. For example, the intervention process may be performed using a pointing device such as a mouse, or may be performed by voice input or the like.

Hereinafter, the configuration and processing of the output device 100 that performs the output processing as described above and the output system 1 including the output device 100 will be described in more detail.

[2. Output system configuration]
Next, the configuration of the output system 1 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the output system 1 according to the embodiment. As shown in FIG. 3, the output system 1 includes an input device 10, an output device 20, a display control device 30, and an output device 100. Each device included in the output system 1 is communicably connected by wire or wirelessly via a network N (for example, the Internet) which is a communication network. The number of each device included in the output system 1 shown in FIG. 3 is not limited to that shown in the figure. For example, the output system 1 may include a plurality of input devices 10.

The input device 10 is a device (information processing device) used for inputting various information. For example, the input device 10 is a camera 50, a camera 75, or the like shown in FIG. Further, the input device 10 includes a desktop PC (Personal Computer) having a camera function and a microphone function, a notebook PC, a mobile phone such as a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), and the like. You may. The input device 10 transmits the input information to the output device 100, the display control device 30, and the like.

The output device 20 is a device that outputs various information. For example, the output device is the display 60, the display 70, the display 80, or the like shown in FIG. The input device 10 and the output device 20 may be integrated instead of separate devices. For example, when the output device 20 is a display provided with a touch panel, the display is not only the output device 20 but also functions as the input device 10. The output device 20 outputs (displays) information such as an image based on the information transmitted from the output device 100 and the display control device 30.

The display control device 30 is a server device that controls the output device 100 to acquire an intermediate image output from the intermediate layer and display the intermediate image on the output device 20. Further, the display control device 30 acquires intervention information for the intermediate image based on the information input to the input device 10. Specifically, the display control device 30 acquires intervention information based on the user's touch operation, image information obtained by capturing the user, and the like. Then, the display control device 30 transmits the acquired intervention information to the output device 100.

As described above, the output device 100 outputs an intermediate image from the intermediate layer of the model, receives intervention information for the intermediate image, and outputs a second image from the output layer of the model based on the received intervention information. It is a server device that executes processing.

[3. Output device configuration]
Next, the configuration of the output device 100 according to the embodiment will be described with reference to FIG. FIG. 4 is a diagram showing a configuration example of the output device 100 according to the embodiment. As shown in FIG. 4, the output device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The output device 100 includes an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from an administrator or the like who uses the output device 100, and an output unit (for example, a liquid crystal display, etc.) for outputting various information. You may have.

(About communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the input device 10, the output device 20, and the display control device 30 via the network N.

(About the storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The storage unit 120 according to the embodiment includes a model storage unit 121 and an image storage unit 122.

(About model storage unit 121)
The model storage unit 121 stores information about the model held by the output device 100. Here, FIG. 5 shows an example of the model storage unit 121 according to the embodiment. FIG. 5 is a diagram showing an example of the model storage unit 121 according to the embodiment. As shown in FIG. 5, the model storage unit 121 has items such as “model ID”, “input data”, “connection coefficient”, and “output data”.

The "model ID" indicates identification information that identifies the model. In the following description, the identification information as shown in FIG. 5 may be used as a reference code. For example, a model whose identification information is indicated by "M01" may be referred to as "model M01".

"Input data" indicates the format (mode) of the data input to the model. The "connection coefficient" indicates the connection coefficient (weight value) in the model. "Output data" indicates the format of the data output from the model. In the example shown in FIG. 5, "input data", "connection coefficient", and "output data" are shown by a concept such as "A01", but in reality, specific information corresponding to each item is provided. It will be remembered. For example, in the item of "input data", a specific format of data that can be input to the model (for example, the number of pixels of the input image and color information are represented by 3 channels) is stored. The "connection factor" includes the number of intermediate layers in the model, the number of edges connecting the nodes of each intermediate layer, information indicating which node is connected to which node at the edge, and the connection of each edge. The value of the coefficient etc. is stored. In the "output data", a specific format of the data output as an output result (for example, the number of pixels of the output image and the color information are represented by 3 channels) is stored.

That is, in FIG. 5, as an example of the information stored in the model storage unit 121, the model M01 identified by the model ID “M01” has the input data “A01”, the connection coefficient “B01”, and is output. It shows that the data is a model such as "C01".

Although not shown in FIG. 5, the model storage unit 121 may store a data set such as learning data for learning the model.

The model according to the embodiment may be a neural network having one intermediate layer, or may have various structures such as a DNN having a plurality of intermediate layers.

(About the image storage unit 122)
The image storage unit 122 stores image information input to the model. Here, FIG. 6 shows an example of the image storage unit 122 according to the embodiment. FIG. 6 is a diagram showing an example of the image storage unit 122 according to the embodiment. As shown in FIG. 6, the image storage unit 122 has items such as “image ID”, “number of pixels”, and “color information”.

The "image ID" indicates identification information for identifying an image. "Number of pixels" indicates the number of pixels included in the image. "Color information" indicates the color information of each pixel. In the example shown in FIG. 6, "the number of pixels" and "color information" are shown by a concept such as "E01", but in reality, specific information corresponding to each item is stored. For example, in the item of "number of pixels", a specific value of the number of pixels constituting the image is stored. Further, in the "color information", specific information indicating the color of each pixel (for example, a value for three channels of RGB) is stored.

That is, in FIG. 6, as an example of the information stored in the image storage unit 122, the image D01 having the image ID “D01” has the number of pixels “E01” and the color information is “F01”. ing.

(About control unit 130)
Returning to FIG. 4, the explanation will be continued. The control unit 130 is a controller, and is, for example, various programs (as an example of an output program) stored in a storage device inside the output device 100 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. (Equivalent) is realized by executing the RAM as a work area. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 130 converts the first image (more accurately, the input signal corresponding to each pixel constituting the image) input to the input layer of the model by the information processing according to the model stored in the storage unit 120. On the other hand, an operation is performed based on the connection coefficient of the model (that is, the weight value corresponding to the feature learned by the model), and the second image (more accurately, each pixel constituting the image) is supported from the output layer of the model. Output signal) is output.

As shown in FIG. 4, the control unit 130 according to the embodiment includes an acquisition unit 131, an input unit 132, a calculation unit 133, an intermediate output unit 134, a reception unit 135, and a result output unit 136. , Realize or execute the functions and actions of information processing described below. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be any other configuration as long as it is configured to perform information processing described later. Further, the connection relationship of each processing unit included in the control unit 130 is not limited to the connection relationship shown in FIG. 4, and may be another connection relationship.

(About acquisition unit 131)
The acquisition unit 131 acquires various information. For example, the acquisition unit 131 acquires a trained model for executing the output process according to the embodiment. The acquisition unit 131 stores the acquired model in the model storage unit 121.

In addition, the acquisition unit 131 acquires the user's face image. For example, the acquisition unit 131 acquires the user's face image via an input device 10 such as a camera that captures the user. The acquisition unit 131 stores the acquired image in the image storage unit 122. Further, the acquisition unit 131 sends the acquired face image to the input unit 132.

(About input unit 132)
The input unit 132 is trained as a neural network that outputs an image, and inputs a first image to be processed into the generated model. In the embodiment, the input unit 132 inputs a face image obtained by capturing the user's face into the model as the first image.

For example, the input unit 132 has a function as an encoder for encoding the first image, a function of generating a vector by applying a predetermined matrix to the encoded information, and the like. That is, the input unit 132 converts the first image, which is the input data, into a format that can be input to the model learned as the neural network, and inputs the converted information to the model.

(About calculation unit 133)
The calculation unit 133 multiplies the information (that is, each node) input to the model by the input unit 132 by the connection coefficient to calculate the value of the node corresponding to the intermediate layer of the next stage. For example, the calculation unit 133 multiplies the value of each node in the previous stage connected to the node as the value of the node in the middle layer of the next stage by the connection coefficient of the edge between the nodes connected to each other, and each node. Calculate the total value of. Further, the calculation unit 133 inputs the calculated total value into a predetermined activation function, and calculates a value corresponding to the node in the next stage.

(About the intermediate output unit 134)
The intermediate output unit 134 outputs an intermediate image which is an image in the intermediate layer of the model to which the first image is input. Specifically, the intermediate output unit 134 generates an intermediate image based on the value of each node of the intermediate layer, and outputs the generated intermediate image.

The configuration of the intermediate image does not necessarily have to match the final output result (second image). For example, depending on the structure of the neural network, the number of nodes in the intermediate layer and the number of nodes in the output layer may differ. In this case, the image information such as the number of pixels of each of the intermediate image and the second image may be different from each other.

Further, the intermediate image does not have to be in a form that can be recognized as a face by the user who sees the image. For example, if the model has the structure of a convolutional neural network, the intermediate image may be represented by a mosaic-like image separated by the size of the filter for extracting the features used in the convolution. In this case, the intermediate image may show a characteristic portion of the user's face (for example, the vicinity of the eyes and nose constituting the face) as an image in which they are arranged like a mosaic.

Further, the intermediate output unit 134 outputs an intermediate image for each intermediate layer. Specifically, the intermediate output unit 134 outputs a first intermediate image which is an image in the first intermediate layer of the model to which the first image is input. After that, the intermediate output unit 134 is in the middle of the next stage of the model in which the first intervention information is input when the first intervention information reflecting the intervention processing for the first intermediate image is received by the reception unit 135 described later. A second intermediate image, which is an image in the layer, is output. For example, if the model has m layers (m is an arbitrary number), the intermediate output unit 134 may output intermediate images for the first layer to the mth layer.

(About reception desk 135)
The reception unit 135 receives intervention information, which is information that reflects the intervention process for the intermediate image. Specifically, the reception unit 135 receives intervention information indicating the intervention process via the display control device 30 that has received the user's intervention process for the intermediate image.

For example, the reception unit 135 receives intervention information, which is information in which part or all of the information of the intermediate image is omitted by the intervention process for the intermediate image. Specifically, the reception unit 135 receives intervention information indicating that the intermediate image displayed on the output device 20 lacks the information of the pixel corresponding to the portion touch-operated by the user. Further, the reception unit 135 provides intervention information indicating that the intermediate image displayed on the output device 20 lacks the information of the pixel corresponding to the portion superimposed with the binarized information of the image captured by the user. accept.

The reception unit 135 sends the received intervention information to the calculation unit 133. The calculation unit 133 calculates the node in the intermediate layer of the next stage of the intermediate layer from which the intermediate image is output, based on the received intervention information. That is, the reception unit 135 receives the intervention information until the calculation of the model reaches the output layer, and the calculation unit 133 repeats the calculation up to the output layer of the model.

(About the result output unit 136)
The result output unit 136 outputs the second image from the output layer of the model based on the intervention information received by the reception unit 135. Specifically, the result output unit 136 acquires the information of the node corresponding to the output layer as a result of the calculation unit 133 performing the calculation based on the intervention information received by the reception unit 135. Then, the result output unit 136 outputs the second image which is the calculation result of the model based on the node of the output layer. For example, the result output unit 136 outputs the second image as image data in which the number of pixels and color information in a format that can be displayed on the output device 20 are set.

[4. Display control device configuration]
Next, the configuration of the display control device 30 according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing a configuration example of the display control device 30 according to the embodiment. As shown in FIG. 7, the display control device 30 includes a communication unit 31, a storage unit 33, and a control unit 32. The display control device 30 includes an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from an administrator or the like who uses the display control device 30, and an output unit (for example, a liquid crystal display, etc.) for outputting various information. ) May have.

(About communication unit 31)
The communication unit 31 is realized by, for example, a NIC or the like. The communication unit 31 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the input device 10, the output device 20, and the output device 100 via the network N.

(About the storage unit 33)
The storage unit 33 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 33 according to the embodiment includes an intermediate image storage unit 34 and an intervention information storage unit 35.

(About the intermediate image storage unit 34)
The intermediate image storage unit 34 stores the intermediate image transmitted from the output device 100. FIG. 8 shows an example of the intermediate image storage unit 34 according to the embodiment. FIG. 8 is a diagram showing an example of the intermediate image storage unit 34 according to the embodiment. As shown in FIG. 8, the intermediate image storage unit 34 has items such as “intermediate image ID”, “number of pixels”, and “color information”.

The "intermediate image ID" indicates identification information for identifying the intermediate image. The "number of pixels" and "color information" correspond to the same items shown in FIG.

That is, in FIG. 8, as an example of the information stored in the intermediate image storage unit 34, the intermediate image G01 having the intermediate image ID “G01” has the number of pixels “H01” and the color information “J01”. It is shown that.

(About the intervention information storage unit 35)
The intervention information storage unit 35 stores the intervention information. Here, FIG. 9 shows an example of the intervention information storage unit 35 according to the embodiment. FIG. 9 is a diagram showing an example of the intervention information storage unit 35 according to the embodiment. As shown in FIG. 9, the intervention information storage unit 35 has items such as "intervention information ID" and "missing part information".

The "intervention information ID" indicates identification information that identifies the intervention information. The “missing part information” indicates a part of the pixels included in the intermediate image that is missing by the user's operation. In the example shown in FIG. 9, "missing part information" is shown by a concept such as "L01", but in reality, specific information corresponding to the missing part is stored in the item of missing part information. Will be done. For example, the missing part information is the specific coordinate information in the intermediate image corresponding to the part where the user has touched, or the specific part determined to be black (shadow) in the binarization information of the image captured by the user. Coordinate information etc. are stored.

That is, in FIG. 9, as an example of the information stored in the intervention information storage unit 35, the intervention information K01 whose intervention information ID is “K01” indicates that the missing portion information is “L01”.

(About control unit 32)
Returning to FIG. 7, the explanation will be continued. The control unit 32 is a controller, and is realized by, for example, using a CPU, an MPU, or the like to execute various programs stored in the storage device inside the display control device 30 using the RAM as a work area. Further, the control unit 32 is a controller, and is realized by, for example, an integrated circuit such as an ASIC or FPGA.

As shown in FIG. 7, the control unit 32 according to the embodiment includes a reception unit 36, a display control unit 37, a generation unit 38, and a transmission unit 39, and has functions and operations of information processing described below. Realize or execute. The internal configuration of the control unit 32 is not limited to the configuration shown in FIG. 7, and may be any other configuration as long as it is configured to perform information processing described later. Further, the connection relationship of each processing unit included in the control unit 32 is not limited to the connection relationship shown in FIG. 7, and may be another connection relationship.

(About receiver 36)
The receiving unit 36 receives various information. For example, the receiving unit 36 receives information about the intermediate image from the output device 100. Specifically, the receiving unit 36 receives information (number of pixels, color information, etc.) about the pixels constituting the intermediate image. The receiving unit 36 stores the received intermediate image in the intermediate image storage unit 34.

(About the display control unit 37)
The display control unit 37 displays the intermediate image output by the intermediate output unit 134 related to the output device 100 on an arbitrary display device (output device 20 in the embodiment).

Further, the display control unit 37 displays an intervention image, which is an image in which the intervention information generated by the generation unit 38 described later is reflected in the intermediate image, on the display device. Specifically, when a touch operation is performed on the intermediate image displayed on the output device 20, the display control unit 37 identifies the pixel on which the touch operation is performed and controls the pixel to be displayed in black. In this way, the intervention image is displayed on the output device 20 by the process of displaying the intermediate image in black. As a result, the user can recognize which position he / she touched in the intermediate image.

(About the generator 38)
The generation unit 38 generates intervention information based on the intervention processing for the intermediate image displayed by the display control unit 37. Specifically, the generation unit 38 specifies the coordinate information of the pixel that has received the intervention process from the user in the intermediate image, and generates the intervention information indicating the specified coordinate information. The generation unit 38 stores the generated intervention information in the intervention information storage unit 35.

Specifically, the generation unit 38 generates intervention information in which the information of the selected portion in the intermediate image is omitted, based on the user's selection operation for the output device 20 in which the intermediate image is displayed. The user selection operation means, for example, a touch operation on the output device 20 having a touch sensor, or an operation in which the user selects pixels constituting an intermediate image by using another pointing device. This allows the user to visually select the information to be omitted while visually recognizing the intermediate image, and influence the processing of the neural network.

The processing performed by the generation unit 38 will be described with reference to FIGS. 10 and 11. FIG. 10 is a diagram (1) showing an example of the intervention process according to the embodiment. FIG. 10 shows an example in which the user performs an intervention process by a touch operation on the intermediate image 61 displayed on the display 60.

In the example of FIG. 10, the generation unit 38 detects the user's touch operation via the display 60 (step S31). For example, the user touches a part of the intermediate image 61 with the finger 65. The generation unit 38 identifies the missing portion based on the coordinate information corresponding to the portion touched by the user (step S32). Specifically, the generation unit 38 specifies the coordinate position corresponding to the display 66 indicating the portion touched by the finger 65 as the missing portion. Then, the generation unit 38 generates intervention information, which is information in which some information is missing from the intermediate image 61. The intervention information may be represented by information indicating only the missing portion, or may be represented as image information in which some information is missing from the intermediate image 61.

Next, with reference to FIG. 11, the flow of intervention information generated by the generation unit 38 is shown. FIG. 11 is a diagram (1) showing the procedure of the intervention process according to the embodiment. As shown in FIG. 11, first, the receiving unit 36 receives the intermediate layer output of the mth layer (m is an arbitrary integer) from the output device 100. In FIG. 11, “ ^{RN (m) × W (m) × H (m)} ” indicates image information of the intermediate layer of the m layer. Note that "N" indicates the number of channels, "W" indicates the coordinates of the horizontal axis in the image, and "H" indicates the coordinates of the vertical axis in the image. “H (m)” indicates an element of the intermediate layer output of the m layer.

The display control unit 37 shapes the received intermediate layer output and converts it into a format that can be displayed by the output device 20 (display). The image after shaping is represented by, for example, "RW ^{× H} ", and each element (information of each pixel) is represented by "X".

On the other hand, as shown in FIG. 11, the generation unit 38 identifies the coordinate position touched by the user via the touch sensor on which the shaped intermediate image is displayed, and generates intervention information. In addition, "M" shown in FIG. 11 indicates an element of intervention information. For example, in FIG. 11, the intervention information is shown to be the coordinates “M” included in the image indicated by the ^{intermediate image “RW × H”.} That is, in FIG. 11, the intervention information is information indicating the coordinates “M” touched by the user in the intermediate image.

Then, the transmission unit 39 transmits the intervention information generated by the generation unit 38 to the output device 100. Further, the display control unit 37 performs an operation related to the intervention process based on the intervention information and the intermediate image. For example, the display control unit 37 calculates the element product of the element "X" of the intermediate image and the element "M" of the intervention information, and calculates the information regarding the intermediate image after the intervention ("○" in FIG. 11). Indicates the calculation of the element product). Then, the display control unit 37 displays the intermediate image after the intervention on the display.

Further, the generation unit 38 is one of the intermediate images based on the binarization information generated by controlling the image pickup device (camera) installed in the output device 20 and the intermediate image displayed on the output device 20. Generate intervention information with some or all information missing. As a result, the output device 100 can change the output result of the neural network each time the user moves his / her body, so that the user can be presented with an unexpected output result to entertain the user.

The processing performed by the generation unit 38 will be described with reference to FIGS. 12 and 13. FIG. 12 is a diagram (2) showing an example of the intervention process according to the embodiment. FIG. 12 shows an example in which a user is imaged by a camera 75 installed in the vicinity of the user and intervention information is generated based on the captured image.

In the example of FIG. 12, the generation unit 38 acquires an image 90 captured by the user via the camera 75 (step S41). Subsequently, the generation unit 38 converts the acquired image 90 into binarized data (step S42). For example, the generation unit 38 sets a pixel having a brightness lower than a predetermined threshold value as “0” and a pixel having a brightness higher than a predetermined threshold value as “1”.

Further, the generation unit 38 acquires the intermediate image 91 received from the output device 100 (step S43). Then, the generation unit 38 superimposes the binarized data of the image 90 on the intermediate image 91 (step S44). Then, the generation unit 38 specifies, among the intermediate images 91, the pixels that are superimposed on the pixels determined to be “0” when the image 90 is binarized, as the data to be omitted. The generation unit 38 generates intervention information based on the missing data.

Next, with reference to FIG. 13, the flow of the intervention information generated by the generation unit 38 is shown. FIG. 13 is a diagram (2) showing the procedure of the intervention process according to the embodiment. As shown in FIG. 13, first, as in FIG. 11, the receiving unit 36 receives the intermediate layer output of the mth layer (m is an arbitrary integer) from the output device 100.

Further, as shown in FIG. 13, the generation unit 38 binarizes the camera image acquired from the camera to obtain a binarized image. Then, the generation unit 38 resizes the binarized image so that it can be superimposed on the intermediate image. Then, the generation unit 38 generates intervention information (indicated by the element “Y” shown in FIG. 13) based on the binarization information. The transmission unit 39 transmits the intervention information generated by the generation unit 38 to the output device 100. Further, the display control unit 37 calculates the element product of the element "X" of the intermediate image and the element "Y" of the intervention information, and calculates the information regarding the intermediate image after the intervention. Then, the display control unit 37 displays the intermediate image after the intervention on the display. The generation unit 38 may use the element product of the element "X" of the intermediate image and the binarization information as intervention information.

Here, with reference to FIG. 14, an outline of the calculation performed by the output device 100 will be described based on the intervention information acquired from the display control device 30. FIG. 14 is a schematic diagram showing a procedure of output processing according to the embodiment. As shown in FIG. 14, after the user face image is input, the output device 100 performs the calculation of the first layer and obtains the output of the first layer (first intermediate image) (note that "f" is active. (Indicates the conversion function). Then, the output device 100 executes the intervention process of the first layer (process similar to the “intervention process” shown in FIGS. 11 and 13), and obtains the output of the first layer that has undergone the intervention process.

After that, the output device 100 performs the calculation of the second layer and obtains the output of the second layer (second intermediate image). By repeating these processes, the output device 100 performs operations up to the output layer. In the example shown in FIG. 14, since the output device 100 performs an operation using a DNN having a plurality of intermediate layers called the M layer, a DNN output is obtained from the output layer as a final result. In the embodiment, the DNN output is image information.

(About transmitter 39)
The transmission unit 39 transmits various information. For example, the transmission unit 39 transmits the intervention information generated by the generation unit 38 to the output device 100.

[5. Processing procedure]
Next, the procedure of processing by the output device 100 according to the embodiment will be described with reference to FIG. FIG. 15 is a flowchart showing a processing procedure according to the embodiment.

As shown in FIG. 15, the output device 100 acquires a user's face image via an input device 10 such as a camera (step S101). The output device 100 inputs the acquired face image to the model (step S102).

After that, the output device 100 determines whether or not a predetermined time (for example, a time set as an experience time for the user to experience a change in the output of the neural network) has elapsed (step S103). When the predetermined time has not elapsed (step S103; No), the output device 100 receives the intervention information for the output intermediate image from the display control device 30 (step S105). Subsequently, the output device 100 calculates the information of the next layer based on the intervention information (step S106). Specifically, the output device 100 calculates the value of each node constituting the next layer based on the output of the previous stage that has undergone the intervention process.

After that, the output device 100 determines whether or not the calculated layer (next layer) is an output layer (step S107). If it is not an output layer (step S107; No), the output device 100 repeats a process of outputting an intermediate image of the next intermediate layer (step S104).

On the other hand, when the next layer is an output layer (step S107; Yes), the output device 100 outputs a second image which is an output result of the model from the output layer (step S108). Then, the output device 100 displays the output second image on the output device 20 or the like (step S109). After that, when the predetermined time has elapsed (step S103; Yes), the output device 100 ends a series of output processes.

Although not shown in FIG. 15, the output device 100 executes a process of initializing the intervention information up to that point at an arbitrary timing (for example, immediately before the user's face image is first input to the model). You may.

[6. Modification example]
The output system 1 according to the above-described embodiment may be implemented in various different forms other than the above-described embodiment. Therefore, in the following, other embodiments of each device included in the above output system 1 will be described.

[6-1. Output device configuration]
In the above embodiment, an example is shown in which the output device 100 outputs an intermediate image to the display control device 30 and receives intervention information via the display control device 30. Here, these processes may be performed only by the output device 100.

This point will be described with reference to FIG. FIG. 16 is a diagram showing a configuration example of the output device 200 according to the modified example. As shown in FIG. 16, the output device 200 further includes an intermediate image storage unit 123, an intervention information storage unit 124, a display control unit 137, and a generation unit 138, as compared with the output device 100.

The intermediate image storage unit 123 stores the same information as the intermediate image storage unit 34 related to the display control device 30. The intervention information storage unit 124 stores the same information as the intervention information storage unit 35 related to the display control device 30. Further, the display control unit 137 executes the same processing as the display control unit 37 related to the display control device 30. The generation unit 138 executes the same processing as the generation unit 38 related to the display control device 30.

That is, the output device 100 may execute the process executed by the display control device 30 by its own device. As a result, the output device 100 can execute the output process according to the embodiment with a simple system design.

[6-2. Each device of the output system]
Each device included in the output system 1 may be realized by various modifications. For example, the output system 1 may include a shooting PC for controlling the camera 50 that captures the user. The output device 100 may include a display that displays a second image that is the output result of the model. In this case, the output device 100 is realized by an information processing terminal that also functions as a display device, such as a notebook PC or a tablet.

[6-3. Intervention processing]
In the above embodiment, as an intervention process, an example of a touch operation by a user, image information captured by a camera, or the like is shown. However, the intervention process may be various user operations performed on the intermediate image, voice data generated by the user, or the like.

Further, in the above embodiment, it is shown that the output device 100 uses the information in which a part of the information displayed as the intermediate image is omitted as the intervention information. However, the intervention information is not necessarily the one in which a part of the intermediate image is omitted, but may be the one in which a part of the intermediate image is changed. For example, the intervention information may be information obtained by arbitrarily changing the information of the pixels shown as the intermediate image (for example, increasing / decreasing the brightness and the color information).

[7. others〕
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the model storage unit 121 and the image storage unit 122 shown in FIG. 4 may not be held by the output device 100 but may be held by an external storage server or the like. In this case, the output device 100 acquires the model, image information, and the like by accessing the storage server.

Further, for example, the output device 100 described above is a front-end server that mainly executes communication with an external device, such as acquiring intervention information from the display control device 30 and outputting a second image to the output device 20. It may be distributed between the side and the back-end server side that mainly executes arithmetic processing using the model.

[8. Hardware configuration]
Further, the output device 100, the input device 10, the output device 20, the display control device 30, and the like according to the above-described embodiment are realized by, for example, a computer 1000 having a configuration as shown in FIG. Hereinafter, the output device 100 will be described as an example. FIG. 17 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the output device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F). ) Has 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via the communication network 500 (corresponding to the network N shown in FIG. 3) and sends the data to the CPU 1100, and also receives data generated by the CPU 1100 via the communication network 500. Send to the device of.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the data generated via the input / output interface 1600 to the output device.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the output device 100, the CPU 1100 of the computer 1000 realizes the function of the control unit 130 by executing the program loaded on the RAM 1200. Further, each data in the storage unit 120 is stored in the HDD 1400. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired from another device via the communication network 500.

[9. effect〕
As described above, the output device 100 according to the embodiment includes an input unit 132, an intermediate output unit 134, a reception unit 135, and a result output unit 136. The input unit 132 inputs a first image to be processed into a model that is a neural network that outputs an image. The intermediate output unit 134 outputs an intermediate image which is an image in the intermediate layer of the model to which the first image is input. The reception unit 135 receives intervention information, which is information that reflects the intervention process for the intermediate image. The result output unit 136 outputs the second image from the output layer of the model based on the intervention information received by the reception unit 135.

As described above, the output device 100 according to the embodiment enables the user to intervene in the neural network by receiving the intervention information for the intermediate image output from the intermediate layer in the process of the calculation of the neural network. As a result, the output device 100 can dynamically correct the result output from the model.

Further, the intermediate output unit 134 outputs a first intermediate image which is an image in the first intermediate layer of the model to which the first image is input, and then the reception unit 135 reflects the intervention process for the first intermediate image. When the first intervention information is accepted, the second intermediate image, which is an image in the intermediate layer of the next stage of the model in which the first intervention information is input, is output.

As described above, when the output device 100 according to the embodiment extends over a plurality of intermediate layers, the output device 100 may output an intermediate image for each intermediate layer. As a result, the output device 100 can accept the intervention by the user in various processes of the neural network, so that the intervention of the user can be reflected in the output result in more detail.

In addition, the reception unit 135 receives intervention information, which is information in which part or all of the information of the intermediate image is omitted by the intervention process for the intermediate image.

As described above, the output device 100 according to the embodiment receives the intervention information in which the intermediate image is omitted. In other words, the output device 100 receives information that omits the characteristic portion extracted by the neural network. As a result, the output device 100 can delete some features of the input image from the calculation of the neural network, so that the user can easily understand how the output result is changed by the intervention process. Can be done.

Further, the output device 200 according to the modified example intervenes based on the display control unit 137 that displays the intermediate image output by the intermediate output unit 134 on the display device and the intervention process for the intermediate image displayed by the display control unit 137. Further includes a generation unit 138 for generating information. Further, the reception unit 135 receives the intervention information generated by the generation unit 138.

As described above, the output device 200 according to the modified example may execute the process of displaying the intermediate image and the process of generating the intervention information by the own device. As a result, the output device 200 can execute the output process with a simpler system.

Further, the generation unit 138 generates intervention information in which the information of the selected portion in the intermediate image is omitted, based on the user's selection operation on the display device on which the intermediate image is displayed.

As described above, the output device 200 according to the modification can generate the missing information based on the user's selection operation, so that the change of the second image (output result) linked with the user's movement can be presented to the user. It is possible to provide the user with entertainment such as enjoying the change of the second image. Further, the output device 200 can allow the user to experience the internal processing of the model, which cannot be normally recognized, such as what part of the intermediate image is changed and how the output result is affected. ..

Further, the generation unit 138 information on a part or all of the intermediate image based on the binarization information generated by controlling the image pickup device installed in the display device and the intermediate image displayed on the display device. Generate intervention information that is missing.

As described above, the output device 200 according to the modified example may generate intervention information based on the landscape or the like captured by the camera or the like. As a result, the output device 200 can reflect changes not intended by the user in the output result, so that an impressive demonstration or the like can be performed.

Further, the display control unit 137 displays an intervention image, which is an image in which the intervention information generated by the generation unit 138 is reflected in the intermediate image, on the display device.

As described above, the output device 200 according to the modified example presents to the user an intermediate image reflecting the intervention information so that the portion touched by the user can be understood. As a result, the output device 200 allows the user to perceive how the output result changes by touching which part, so that the user can realize the change due to the calculation by the model or the intervention. ..

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other modified forms.

Further, the output device 100 described above may be realized by a plurality of server computers, and depending on the function, an external platform or the like may be called by API (Application Programming Interface), network computing, or the like to realize the configuration. It can be changed flexibly.

Further, the "section, module, unit" described in the claims can be read as "means" or "circuit". For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

1 Output system 10 Input device 20 Output device 30 Display control device 100 Output device 110 Communication unit 120 Storage unit 121 Model storage unit 122 Image storage unit 123 Intermediate image storage unit 124 Intervention information storage unit 130 Control unit 131 Acquisition unit 132 Input unit 133 Calculation unit 134 Intermediate output unit 135 Reception unit 136 Result output unit 137 Display control unit 138 Generation unit

Claims (10)

An input unit that inputs the first image to be processed into a model that is a neural network that outputs images,
An intermediate output unit that outputs an intermediate image that is an image in the intermediate layer of the model to which the first image is input, and an intermediate output unit.
A reception unit that receives intervention information, which is information that reflects the intervention processing for the intermediate image, and
A result output unit that outputs a second image from the output layer of the model based on the intervention information received by the reception unit.
Equipped with
The reception department
An output device that is an intermediate image output by the intermediate output unit and that receives the intervention information generated based on an intervention process for the intermediate image displayed on the display device. The intermediate output unit
The first intermediate image, which is an image in the first intermediate layer of the model to which the first image is input, is output, and then the reception unit receives the first intervention information reflecting the intervention process for the first intermediate image. If so, the second intermediate image, which is an image in the intermediate layer of the next stage of the model in which the first intervention information is input, is output.
The output device according to claim 1. The reception department
The intervention information, which is information in which a part or all of the information of the intermediate image is omitted by the intervention process for the intermediate image, is accepted.
The output device according to claim 1 or 2, wherein the output device is characterized by the above. A display control unit that displays the intermediate image output by the intermediate output unit on the display device, and
A generation unit that generates the intervention information based on the intervention processing for the intermediate image displayed by the display control unit.
Further prepare
The reception department
Receiving the intervention information generated by the generator,
The output device according to any one of claims 1 to 3. The generator is
Based on the user's selection operation for the display device on which the intermediate image is displayed, the intervention information in which the information of the selected portion in the intermediate image is omitted is generated.
The output device according to claim 4. The generator is
Based on the binarization information generated by controlling the image pickup device installed in the display device and the intermediate image displayed on the display device, a part or all of the information of the intermediate image is omitted. Generate the intervention information,
The output device according to claim 4 or 5. The display control unit
An intervention image, which is an image in which the intervention information generated by the generation unit is reflected in the intermediate image, is displayed on the display device.
The output device according to any one of claims 4 to 6. It ’s an output method that a computer uses.
An input process for inputting the first image to be processed into a model that is a neural network that outputs images,
An intermediate output step of outputting an intermediate image which is an image in the intermediate layer of the model to which the first image is input, and an intermediate output step.
The reception process for receiving intervention information, which is information that reflects the intervention process for the intermediate image,
A result output process for outputting a second image from the output layer of the model based on the intervention information received by the reception process.
Only including,
The reception process is
An output method that is an intermediate image output by the intermediate output step and that receives the intervention information generated based on an intervention process for the intermediate image displayed on the display device . An input procedure for inputting the first image to be processed into a model that is a neural network that outputs images,
An intermediate output procedure for outputting an intermediate image which is an image in the intermediate layer of the model to which the first image is input, and an intermediate output procedure.
The reception procedure for accepting intervention information, which is information that reflects the intervention process for the intermediate image, and
A result output procedure for outputting a second image from the output layer of the model based on the intervention information received by the reception procedure, and a result output procedure.
Let the computer run
The reception procedure is
Wherein an intermediate image that is output by the intermediate output procedure, the output program characterized Rukoto receiving the intervention information generated on the basis of the intervention process for intermediate image displayed on the display device. An output system having an output device and a display control device.
The output device is
An input unit that inputs the first image to be processed into a model that is a neural network that outputs images,
An intermediate output unit that outputs an intermediate image that is an image in the intermediate layer of the model to which the first image is input, and an intermediate output unit.
A receiver for receiving the intervention information is information reflecting the intervention process for pre-Symbol intermediate image,
A result output unit that outputs a second image from the output layer of the model based on the intervention information received by the reception unit.
Equipped with
The display control device is
A display control unit that displays the intermediate image output by the intermediate output unit on the display device, and
A generation unit that generates the intervention information based on the intervention processing for the intermediate image displayed by the display control unit.
Equipped with
The reception department
Receiving the intervention information generated by the generator,
An output system characterized by that.

Images