JP7405198B2 - Image processing device, image processing method, and image processing program - Google Patents

Description

Embodiments of the present invention relate to an image processing device, a learning device, an image processing method, a learning method, an image processing program, and a learning program.

2. Description of the Related Art Conventionally, a background subtraction method is known as a method for detecting a moving object appearing in the foreground (hereinafter also referred to as foreground) from a moving image captured by a camera. In the background subtraction method, a background image (also called a background model) in which a detection target object is not photographed is detected and stored from a moving image photographed by a camera. Then, by calculating the difference between the background images from the moving images captured by the camera, an image area corresponding to the foreground is detected.

In this way, the method for determining background/foreground from an image is to input image information output from an image sensor and temporally delayed image information into the input layer, and output information corresponding to the difference from the output layer. A method using a neural network is known.

Japanese Patent Application Publication No. 2-173877

However, the above-mentioned conventional technology has a problem in that it is difficult to robustly distinguish between background and foreground when the foreground includes a similar color similar to the background, and against noise and the like.

For example, when the number of intermediate layers in a neural network is small, discrimination is based on local features such as edges and colors, so if the foreground contains a similar color that is similar to the background, the background/foreground It becomes difficult to distinguish. In addition, it is susceptible to the effects of noise and the like, which may result in false detection.

Additionally, if you increase the number of intermediate layers in a neural network, the connection weights are initialized with small random numbers at the beginning of learning, so the input signal will spread as it passes through the layers. , almost zero noise will be obtained from the neural network. For this reason, it is difficult to simply increase the number of intermediate layers in a neural network because meaningful information cannot be obtained even when compared with training data, and the learning of the neural network does not progress.

One aspect of the present invention is to provide an image processing device, a learning device, an image processing method, a learning method, an image processing program, and a learning program that enable background/foreground discrimination that is robust to noise and the like.

In the first proposal, the image processing device includes an acquisition section, a difference generation section, a neural network, and an output section. The acquisition unit acquires a background image of a background photographed in advance and a target image to be determined. The difference generation unit generates a difference image showing a first difference between the background image and the target image. By inputting each of the background image and the target image into the neural network, the neural network shows the second difference between the difference image and the output target map image, which distinguishes the background and foreground from the target image. Estimate the residuals. The output unit outputs a map image in which the background and foreground are distinguished from the target image based on the difference image generated by the difference generation unit and the difference estimated by the neural network.

According to one embodiment of the present invention, it is possible to perform background/foreground discrimination that is robust to noise and the like.

FIG. 1 is an explanatory diagram illustrating an overview of the embodiment. FIG. 2 is an explanatory diagram illustrating an example of a background image, a target image, and a teacher image. FIG. 3 is an explanatory diagram of a conventional method of estimating a foreground map using a neural network. FIG. 4 is an explanatory diagram illustrating discrimination based on high-order features/low-order features. FIG. 5 is a block diagram showing an example of the functional configuration of the image processing device according to the embodiment. FIG. 6 is a flowchart illustrating an example of the operation of the image processing apparatus according to the embodiment. FIG. 7 is an explanatory diagram illustrating the neural network of the residual estimation section. FIG. 8 is a block diagram showing an example of the functional configuration of the learning device according to the embodiment. FIG. 9 is a flowchart illustrating a learning flow. FIG. 10 is an explanatory diagram showing an example of a computer that executes a program.

Hereinafter, an image processing device, a learning device, an image processing method, a learning method, an image processing program, and a learning program according to embodiments will be described with reference to the drawings. In the embodiments, components having the same functions are denoted by the same reference numerals, and redundant explanations will be omitted. Note that the image processing device, learning device, image processing method, learning method, image processing program, and learning program described in the following embodiments are merely examples, and do not limit the embodiments. Furthermore, the embodiments described below may be combined as appropriate within a range that does not contradict each other.

FIG. 1 is an explanatory diagram illustrating an overview of the embodiment. As shown in FIG. 1, in this embodiment, a target image G1 whose background and foreground are to be determined and a background image G2 of the background photographed in advance are input, and the background and foreground included in the target image G1 are input. A foreground map G5 that distinguishes the foreground is obtained.

FIG. 2 is an explanatory diagram illustrating an example of the target image G1, the background image G2, and the teacher image G6. As shown in FIG. 2, the target image G1 is image data for distinguishing between the foreground, such as a person H in the shooting range, and the background, and is, for example, image data of a surveillance camera for detecting a suspicious person. be. The background image G2 is image data of a background photographed in advance. Note that the background image G2 may be a background model generated by overlapping several images.

The foreground map G5 is an example of a map image, and is image data in which, for example, an area corresponding to the background in the target image G1 is set as black pixels, and an area corresponding to the foreground is set as white pixels. By multiplying the foreground map G5 obtained in this manner by the target image G1, for example, the foreground included in the target image G1 can be identified. For example, by using the identification result of the foreground included in the target image G1, it can be applied to extraction of a silhouette of a subject in free viewpoint video generation technology, or to extraction of a suspicious person in video surveillance technology.

Specifically, in the inference phase for obtaining the foreground map G5, difference generation (S1) is performed to generate a difference between the input target image G1 and the background image G2, and a difference image G3 is generated. In addition, based on the input target image G1 and background image G2, a foreground map G5 that distinguishes the background and foreground from the target image G1 and a residual G4 indicating the difference between the difference image G3 (in the difference image G3, the foreground map that is the correct answer) are created. Residual estimation (S2) is performed to estimate the information missing from map G5 using a neural network. Next, a foreground map G5 is obtained by adding together the difference image G3 generated in the difference generation (S1) and the residual G4 estimated in the residual estimation (S2) (S3).

In addition, in the learning phase in which the neural network that performs residual estimation (S2) is trained, the connection weight of each node constituting the neural network is adjusted by comparing the foreground map G5 and the teacher image G6.

As shown in FIG. 2, the teacher image G6 is teacher data indicating the correct answer regarding the background/foreground in the target image G1 input during learning of the neural network. As an example, the teacher image G6 includes image data in which a foreground region R1 corresponding to the foreground (person H) included in the target image G1 is a white pixel, and a region other than the foreground region R1 (background region) is a black pixel. .

By performing supervised learning using this teacher image G6, the neural network of the residual estimation unit 11 is trained to appropriately estimate the residual G4 of the portion lacking in the correct foreground map G5. .

FIG. 3 is an explanatory diagram of a conventional method of estimating a foreground map using a neural network. As shown in FIG. 3, in the conventional method, the target image G1 and the background image G2 are input to the input layer 201 of the neural network 200, and the background/foreground discrimination result G4a is directly output from the output layer 203 via the intermediate layer 202. be done.

When the number of layers in the intermediate layer 202 in the neural network 200 is small, high-order features cannot be obtained from the input image, and discrimination is performed based on low-order features, that is, local features such as edges and colors. I will do it. Here, the high-level features are features that include semantic information, such as whether a certain area on the image is a person or a car, or whether it is inside or outside. The low-order features are features of the local structure of the image, such as vertical and horizontal edges and flat areas.

FIG. 4 is an explanatory diagram illustrating discrimination based on high-order features/low-order features. As shown in FIG. 4, in case C1, a discrimination result G11 is obtained from the input image G10 in which the human area is determined to be a white pixel using a neural network 200a of a deep neural network (DNN) having multiple intermediate layers. ing. In DNN, features are gradually abstracted from the target image G1 through layers, such as edges → squares and circles (combinations of edges) → tires and faces (combinations of squares and circles) →... The following features can be extracted. When discrimination is performed based on such high-order features, background/foreground discrimination can be performed robustly against noise and the like.

On the other hand, in case C2, the three-layer neural network 200b is used to obtain the discrimination result G11 from the input image G10. Since the three-layer neural network 200b performs discrimination based on local low-order features, it is susceptible to the effects of noise and the like, resulting in false detections. For example, the input image G10 is an image of a person playing across the lines of the court, and the person includes a similar color similar to the color of the court in the area to the right of the line. Furthermore, in the area on the left side of the line, the color of the coat and the color of the person (left foot) are not similar. Therefore, in the determination result G11 of case C2, a portion where the foreground includes a similar color similar to the background is not determined as the foreground (part of the left foot is correctly determined as the foreground).

Returning to FIG. 3, when increasing the number of intermediate layers 202 in the neural network 200, the input signal from the input layer 201 ( The target image G1 and the background image G2) are diffused and weakened with each layer. Therefore, the discrimination result G4a obtained from the output layer 203 becomes approximately 0 noise at the initial stage of learning. Therefore, even if compared with the teacher image G6, no meaningful information can be obtained (because the direction of the gradient is not determined), and it becomes difficult to adjust the parameters indicating the connection weights of each node in the neural network 200, and the learning will not progress.

On the other hand, in the present embodiment, as shown in FIG. (information missing from the foreground map G5) is estimated using a neural network. Then, the foreground map G5 is obtained by adding the difference image G3 and the residual G4.

Therefore, even if the residual G4 obtained at the beginning of learning is 0, a meaningful output (foreground map G5) can be obtained because the difference image G3 is added, and a comparison with the teacher image G6 can be made. This allows neural network learning to proceed. Therefore, it is possible to obtain a foreground map G5 based on the residual G4 discriminated based on high-order features using DNN, and it is possible to robustly distinguish between background and foreground even when background similar colors, noise, etc. are included. be able to.

FIG. 5 is a block diagram showing an example of the functional configuration of the image processing device according to the embodiment. As shown in FIG. 5, the image processing device 1 includes a difference generation section 10, a residual estimation section 11, and an output section 12.

The difference generation unit 10 generates a difference image G3 based on the difference between the input target image G1 and the background image G2. That is, the difference generation unit 10 is an example of a generation unit. For example, the difference generation unit 10 generates the difference image G3 by determining the difference in pixel values between corresponding pixels in the target image G1 and the background image G2. For this difference, the L1 norm (absolute value of difference), L2 norm, etc. of pixel values can be used.

Furthermore, the targets for which the difference generation unit 10 calculates the difference are not limited to pixel values in mutually corresponding pixels. For example, the difference generation unit 10 may calculate the difference between the feature amounts calculated from each pixel of the target image G1 and the background image G2, and generate the difference image G3.

As an example, local features may be used as shown in the following document.
・SIFT (Scale-Invariant Feature Transform): David G.Lowe, “Distinctive image features from scale-invariant keypoints”, Int.Journal of Computer Vision,Vol.60, No.2, pp.91-110, 2004.
・SURF (Speeded-Up Robust Features): H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded Up Robust. Features”, In ECCV , pp.404-417, 2006.
・BRIEF (Features from Accelerated Segment Test): M.Calonder, V.Lepetit and C.Strecha and P.Fua, “BRIEF: Binary Robust Independent Elementary Features”, In Proc. European Conference on Computer Vision, pp.778-792 , 2010.
・ORB (Oriented FAST and Rotated BRIEF): E.Rublee, V.Rabaud, K.Konolige and G.Bradski “ORB: an efficient alternative to SIFT or SURF”, In Proc. International Conference on Computer Vision, 2011.

Further, the difference generation unit 10 may use, as the feature amount, the value of the intermediate layer of a DNN that has been trained for general object recognition, as shown in the following document.
・AlexNet: Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
・GoogLeNet: Szegedy, Christian, et al. "Going deeper with convolutions." Cvpr, 2015.
・VGG: Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
・ResNet: He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

The residual estimation unit 11 uses the input target image G1 and background image G2 to calculate a residual G4 indicating the difference between a difference image G3 and a foreground map G5 that distinguishes the background and foreground from the target image G1 using a neural network. Estimate using That is, the residual estimation unit 11 is an example of an estimation unit.

The output unit 12 outputs a foreground map G5 based on the difference image G3 generated by the difference generation unit 10 and the residual G4 estimated by the residual estimation unit 11. Specifically, the output unit 12 outputs a foreground map G5 obtained by adding pixel values of corresponding pixels in the difference image G3 and the residual G4. This addition may be weighted addition. This weight may be a value predetermined by the designer, or may be a variable parameter. Variable parameters may be optimized when the neural network of the residual estimation unit 11 is trained.

FIG. 6 is a flowchart showing an example of the operation of the image processing device 1 according to the embodiment. As shown in FIG. 6, when the process is started, the image processing device 1 acquires a background image G2 stored in advance in a memory, hard disk, database, storage on a network, etc. (S10). Similarly, the image processing device 1 acquires the target image G1 (S11).

Note that when an image from a surveillance camera is used as the target image G1, the image processing device 1 may directly acquire the target image G1 from the surveillance camera via an interface connected to the surveillance camera. Further, the image processing device 1 may perform preprocessing such as noise removal and color correction when acquiring the target image G1 and the background image G2.

Next, the image processing device 1 inputs the target image G1 and the background image G2 to the difference generation unit 10 (S12, S13). Next, the difference generation unit 10 generates a difference between the target image G1 and the background image G2 (S14), and obtains a difference image G3.

Next, the image processing device 1 inputs the target image G1 and the background image G2 to the residual estimation unit 11 (S15, S16). Next, the residual estimation unit 11 estimates a residual G4 indicating the difference between the foreground map G5 and the difference image G3 from the input target image G1 and background image G2 using a neural network (S17).

The neural network of the residual estimation unit 11 is subjected to parameter adjustment in a learning phase by a learning device, which will be described later, so as to appropriately estimate the residual G4.

FIG. 7 is an explanatory diagram illustrating the neural network of the residual estimation unit 11. As shown in FIG. 11, the neural network 11a has a network structure in which units resembling brain neurons are connected in a hierarchical manner. There are many neurons (nerve cells) in the brain. Each neuron receives signals from and passes signals to other neurons. The brain performs various information processing based on this signal flow. The neural network 11a is a model that realizes the characteristics of such brain functions on a computer.

Specifically, the neural network 11a may be a deep neural network having multiple intermediate layers from a layer into which the target image G1 and background image G2 are input to a layer that outputs the residual G4. The plurality of intermediate layers include, for example, a convolution layer, an activation function layer, a pooling layer, a fully connected layer, and a softmax layer. The number and location of each layer may be changed at any time depending on the required architecture. That is, the hierarchical structure of the neural network 11a and the configuration of each layer can be determined in advance by the designer depending on the object to be identified.

In addition, the neural network 11a has a configuration as a CNN (convolutional neural network) in which convolutional layers and pooling layers are alternately stacked to enable feature extraction from input image data. You can. Further, the neural network 11a may be configured of a multilayer array of fully connected layers instead of a CNN. In this case, for the target image G1 and the background image G2, vectors arranged in a line according to a specific method such as raster scan order may be input.

For example, the neural network 11a may use a network structure as shown in the following document.
・FCN (Fully Convolutional Networks): Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
・U-Net: Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

The neural network 11a in FIG. 7 is an example of a network structure when the above-mentioned U-Net is applied. For example, in the neural network 11a, when the target image G1 and the background image G2 are color images of three RGB channels, the input is six channels in which they are superimposed. This input passes through a convolution layer, an expanded convolution layer, a pooling layer, a batch normalization layer, an activation function layer, etc., and outputs a one-channel residual G4.

Returning to FIG. 6, following S17, the output unit 12 adds the difference image G3 and the residual G4 (S18), and outputs the foreground map G5 (S19).

In the image processing device 1, the obtained foreground map G5 can be applied to extraction of a silhouette of a subject in free viewpoint video generation technology and extraction of a suspicious person in video surveillance technology.

For example, a technique for creating an arbitrary viewpoint video from camera images from multiple viewpoints is called free viewpoint video generation. By using this free-viewpoint video generation technology, it is possible to generate videos from a viewpoint other than the camera that took the picture, or to create videos with camera work that is impossible in reality, and it is possible to create dynamic and realistic video content. Users can view the video from their own preferred angles.

Examples of applications of this free-viewpoint video generation technology include the following. A foreground map G5 is obtained for each camera image, and a silhouette of a foreground object such as a person is extracted based on the obtained foreground map G5. Next, a method called Visual Hull is used to restore the three-dimensional structure of the foreground object and render the image from an arbitrarily set viewpoint.

Further, examples of application of video surveillance technology to the extraction of suspicious persons include the following. A foreground map G5 is obtained for the image of the surveillance camera, and the number of pixels in the foreground region is obtained. If the number of pixels in this foreground area is equal to or greater than a predetermined threshold, a detection signal is output as a result of detecting a suspicious object.

Next, details of the learning device that performs learning (learning phase) of the neural network 11a will be described. FIG. 8 is a block diagram showing an example of the functional configuration of the learning device according to the embodiment.

Note that for the target image G1, background image G2, and teacher image G6 in the learning phase, data of a learning dataset set in advance for learning is used. This learning data set is read out and used, for example, from a memory, hard disk, database, storage on a network, or the like. Further, for the learning data set, processing (data augmentation) may be performed to artificially increase the diversity of data, such as geometric transformation such as image rotation and scaling, or adding noise. Furthermore, when performing mini-batch learning, the number of mini-batches may be acquired for the target image G1, background image G2, and teacher image G6.

As shown in FIG. 8, the learning device 2 includes an error calculation section 20, a gradient calculation section 21, and an updating section 22.

The error calculation unit 20 receives input of the foreground map G5 outputted by the image processing device 1 and the teacher image G6. The error calculation unit 20 compares the input foreground map G5 and the teacher image G6 to calculate an error. That is, the error calculation section 20 is an example of a calculation section.

For example, the error calculation unit 20 calculates the error by determining the squared error of pixel values of mutually corresponding pixels in the foreground map G5 and the teacher image G6. This error is not limited to a squared error, but may be an L1 norm error, a Huber norm error used in robust statistics, or the like.

The gradient calculation section 21 calculates the gradient of the entire neural network 11a based on the error calculated by the error calculation section 20 based on the error backpropagation method commonly used in supervised learning.

The updating unit 22 calculates the update amount of the connection weights (parameters) of each node configuring the neural network 11a based on the gradient calculated by the gradient calculation unit 21. The updating unit 22 updates the parameters in the neural network 11a based on the calculated update amount.

To calculate the update amount in the update unit 22, for example, an optimization method as shown in the following literature can be used.
・SGD (stochastic gradient descent) with momentum: Goodfellow, Ian, et al. Deep learning. Vol. 1. Cambridge: MIT press, 2016.
・RMSProp: Geoffrey Hinton, Nitish Srivastava, Kevin Swersky. 2014. Lecture 6e: RMSProp: Divide the gradient by a running average of its recent magnitude (CSC321 Winter 2014).
・Adam: Diederik Kingma, Jimmy Ba. 2015. Adam: a method for stochastic optimization. the 3rd International Conference for Learning Representations (ICLR 2015).

FIG. 9 is a flowchart illustrating a learning flow. As shown in FIG. 9, when the process is started, a target image G1 and a background image G2 are acquired from the learning data set (S20, S21). Next, the image processing device 1 performs the same processing as S12 to S19, and outputs the foreground map G5 (S22 to S29). The foreground map G5 output from the image processing device 1 is input to the error calculation unit 20 of the learning device 2.

Next, the error calculation unit 20 of the learning device 2 obtains the teacher image G6 from the learning data set (S30). Next, the error calculation unit 20 compares the foreground map G5 output from the image processing device 1 and the teacher image G6 to calculate an error (S31).

Next, the gradient calculation unit 21 calculates the gradient of the entire neural network 11a based on the error calculated by the error calculation unit 20 (S32). Next, the updating unit 22 updates the connection weight of each node in the neural network 11a of the difference generating unit 10 according to the gradient calculated by the gradient calculating unit 21 (S33).

Next, the learning device 2 determines whether a predetermined learning completion condition is satisfied, such as whether learning using all data included in the learning data set has been completed (S34). If not satisfied (S34: NO), the learning device 2 returns the process to S20 and continues learning. If satisfied (S34: YES), the learning device 2 ends learning.

As described above, the difference generation unit 10 of the image processing device 1 generates the difference image G3 between the target image G1, which is a target for background and foreground discrimination, and the background image G2, which is the background. From the input target image G1 and background image G2, the residual estimation unit 11 of the image processing device 1 generates a residual G4 indicating the difference between the difference image G3 and a foreground map G5 that distinguishes the background and foreground from the target image G1. is estimated using the neural network 11a. The output unit 12 of the image processing device 1 outputs a foreground map G5 based on the difference image G3 generated by the difference generation unit 10 and the residual G4 estimated by the residual estimation unit 11.

As a result, the image processing device 1 can use the multilayered residual estimation unit 11 that discriminates high-level features, and is robust against noise, etc., when the foreground contains a similar color similar to the background. It becomes possible to perform background/foreground discrimination.

Further, the neural network 11a of the residual estimation unit 11 is a deep neural network (DNN) having multiple intermediate layers. Thereby, the residual estimation unit 11 can estimate the residual G4 based on the high-order features included in the input target image G1 and background image G2. Therefore, the image processing device 1 can perform background/foreground discrimination that is more robust against noise and the like.

Further, the neural network 11a of the residual estimation unit 11 is a convolutional neural network. Thereby, the residual estimation unit 11 can abstract the input target image G1 and background image G2 and obtain high-order features.

Further, the difference generation unit 10 generates a difference image G3 based on the difference in feature amount based on each pixel of the target image G1 and the background image G2. In this way, the difference image G3 may be a difference in feature amounts between the target image G1 and the background image G2.

The error calculation unit 20 of the learning device 2 also generates a foreground map G5 based on the difference image G3 generated by the difference generation unit 10 and the residual G4 estimated by the neural network 11a of the residual estimation unit 11. accept. The learning device 2 calculates the error between the received foreground map G5 and the teacher image G6. The updating unit 22 of the learning device 2 updates the parameters of the neural network 11a based on the calculated error. Thereby, the learning device 2 can perform learning of the neural network 11a that estimates the residual G4 indicating the difference between the foreground map G5 that distinguishes the background and foreground from the target image G1 and the difference image G3.

Note that each component of each illustrated device does not necessarily need to be physically configured as illustrated. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

The various processing functions performed by the image processing device 1 and the learning device 2 may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or an MCU (Micro Controller Unit)). good. In addition, various processing functions may be executed in whole or in part on a program that is analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware using wired logic. Needless to say, it's a good thing. Furthermore, various processing functions performed by the image processing device 1 and the learning device 2 may be executed by multiple computers working together through cloud computing.

By the way, the various processes described in the above embodiments can be realized by executing a program prepared in advance on a computer. Therefore, an example of a computer (hardware) that executes a program having the same functions as those of the above embodiment will be described below. FIG. 10 is an explanatory diagram showing an example of a computer that executes a program.

As shown in FIG. 10, the computer 3 includes a CPU 101 that executes various calculation processes, an input device 102 that accepts data input, a monitor 103, and a speaker 104. The computer 3 also includes a medium reading device 105 for reading programs and the like from a storage medium, an interface device 106 for connecting to various devices, and a communication device 107 for connecting to external devices by wire or wirelessly. The computer 3 also includes a RAM 108 for temporarily storing various information and a hard disk device 109. Further, each section (101 to 109) within the computer 3 is connected to a bus 110.

The hard disk device 109 executes various processes in functional units such as the difference generation unit 10, residual estimation unit 11, output unit 12, error calculation unit 20, gradient calculation unit 21, and update unit 22 described in the above embodiments. A program 111 for doing this is stored. Further, the hard disk device 109 stores various data 112 that the program 111 refers to. The input device 102 receives input of operation information from the operator of the computer 3, for example. The monitor 103 displays, for example, various screens operated by an operator. The interface device 106 is connected to, for example, a printing device. The communication device 107 is connected to a communication network such as a LAN (Local Area Network), and exchanges various information with external devices via the communication network.

The CPU 101 reads the program 111 stored in the hard disk device 109, expands it to the RAM 108, and executes it, thereby generating the difference generation unit 10, residual estimation unit 11, output unit 12, error calculation unit 20, and gradient calculation unit 21. and performs various processes related to the update unit 22 and the like. Note that the program 111 does not need to be stored in the hard disk device 109. For example, the computer 3 may read and execute the program 111 stored in a storage medium readable by the computer 3. Examples of the storage medium readable by the computer 3 include a CD-ROM, a DVD disk, a portable storage medium such as a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, a hard disk drive, and the like. Alternatively, the program 111 may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 3 may read the program 111 from there and execute it.

Regarding the above embodiments, the following additional notes are further disclosed.

(Additional Note 1) A generation unit that generates a difference image between a target image to be determined as a background and a foreground, and a background image related to the background;
an estimation unit that uses a neural network to estimate a residual indicating a difference between a map image that distinguishes the background and the foreground from the target image and the difference image;
an output unit that outputs a map image based on the generated difference image and the estimated residual;
An image processing device comprising:

(Additional Note 2) The neural network is a deep neural network with multiple intermediate layers,
The image processing device according to supplementary note 1.

(Additional Note 3) The neural network is a convolutional neural network,
The image processing device according to appendix 2, characterized in that:

(Additional Note 4) The generation unit generates the difference image based on a difference in feature amounts based on each pixel of the target image and the background image,
The image processing device according to any one of Supplementary Notes 1 to 3, characterized in that:

(Additional Note 5) Estimated by a neural network based on a difference image between a target image to be determined as a background and foreground and a background image related to the background, and a residual indicating the difference between the difference image and the difference image. a calculation unit that receives a map image that distinguishes the background and the foreground from the target image, and calculates an error between the map image and the teaching data;
an updating unit that updates parameters related to the neural network based on the calculated error;
A learning device characterized by having.

(Additional Note 6) The neural network is a deep neural network with multiple intermediate layers.
The learning device according to appendix 5, characterized in that:

(Additional Note 7) The neural network is a convolutional neural network,
The learning device according to appendix 6, characterized in that:

(Additional Note 8) Generating a difference image between a target image to be determined as a background and a foreground and a background image related to the background,
estimating a residual indicating a difference between a map image that distinguishes the background and the foreground from the target image and the difference image using a neural network;
outputting a map image based on the generated difference image and the estimated residual;
An image processing method characterized in that processing is performed by a computer.

(Additional Note 9) The neural network is a deep neural network with multiple intermediate layers,
The image processing method according to appendix 8, characterized in that:

(Additional Note 10) The neural network is a convolutional neural network,
The image processing method according to appendix 9, characterized in that:

(Additional Note 11) The generation process generates the difference image based on the difference in feature amounts based on each pixel of the target image and the background image,
The image processing method according to any one of appendices 8 to 10, characterized in that:

(Additional Note 12) Estimated by a neural network based on a difference image between a target image to be determined as a background and foreground and a background image related to the background, and a residual indicating the difference between the difference image and the difference image. Accepting a map image that distinguishes the background and the foreground from the target image, calculating an error between the map image and training data,
updating parameters related to the neural network based on the calculated error;
A learning method characterized in that processing is performed by a computer.

(Additional Note 13) The neural network is a deep neural network with multiple intermediate layers,
The learning method according to appendix 12, characterized in that:

(Additional Note 14) The neural network is a convolutional neural network,
The learning method according to appendix 13, characterized in that:

(Additional Note 15) Generating a difference image between a target image to be determined as a background and a foreground and a background image related to the background,
estimating a residual indicating a difference between a map image that distinguishes the background and the foreground from the target image and the difference image using a neural network;
outputting a map image based on the generated difference image and the estimated residual;
An image processing program that causes a computer to perform processing.

(Additional Note 16) The neural network is a deep neural network with multiple intermediate layers,
The image processing program according to appendix 15, characterized in that:

(Additional Note 17) The neural network is a convolutional neural network,
The image processing program according to appendix 16, characterized in that:

(Additional Note 18) The generation process generates the difference image based on the difference in feature amounts based on each pixel of the target image and the background image,
18. The image processing program according to any one of appendices 15 to 17.

(Additional Note 19) Estimated by a neural network based on a difference image between a target image to be determined as a background and foreground and a background image related to the background, and a residual indicating the difference between the difference image and the difference image. Accepting a map image that distinguishes the background and the foreground from the target image, calculating an error between the map image and training data,
updating parameters related to the neural network based on the calculated error;
A learning program that causes a computer to perform processing.

(Additional Note 20) The neural network is a deep neural network with multiple intermediate layers,
The learning program according to appendix 19, characterized in that:

(Additional Note 21) The neural network is a convolutional neural network,
The learning program according to appendix 20, characterized in that:

1... Image processing device 2... Learning device 3... Computer 10... Difference generation section 11... Residual estimation section 11a, 200, 200a, 200b... Neural network 12... Output section 20... Error calculation section 21... Gradient calculation section 22... Update unit 101...CPU
102...Input device 103...Monitor 104...Speaker 105...Media reading device 106...Interface device 107...Communication device 108...RAM
109...Hard disk device 110...Bus 111...Program 112...Various data 201...Input layer 202...Intermediate layer 203...Output layer C1, C2...Case G1...Target image G2...Background image G3...Difference image G4...Residual G4a...Discrimination Result G5...Foreground map G6...Teacher image G10...Input image G11...Discrimination result H...Person R1...Foreground region

Claims (3)

an acquisition unit that acquires a background image of a background photographed in advance and a target image to be determined;
a difference generation unit that generates a difference image showing a first difference between the background image and the target image;
By inputting each of the background image and the target image into a neural network, a map image to be outputted in which the background and foreground are distinguished from the target image, and a residual representing a second difference between the difference image and the difference image are generated. the neural network for estimating a difference;
an output unit that adds the difference image generated by the difference generation unit and the residual estimated by the neural network and outputs a map image in which a background and a foreground are distinguished from the target image. ,
The neural network includes a map image based on an output result output from the neural network when the background image and the target image are input to the neural network, and training data indicating correct answer data of the map image. the comparison changes parameters of the neural network;
An image processing device characterized by: Obtain a background image taken in advance and a target image to be determined,
generating a difference image showing a first difference between the background image and the target image;
By inputting each of the background image and the target image to a neural network, a map image to be output in which the background and foreground are distinguished from the target image, and a residual representing a second difference between the difference image and the difference image are generated. identifying the neural network that estimates the difference;
A computer executes a process of adding together the generated difference image and the residual estimated by the neural network to output a map image in which a background and a foreground are distinguished from the target image ,
The neural network includes a map image based on an output result output from the neural network when the background image and the target image are input to the neural network, and training data indicating correct data for the map image. the comparison changes parameters of the neural network;
An image processing method characterized by: Obtain a background image taken in advance and a target image to be determined,
generating a difference image showing a first difference between the background image and the target image;
By inputting each of the background image and the target image into a neural network, a map image to be outputted in which the background and foreground are distinguished from the target image, and a residual representing a second difference between the difference image and the difference image are generated. identifying the neural network that estimates the difference;
causing a computer to perform a process of adding together the generated difference image and the residual estimated by the neural network to output a map image in which a background and a foreground are distinguished from the target image ;
The neural network includes a map image based on an output result output from the neural network when the background image and the target image are input to the neural network, and training data indicating correct answer data of the map image. the comparison changes parameters of the neural network;
An image processing program characterized by:

Images