JP2022145825A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus for determining background/foreground robust against noise, or the like.

SOLUTION: An image processing apparatus includes, an acquisition unit, a difference generation unit, a neural network, and an output unit. The acquisition unit acquires a background image of a background taken in advance and a target image to be determined. The difference generation unit generates a difference image indicating first difference between the background image and the target image. The neural network estimates a residual indicating second difference between the difference image and a map image that distinguishes the background and the foreground from the target image and serves as an output target by inputting each of the background image and the target image into the neural network. The output unit outputs the map image that distinguishes the background and the foreground from the target image based on the generated difference image by the difference generation unit and difference estimated by the neural network.

SELECTED DRAWING: Figure 4

COPYRIGHT: (C)2023,JPO&INPIT

Description

The embodiments of the present invention relate to an image processing device, a learning device, an image processing method, a learning method, an image processing program, and a learning program.

2. Description of the Related Art Conventionally, a background subtraction method is known as a technique for detecting a moving object that appears as the foreground (hereinafter also referred to as the foreground) from a moving image captured by a camera. In the background subtraction method, a background image (also called a background model) in which the object to be detected is not captured is detected from moving images captured by a camera and stored. Then, an image area corresponding to the foreground is detected by obtaining the difference of the background image from the moving image captured by the camera.

In this way, as a method for determining background/foreground from an image, image information output from the image sensor and temporally delayed image information are taken into the input layer, and information corresponding to the difference is output from the output layer. A technique using a neural network that

JP-A-2-173877

However, in the above-described conventional technology, there is a problem that it is difficult to robustly distinguish between the background and the foreground when a similar color similar to the background is included in the foreground or against noise or the like.

For example, when the number of intermediate layers in a neural network is small, discrimination is based on local features such as edges and colors. Difficult to distinguish. In addition, it is susceptible to noise and the like, which may cause erroneous detection.

Also, when the number of intermediate layers in the neural network is increased, the connection weights are initialized with small random numbers at the beginning of learning, so the input signal spreads as it passes through the layers. , we get almost zero noise from the neural network. For this reason, meaningful information cannot be obtained even when compared with teacher data, and learning of the neural network does not progress. Therefore, it is difficult to simply increase the number of intermediate layers in the neural network.

An object of one aspect of the present invention is to provide an image processing device, a learning device, an image processing method, a learning method, an image processing program, and a learning program that enable robust background/foreground discrimination against noise and the like.

In a first scheme, the image processing device has an acquisition unit, a difference generation unit, a neural network, and an output unit. The obtaining unit obtains a background image of a background photographed in advance and a target image to be determined. The difference generator generates a difference image representing a first difference between the background image and the target image. The neural network inputs the background image and the target image respectively to the neural network to show the second difference between the map image to be output in which the background and foreground are distinguished from the target image and the difference image. Estimate the residuals. The output unit outputs a map image in which the background and foreground are distinguished from the target image based on the difference image generated by the difference generation unit and the difference estimated by the neural network.

According to one embodiment of the present invention, background/foreground discrimination robust against noise and the like can be performed.

FIG. 1 is an explanatory diagram for explaining the outline of the embodiment. FIG. 2 is an explanatory diagram illustrating examples of a background image, a target image, and a teacher image. FIG. 3 is an explanatory diagram of a conventional method of estimating a foreground map with a neural network. FIG. 4 is an explanatory diagram for explaining discrimination based on high-order features/low-order features. FIG. 5 is a block diagram of a functional configuration example of the image processing apparatus according to the embodiment; 6 is a flowchart illustrating an operation example of the image processing apparatus according to the embodiment; FIG. FIG. 7 is an explanatory diagram for explaining the neural network of the residual estimator. FIG. 8 is a block diagram of a functional configuration example of the learning device according to the embodiment; FIG. 9 is a flow chart illustrating a learning flow. FIG. 10 is an explanatory diagram of an example of a computer that executes a program.

An image processing device, a learning device, an image processing method, a learning method, an image processing program, and a learning program according to embodiments will be described below with reference to the drawings. Configurations having the same functions in the embodiments are denoted by the same reference numerals, and overlapping descriptions are omitted. Note that the image processing device, learning device, image processing method, learning method, image processing program, and learning program described in the following embodiments are merely examples, and do not limit the embodiments. Moreover, each of the following embodiments may be appropriately combined within a non-contradictory range.

FIG. 1 is an explanatory diagram for explaining the outline of the embodiment. As shown in FIG. 1, in the present embodiment, a target image G1 to be used for background/foreground discrimination and a background background image G2 captured in advance are input, and the background and Obtain a foreground map G5 that distinguishes the foreground.

FIG. 2 is an explanatory diagram illustrating examples of the target image G1, the background image G2, and the teacher image G6. As shown in FIG. 2, the target image G1 is image data for distinguishing between the foreground and the background of the person H, etc. in the shooting range. be. The background image G2 is, for example, image data obtained by photographing the background in advance. Note that the background image G2 may be a background model generated by superimposing several images.

The foreground map G5 is an example of a map image, and is image data in which, for example, the area corresponding to the background in the target image G1 is black pixels and the area corresponding to the foreground is white pixels. By multiplying the foreground map G5 thus obtained by the target image G1, for example, the foreground included in the target image G1 can be identified. For example, by using the identification result of the foreground included in the target image G1, it can be applied to extraction of a subject's silhouette in free-viewpoint video generation technology and extraction of a suspicious person in video monitoring technology.

Specifically, in the inference phase for obtaining the foreground map G5, difference generation (S1) is performed to generate the difference between the input target image G1 and the background image G2 to generate the difference image G3. Further, based on the input target image G1 and background image G2, a foreground map G5 that distinguishes the background and foreground from the target image G1 and a residual G4 that indicates the difference between the difference image G3 (the correct foreground Information missing in the map G5) is estimated using a neural network (S2). Next, a foreground map G5 is obtained by adding the difference image G3 generated in the difference generation (S1) and the residual G4 estimated in the residual estimation (S2) (S3).

In the learning phase for learning the neural network that performs the residual estimation (S2), the foreground map G5 and the teacher image G6 are compared to adjust the connection weight of each node that constitutes the neural network.

As shown in FIG. 2, the teacher image G6 is teacher data indicating the correct answer regarding the background/foreground in the target image G1 input during learning of the neural network. As an example, the teacher image G6 includes image data in which the foreground region R1 corresponding to the foreground (person H) contained in the target image G1 is white pixels, and the region (background region) other than the foreground region R1 is black pixels. .

By performing supervised learning using this teacher image G6, the neural network of the residual estimating unit 11 is trained so as to properly estimate the residual G4 of the missing part of the foreground map G5 as the correct answer. .

FIG. 3 is an explanatory diagram of a conventional method of estimating a foreground map with a neural network. As shown in FIG. 3, in the conventional method, the target image G1 and the background image G2 are input to the input layer 201 of the neural network 200, and the background/foreground discrimination result G4a is directly output from the output layer 203 via the intermediate layer 202. be done.

If the number of intermediate layers 202 in the neural network 200 is small, high-order features cannot be obtained from the input image, and low-order features, i.e., local features such as edges and colors, are used for discrimination. It will be done. Here, the high-order feature is a feature including semantic information such as whether a certain area on the image is a person or a car, or whether it is inside or outside. Low-order features are features of the local structure of the image, such as vertical and horizontal edges and flat areas.

FIG. 4 is an explanatory diagram for explaining discrimination based on high-order features/low-order features. As shown in FIG. 4, in case C1, a deep neural network (DNN) neural network 200a having multiple intermediate layers is used to obtain a determination result G11 in which the region of a person is white pixels from an input image G10. ing. In the DNN, features are gradually abstracted from the target image G1 through layers, such as edges → squares and circles (combination of edges) → tires and faces (combination of squares and circles) → . The following features can be extracted. When discrimination is performed based on such high-order features, background/foreground discrimination can be performed robustly against noise and the like.

On the other hand, in case C2, the discrimination result G11 is obtained from the input image G10 using the three-layer neural network 200b. Since the three-layer neural network 200b performs discrimination based on local low-order features, it is susceptible to noise and the like, resulting in erroneous detection. For example, the input image G10 is an image of a person playing across the line of the court, and in the area on the right side of the line, the person includes a similar color similar to the color of the court. Also, in the area on the left side of the line, the color of the coat and the color of the person (left foot) are not similar. Therefore, in the determination result G11 of case C2, the portion where the foreground includes a similar color similar to the background is not determined as the foreground (part of the left foot is correctly determined as the foreground).

Returning to FIG. 3, when increasing the number of intermediate layers 202 in the neural network 200, since the connection weights are initialized with small random numbers at the beginning of learning, the input signal from the input layer 201 ( The target image G1 and the background image G2) are diffused and weakened as they pass through the layers. Therefore, the discrimination result G4a obtained from the output layer 203 has almost zero noise at the initial stage of learning. Therefore, meaningful information cannot be obtained by comparison with the teacher image G6 (because the direction of the gradient is not determined), and it becomes difficult to adjust the parameter indicating the connection weight of each node in the neural network 200. will not proceed.

On the other hand, in the present embodiment, as shown in FIG. 1, a residual G4 indicating the difference between the foreground map G5 that distinguishes the background and foreground from the target image G1 and the difference image G3 (the difference image G3 (Information lacking in the foreground map G5) is estimated using a neural network. A foreground map G5 is obtained by adding the difference image G3 and the residual G4.

Therefore, for example, even if the residual G4 obtained at the beginning of learning is 0, since the difference image G3 is added, a meaningful output (foreground map G5) can be obtained and compared with the teacher image G6. can advance the learning of the neural network. Therefore, it is possible to obtain the foreground map G5 from the residual G4 discriminated based on the high-order features using the DNN, and robustly discriminate the background/foreground even when the background similar color or noise is included. be able to.

FIG. 5 is a block diagram of a functional configuration example of the image processing apparatus according to the embodiment; As shown in FIG. 5 , the image processing device 1 has a difference generator 10 , a residual estimator 11 and an output unit 12 .

The difference generation unit 10 generates a difference image G3 from the difference between the input target image G1 and the background image G2. That is, the difference generator 10 is an example of a generator. For example, the difference generation unit 10 generates the difference image G3 by calculating the difference in the pixel values of the pixels corresponding to each other in the target image G1 and the background image G2. As for this difference, the L1 norm (absolute value of the difference) or the L2 norm of the pixel value can be used.

Further, the targets for which the difference generation unit 10 calculates the difference are not limited to pixel values of pixels corresponding to each other. For example, the difference generation unit 10 may obtain the difference in feature amount calculated from each pixel of the target image G1 and the background image G2 to generate the difference image G3.

As an example, a local feature amount may be used as shown in the following document.
・SIFT (Scale-Invariant Feature Transform): David G.Lowe, “Distinctive image features from scale-invariant keypoints”, Int.Journal of Computer Vision, Vol.60, No.2, pp.91-110, 2004.
・SURF (Speeded-Up Robust Features): H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded-Up Robust Features”, In ECCV , pp.404-417, 2006.
・BRIEF (Features from Accelerated Segment Test): M.Calonder, V.Lepetit and C.Strecha and P.Fua, “BRIEF: Binary Robust Independent Elementary Features”, In Proc. European Conference on Computer Vision, pp.778-792 , 2010.
・ORB (Oriented FAST and Rotated Brief): E.Rublee, V.Rabaud, K.Konolige and G.Bradski “ORB: an efficient alternative to SIFT or SURF”, In Proc. International Conference on Computer Vision, 2011.

In addition, the difference generation unit 10 may use the value of the intermediate layer of the DNN that has been trained for general object recognition as a feature amount, as described in the following document.
・AlexNet: Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
・GoogLeNet: Szegedy, Christian, et al. "Going deeper with convolutions." Cvpr, 2015.
・VGG: Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
・ResNet: He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

The residual estimating unit 11 uses a neural network to generate a residual G4 indicating the difference between a foreground map G5 that distinguishes the background and foreground from the target image G1 and a difference image G3 from the input target image G1 and background image G2. estimated using That is, the residual estimator 11 is an example of an estimator.

The output unit 12 outputs a foreground map G5 based on the difference image G3 generated by the difference generation unit 10 and the residual G4 estimated by the residual estimation unit 11 . Specifically, the output unit 12 outputs the foreground map G5 obtained by adding the pixel values of the pixels corresponding to each other in the difference image G3 and the residual G4. This addition may be weighted addition. This weight may be a value predetermined by the designer, or may be a variable parameter. Variable parameters may be optimized together when the neural network of the residual estimator 11 is trained.

FIG. 6 is a flow chart showing an operation example of the image processing apparatus 1 according to the embodiment. As shown in FIG. 6, when the process is started, the image processing apparatus 1 acquires a background image G2 stored in advance in a memory, hard disk, database, network storage, or the like (S10). Similarly, the image processing device 1 acquires the target image G1 (S11).

When an image from a surveillance camera is used as the target image G1, the image processing apparatus 1 may directly acquire the target image G1 from the surveillance camera via an interface connected to the surveillance camera. Further, the image processing apparatus 1 may perform preprocessing such as noise removal and color correction when acquiring the target image G1 and the background image G2.

Next, the image processing device 1 inputs the target image G1 and the background image G2 to the difference generator 10 (S12, S13). Next, the difference generator 10 generates the difference between the target image G1 and the background image G2 (S14) to obtain the difference image G3.

Next, the image processing device 1 inputs the target image G1 and the background image G2 to the residual estimator 11 (S15, S16). Next, the residual estimator 11 estimates a residual G4 representing the difference between the foreground map G5 and the difference image G3 from the inputted target image G1 and background image G2 using a neural network (S17).

The neural network of the residual estimator 11 is subjected to parameter adjustment so as to properly estimate the residual G4 in a learning phase by a learning device, which will be described later.

FIG. 7 is an explanatory diagram for explaining the neural network of the residual estimator 11. As shown in FIG. As shown in FIG. 11, the neural network 11a has a network structure in which units resembling neurons in the brain are hierarchically connected. A large number of neurons (nerve cells) exist in the brain. Each neuron receives signals from other neurons and passes signals to other neurons. The brain performs various information processing according to this signal flow. The neural network 11a is a model that realizes such characteristics of brain functions on a computer.

Specifically, the neural network 11a may be a deep neural network having multiple intermediate layers from a layer to which the target image G1 and the background image G2 are input to a layer to output the residual G4. The multiple hidden layers include, for example, convolution layers, activation function layers, pooling layers, fully connected layers and softmax layers. The number and position of each layer may be changed at any time according to the required architecture. That is, the hierarchical structure and the configuration of each layer of the neural network 11a can be determined in advance by the designer according to the object to be identified.

In addition, the neural network 11a has a configuration as a CNN (convolutional neural network) in which convolutional layers and pooling layers are alternately stacked so as to enable feature extraction from input image data. may Also, the neural network 11a may be configured by arranging fully connected layers in multiple layers instead of CNN. In this case, for the target image G1 and the background image G2, the input may be vectors arranged in a line according to a specific method such as raster scan order.

For example, neural network 11a may use a network structure as shown in the following document.
・FCN (Fully Convolutional Networks): Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
・U-Net: Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

A neural network 11a in FIG. 7 is an example of a network structure when the above U-Net is applied. For example, in the neural network 11a, when the target image G1 and the background image G2 are RGB 3-channel color images, the input is 6 channels in which they are superimposed. This input is passed through a convolutional layer, an augmented convolutional layer, a pooling layer, a batch normalization layer, an activation function layer, etc., and a residual error G4 of one channel is output.

Returning to FIG. 6, after S17, the output unit 12 adds the difference image G3 and the residual G4 (S18), and outputs the foreground map G5 (S19).

In the image processing apparatus 1, the obtained foreground map G5 can be applied to extraction of the silhouette of a subject in the free-viewpoint video generation technology and extraction of a suspicious person in the video surveillance technology.

For example, a technique for creating a video of an arbitrary viewpoint from camera videos of multiple viewpoints is called free-viewpoint video generation. By using this free-viewpoint video generation technology, it is possible to generate video from a viewpoint other than the camera that shot it, or to create video with camerawork that is impossible in reality. Users can apply it for viewing from their favorite angles.

Examples of application to this free-viewpoint video generation technology include the following. A foreground map G5 is obtained for each camera image, and a silhouette of a foreground object such as a person is extracted based on the obtained foreground map G5. Next, a technique called Visual Hull is used to restore the three-dimensional structure of the foreground object and render an image from an arbitrarily set viewpoint.

In addition, there are the following examples of application of video surveillance technology to extraction of suspicious persons. A foreground map G5 is obtained for the surveillance camera image, and the number of pixels in the foreground area is obtained. If the number of pixels in the foreground area is greater than or equal to a predetermined threshold value, it is determined that a suspicious object has been detected, and a detection signal is output.

Next, the details of the learning device that performs learning (learning phase) of the neural network 11a will be described. FIG. 8 is a block diagram of a functional configuration example of the learning device according to the embodiment;

For the target image G1, the background image G2, and the teacher image G6 in the learning phase, data of a learning data set preset for learning are used. This learning data set is read and used, for example, from a memory, hard disk, database, network storage, or the like. Further, the learning data set may be subjected to a process (data augmentation) to artificially increase the diversity of the data, such as geometric transformation such as image rotation or scaling, or addition of noise. In the case of mini-batch learning, the target image G1, the background image G2, and the teacher image G6 may be acquired for the number of mini-batches.

As shown in FIG. 8 , the learning device 2 has an error calculator 20 , a gradient calculator 21 and an updater 22 .

The error calculator 20 receives inputs of the foreground map G5 output by the image processing device 1 and the teacher image G6. The error calculator 20 compares the input foreground map G5 and the teacher image G6 to calculate an error. That is, the error calculator 20 is an example of a calculator.

For example, the error calculator 20 calculates the error by obtaining the squared error of the pixel values of corresponding pixels in the foreground map G5 and the teacher image G6. This error is not limited to the square error, and may be an L1 norm error, a Huber norm error used in robust statistics, or the like.

Based on the error calculated by the error calculator 20, the gradient calculator 21 calculates the gradient of the entire neural network 11a based on the error backpropagation method generally used in supervised learning.

Based on the gradients calculated by the gradient calculator 21, the updater 22 calculates the update amount of the connection weights (parameters) of the nodes forming the neural network 11a. The update unit 22 updates parameters in the neural network 11a based on the calculated update amount.

For calculation of the amount of update in the updating unit 22, for example, an optimization method as described in the following document can be used.
・SGD (stocastic gradient descent) with momentum: Goodfellow, Ian, et al. Deep learning. Vol. 1. Cambridge: MIT press, 2016.
・RMSProp: Geoffrey Hinton, Nitish Srivastava, Kevin Swersky. 2014. Lecture 6e: Rmsprop: Divide the gradient by a running average of its recent magnitude (CSC321 Winter 2014).
・Adam: Diederik Kingma, Jimmy Ba. 2015. Adam: a method for stochastic optimization. the 3rd International Conference for Learning Representations (ICLR 2015).

FIG. 9 is a flow chart illustrating a learning flow. As shown in FIG. 9, when the process is started, the target image G1 and the background image G2 are acquired from the learning data set (S20, S21). Next, the image processing apparatus 1 performs the same processing as S12-S19, and outputs the foreground map G5 (S22-S29). The foreground map G5 output from the image processing device 1 is input to the error calculator 20 of the learning device 2 .

Next, the error calculator 20 of the learning device 2 acquires the teacher image G6 from the learning data set (S30). Next, the error calculator 20 compares the foreground map G5 output from the image processing device 1 and the teacher image G6 to calculate the error (S31).

Next, the gradient calculator 21 calculates the gradient of the entire neural network 11a based on the error calculated by the error calculator 20 (S32). Next, the update unit 22 updates the connection weight of each node in the neural network 11a of the difference generation unit 10 according to the gradient calculated by the gradient calculation unit 21 (S33).

Next, the learning device 2 determines whether a predetermined learning end condition is satisfied, such as whether or not learning using all the data included in the learning data set has been completed (S34). If not satisfied (S34: NO), the learning device 2 returns to S20 and continues learning. If the condition is satisfied (S34: YES), the learning device 2 ends learning.

As described above, the difference generation unit 10 of the image processing apparatus 1 generates the difference image G3 between the target image G1, which is a target for discrimination between the background and the foreground, and the background image G2 that is the background. The residual estimating unit 11 of the image processing apparatus 1 generates a residual G4 indicating the difference between the input target image G1 and background image G2, and the foreground map G5 that distinguishes the background and foreground from the target image G1, and the difference image G3. is estimated using the neural network 11a. The output unit 12 of the image processing device 1 outputs a foreground map G5 based on the difference image G3 generated by the difference generation unit 10 and the residual G4 estimated by the residual estimation unit 11 .

As a result, the image processing apparatus 1 can use the multi-layered residual estimating unit 11 that discriminates high-order features, and can robustly deal with noise and the like when the foreground includes similar colors similar to the background. Background/foreground discrimination can be performed.

The neural network 11a of the residual estimator 11 is a deep neural network (DNN) with multiple intermediate layers. Thereby, the residual estimator 11 can estimate the residual G4 based on the high-order features included in the input target image G1 and background image G2. Therefore, the image processing apparatus 1 can perform more robust background/foreground discrimination against noise and the like.

Also, the neural network 11a of the residual estimator 11 is a convolutional neural network. As a result, the residual estimating unit 11 can abstract the input target image G1 and background image G2 and obtain high-order features.

In addition, the difference generation unit 10 generates a difference image G3 based on the difference in feature amount based on each pixel of the target image G1 and the background image G2. In this way, the difference image G3 may be the difference in feature amount between the target image G1 and the background image G2.

Further, the error calculation unit 20 of the learning device 2 generates a foreground map G5 based on the difference image G3 generated by the difference generation unit 10 and the residual G4 estimated by the neural network 11a of the residual estimation unit 11. accept. The learning device 2 calculates the error between the received foreground map G5 and the teacher image G6. The updating unit 22 of the learning device 2 updates the parameters of the neural network 11a based on the calculated error. Thus, the learning device 2 can learn the neural network 11a for estimating the residual G4 indicating the difference between the foreground map G5 that distinguishes the background and foreground from the target image G1 and the difference image G3.

It should be noted that each component of each illustrated device does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Various processing functions performed by the image processing device 1 and the learning device 2 may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). good. Also, various processing functions may be executed in whole or in part on a program analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. It goes without saying that it is good. Further, various processing functions performed by the image processing device 1 and the learning device 2 may be performed in cooperation with a plurality of computers by cloud computing.

By the way, the various processes described in the above embodiments can be realized by executing a prepared program on a computer. Therefore, an example of a computer (hardware) that executes a program having functions similar to those of the above embodiments will be described below. FIG. 10 is an explanatory diagram of an example of a computer that executes a program.

As shown in FIG. 10, the computer 3 has a CPU 101 that executes various arithmetic processes, an input device 102 that receives data input, a monitor 103 and a speaker 104 . The computer 3 also has a medium reading device 105 for reading programs and the like from a storage medium, an interface device 106 for connecting with various devices, and a communication device 107 for communicating with external devices by wire or wirelessly. The computer 3 also has a RAM 108 that temporarily stores various information, and a hard disk device 109 . Also, each unit (101 to 109) in the computer 3 is connected to the bus 110. FIG.

The hard disk device 109 executes various processes in the functional units such as the difference generation unit 10, the residual estimation unit 11, the output unit 12, the error calculation unit 20, the gradient calculation unit 21, and the update unit 22 described in the above embodiments. A program 111 for doing is stored. Various data 112 referred to by the program 111 are stored in the hard disk device 109 . The input device 102 receives input of operation information from an operator of the computer 3, for example. The monitor 103 displays, for example, various screens operated by an operator. The interface device 106 is connected to, for example, a printing device. The communication device 107 is connected to a communication network such as a LAN (Local Area Network), and exchanges various information with external devices via the communication network.

The CPU 101 reads out the program 111 stored in the hard disk device 109, develops it in the RAM 108, and executes it to obtain the difference generator 10, the residual estimator 11, the output unit 12, the error calculator 20, and the gradient calculator 21. and various processing related to the update unit 22 and the like. Note that the program 111 does not have to be stored in the hard disk device 109 . For example, the computer 3 may read and execute the program 111 stored in a storage medium readable by the computer 3 . Examples of storage media readable by the computer 3 include portable recording media such as CD-ROMs, DVD discs, USB (Universal Serial Bus) memories, semiconductor memories such as flash memories, and hard disk drives. Alternatively, the program 111 may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 3 may read and execute the program 111 therefrom.

Further, the following additional remarks are disclosed with respect to the above embodiment.

(Appendix 1) a generation unit that generates a difference image between a target image that is a background and foreground determination target and a background image that is related to the background;
an estimating unit that uses a neural network to estimate a residual indicating a difference between a map image that distinguishes the background and the foreground from the target image and the difference image;
an output unit that outputs a map image based on the generated difference image and the estimated residual;
An image processing device comprising:

(Appendix 2) The neural network is a deep neural network with multiple intermediate layers,
The image processing apparatus according to Supplementary Note 1, characterized by:

(Appendix 3) The neural network is a convolutional neural network.
The image processing apparatus according to appendix 2, characterized by:

(Additional remark 4) The generation unit generates the difference image based on the difference in feature amount based on each pixel of the target image and the background image.
The image processing apparatus according to any one of Appendices 1 to 3, characterized by:

(Appendix 5) Based on a difference image between a target image for background and foreground discrimination and a background image on the background, and a residual indicating the difference between the difference image and the difference image, estimated by a neural network, a calculation unit that receives a map image that distinguishes the background and the foreground from the target image, and calculates an error between the map image and teacher data;
an updating unit that updates parameters for the neural network based on the calculated error;
A learning device characterized by comprising:

(Appendix 6) The neural network is a deep neural network with multiple intermediate layers,
The learning device according to appendix 5, characterized by:

(Appendix 7) The neural network is a convolutional neural network.
The learning device according to appendix 6, characterized by:

(Appendix 8) generating a difference image between a target image to be used for background and foreground discrimination and a background image for the background;
estimating a residual indicating a difference between a map image that distinguishes the background and the foreground from the target image and the difference image using a neural network;
outputting a map image based on the generated difference image and the estimated residual;
An image processing method characterized in that the processing is executed by a computer.

(Appendix 9) The neural network is a deep neural network with multiple intermediate layers,
The image processing method according to appendix 8, characterized by:

(Appendix 10) The neural network is a convolutional neural network.
The image processing method according to appendix 9, characterized by:

(Supplementary Note 11) The generating process generates the difference image based on a difference in feature amount based on each pixel of the target image and the background image.
11. The image processing method according to any one of appendices 8 to 10, characterized by:

(Appendix 12) Based on a difference image between a target image for background and foreground discrimination and a background image on the background, and a residual indicating the difference between the difference image and the difference image, estimated by a neural network, receiving a map image that distinguishes the background and the foreground from the target image, calculating an error between the map image and teacher data;
updating parameters for the neural network based on the calculated error;
A learning method characterized in that processing is executed by a computer.

(Appendix 13) The neural network is a deep neural network with multiple intermediate layers,
The learning method according to appendix 12, characterized by:

(Appendix 14) The neural network is a convolutional neural network.
The learning method according to appendix 13, characterized by:

(Appendix 15) generating a difference image between a target image to be used for background and foreground discrimination and a background image for the background;
estimating a residual indicating a difference between a map image that distinguishes the background and the foreground from the target image and the difference image using a neural network;
outputting a map image based on the generated difference image and the estimated residual;
An image processing program that causes a computer to execute processing.

(Appendix 16) The neural network is a deep neural network with multiple intermediate layers,
16. The image processing program according to appendix 15, characterized by:

(Appendix 17) The neural network is a convolutional neural network.
17. The image processing program according to appendix 16, characterized by:

(Appendix 18) In the generating process, the difference image is generated based on a difference in feature amount based on each pixel of the target image and the background image.
18. The image processing program according to any one of appendices 15 to 17, characterized by:

(Appendix 19) Based on the difference image between the target image for background and foreground discrimination and the background image on the background, and the residual indicating the difference between the difference image and the difference image, estimated by a neural network, receiving a map image that distinguishes the background and the foreground from the target image, calculating an error between the map image and teacher data;
updating parameters for the neural network based on the calculated error;
A learning program characterized by causing a computer to execute processing.

(Appendix 20) The neural network is a deep neural network with multiple intermediate layers,
The learning program according to appendix 19, characterized by:

(Appendix 21) The neural network is a convolutional neural network.
The learning program according to appendix 20, characterized by:

Reference Signs List 1... Image processing device 2... Learning device 3... Computer 10... Difference generation unit 11... Residual error estimation unit 11a, 200, 200a, 200b... Neural network 12... Output unit 20... Error calculation unit 21... Gradient calculation unit 22... Update unit 101 ... CPU
REFERENCE SIGNS LIST 102: Input device 103: Monitor 104: Speaker 105: Medium reading device 106: Interface device 107: Communication device 108: RAM
109 Hard disk device 110 Bus 111 Program 112 Various data 201 Input layer 202 Intermediate layer 203 Output layers C1, C2 Case G1 Target image G2 Background image G3 Difference image G4 Residual G4a Discrimination Result G5... Foreground map G6... Teacher image G10... Input image G11... Discrimination result H... Person R1... Foreground area

Claims (7)

an acquisition unit that acquires a background image of a background captured in advance and a target image to be determined;
a difference generation unit that generates a difference image indicating a first difference between the background image and the target image;
By inputting each of the background image and the target image to a neural network, a map image to be output in which the background and the foreground are distinguished from the target image, and a residual representing a second difference between the difference image and the map image. the neural network for estimating the difference;
an output unit that outputs a map image in which a background and a foreground are distinguished from the target image based on the difference image generated by the difference generation unit and the residual estimated by the neural network;
An image processing device comprising: The output unit outputs the map image obtained by adding pixel values of pixels corresponding to each other in the difference image and the residual.
2. The image processing apparatus according to claim 1, wherein: The neural network, based on the output result output from the neural network when the background image and the target image are input to the neural network, and teacher data indicating correct data of the map image, A neural network in which a parameter of the neural network is changed;
2. The image processing apparatus according to claim 1, wherein: Acquiring a background image of a background photographed in advance and a target image to be determined,
generating a difference image indicating a first difference between the background image and the target image;
By inputting each of the background image and the target image to a neural network, a map image to be output in which the background and the foreground are distinguished from the target image, and a residual representing a second difference between the difference image and the map image. identifying the neural network for estimating the difference;
outputting a map image in which a background and a foreground are distinguished from the target image based on the difference image generated by the difference generation unit and the residual estimated by the neural network;
An image processing method characterized in that the processing is executed by a computer. Acquiring a background image of a background photographed in advance and a target image to be determined,
generating a difference image indicating a first difference between the background image and the target image;
By inputting each of the background image and the target image to a neural network, a map image to be output in which the background and the foreground are distinguished from the target image, and a residual representing a second difference between the difference image and the map image. identifying the neural network for estimating the difference;
outputting a map image in which a background and a foreground are distinguished from the target image based on the generated difference image and the residual estimated by the neural network;
An image processing program that causes a computer to execute processing. outputting the map image obtained by adding pixel values of pixels corresponding to each other in the difference image and the residual;
6. The image processing program according to claim 5, characterized by: The neural network, based on the output result output from the neural network when the background image and the target image are input to the neural network, and teacher data indicating correct data of the map image, A neural network in which a parameter of the neural network is changed;
6. The image processing program according to claim 5, characterized by:

Images