JP2020112928A - Background model generation device, background model generation method, and background model generation program - Google Patents

Abstract

To reduce divergence of a depth in a background model and a dynamic background.SOLUTION: A server device 10 has an acquisition part acquiring a camera image imaged by a camera from a prescribed imaging position, a calculation part calculating a depth image corresponding to the imaging position, a separation part separating a plurality of pixels included in the camera image into a foreground and a background, a correction part correcting a depth value of each pixel in the depth image by using a depth value of each pixel in the depth image corresponding to each pixel that is separated into the background in the camera image, and a background creation part creating a background model regarding the background by using a new depth image based on a depth value of each pixel in the depth image that the correction part has corrected.SELECTED DRAWING: Figure 6

Description

The present invention relates to a background model generation device, a background model generation method, and a background model generation program.

A technique called free-viewpoint image generation is known. For example, when a free viewpoint video is generated, the foreground and the background are separated from each of the images captured for each of a plurality of viewpoints, and then the three-dimensional model is reproduced for each of the foreground portion and the background portion. In this way, the 3D space reproduced by the 3D model of the foreground part and the 3D model of the background part is provided as an image viewed from the designated virtual viewpoint.

Of these three-dimensional models, Visual Hull is used to generate the three-dimensional model of the foreground part. On the other hand, the three-dimensional model of the background portion is generated in advance using computer graphics, three-dimensional distance measurement, or the like. Then, at the time of rendering, of the camera images captured from a plurality of viewpoints, the camera image corresponding to the designated virtual viewpoint is projected onto the three-dimensional model of the foreground portion and the three-dimensional model of the background portion. The virtual viewpoint is not limited to the viewpoint of the camera, and any arbitrary viewpoint in the three-dimensional space can be designated.

Here, the three-dimensional background model prepared in advance is nothing but a model of a stationary subject among the subjects included in the background, for example, a structure such as a stadium where a sports watching is held, and facilities such as its seats. Absent. Therefore, when a dynamic background such as a spectator watching a sport in the spectator seat is included, the image quality of the free viewpoint image is deteriorated.

This is because when a dynamic background is included, a gap occurs between the depth from the virtual viewpoint to the background model and the depth from the virtual viewpoint to the dynamic background. This displacement of depth contributes to the mapping of pixels with incorrect texture coordinates in the camera image used as the texture, resulting in deterioration of the image quality of the free viewpoint image.

From the aspect corresponding to such a dynamic background, the following free-viewpoint image generation device has been proposed. The free-viewpoint video generation device first generates a temporary free-viewpoint image of each frame from the reference image and the depth map. Then, the free viewpoint video generation device extracts the background image and its depth value to be stored in the curved background buffer from the reference image and the depth map as the background area. Then, the free viewpoint video generation device complements the temporary free viewpoint image with the background image stored in the curved background buffer and its depth value.

JP, 2006-146846, A

However, the above technique may still fail to reduce the depth shift of the background model and the dynamic background.

That is, in the above free-viewpoint image generation device, the background area is extracted by using the depth corresponding to the minimum value in the Gaussian distribution in which the depth distribution of the reference image is smoothed for dividing the foreground and the background. However, when the depth value is used to extract the background region, it becomes difficult to divide the foreground and the background as the depth values become closer to each other. Therefore, in the curved background buffer, the subject corresponding to the foreground is erroneously stored as the background, resulting in an increase in the depth shift.

In one aspect, an object of the present invention is to provide a background model generation device, a background model generation method, and a background model generation program capable of reducing the depth shift of the background model and the dynamic background.

In one aspect, the background model generation device is included in the camera image, an acquisition unit that acquires a camera image captured by a camera from a predetermined imaging position, a calculation unit that calculates a depth image corresponding to the imaging position. A separation unit that separates a plurality of pixels into a foreground and a background, and a depth value of each pixel of the depth image using a depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image. A correction unit that corrects a value, and a background generation unit that generates a background model related to the background by using a new depth image based on the depth value of each pixel of the depth image corrected by the correction unit. ..

The depth shift of the background model and the dynamic background can be reduced.

FIG. 1 is a diagram illustrating a configuration example of a video generation system according to the first embodiment. FIG. 2A is a diagram showing an example of a camera image. FIG. 2B is a diagram showing an example of a silhouette image. FIG. 3 is a diagram illustrating an example of the Visual Hull. FIG. 4 is a diagram showing an example of rendering. FIG. 5: is a figure which shows an example of the cross section of a stadium. FIG. 6 is a block diagram illustrating the functional configuration of the server device according to the first embodiment. FIG. 7 is a diagram illustrating an example of data transmitted and received between the functional units according to the first embodiment. FIG. 8A is a diagram showing an example of a silhouette image. FIG. 8B is a diagram showing an example of the depth image. FIG. 9A is a diagram showing an example of the image ID. FIG. 9B is a diagram showing an example of the convolution operation of the filter. FIG. 9C is a diagram showing an example of the convolution operation of the filter. FIG. 10A is a diagram showing an example of a pixel of interest in temporal filtering. FIG. 10B is a diagram illustrating an example of the convolution operation of the filter. FIG. 11 is a flowchart illustrating the procedure of the video generation process according to the first embodiment. FIG. 12 is a diagram illustrating an example of data exchanged between the functional units in the application example 1. FIG. 13 is a diagram showing an example of a graph of the evaluation value and the depth. FIG. 14 is a flowchart illustrating the procedure of the video generation process according to the application example 1. FIG. 15 is a diagram illustrating a hardware configuration example of a computer that executes the background model generation program according to the first and second embodiments.

A background model generation device, a background model generation method, and a background model generation program according to the present application will be described below with reference to the accompanying drawings. Note that this embodiment does not limit the disclosed technology. Then, the respective embodiments can be appropriately combined within the range in which the processing contents do not contradict each other.

[System configuration]
FIG. 1 is a diagram illustrating a configuration example of a video generation system according to the first embodiment. As one aspect, the video generation system 1 illustrated in FIG. 1 provides a video generation service that generates a free-viewpoint video by combining multi-viewpoint camera images captured by a plurality of cameras 5A to 5N having different viewpoints. ..

As shown in FIG. 1, the video generation system 1 includes cameras 5A to 5N, a server device 10, and a client terminal 30. Hereinafter, the cameras 5A to 5N may be referred to as "camera 5". Although only one client terminal 30 is shown in FIG. 1 as an example, any number of client terminals 30 may be included in the video generation system 1.

The server device 10 and the client terminal 30 are connected via a predetermined network NW. For example, the network NW can be constructed by any type of communication network such as the Internet, LAN (Local Area Network) or VPN (Virtual Private Network) regardless of wired or wireless. As an example, FIG. 1 illustrates a case where the free-viewpoint video is provided via the network NW, but this is merely an example of a video providing form, and the free-viewpoint video is provided between the server device 10 and the client terminal 30. It does not matter that bidirectional communication is not necessarily performed. For example, the free viewpoint video may be provided to the client terminal 30 via a broadcast wave without passing through the network NW.

The camera 5 is an imaging device equipped with an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

For example, by combining the shooting ranges of the plurality of cameras 5, each camera 5 is installed in such an arrangement that the entire area of the three-dimensional space in which the free-viewpoint image is generated falls within the shooting range of the plurality of cameras 5. Further, in order to calculate the three-dimensional shape of the subject 3 existing in the three-dimensional space from the camera images picked up by two or more cameras 5, each camera 5 has a shooting range between the other cameras 5. They are arranged in a partially overlapping state. Under such an arrangement, the plurality of cameras 5 shoot in synchronization with each frame, so that a plurality of images taken at different timings at the same timing, that is, multi-view camera images are obtained in frame units. ..

The server device 10 corresponds to an example of a computer that provides the above-described video generation service. The server device 10 also corresponds to an example of a correction device. Here, the server device is given as an example of the computer to the last, but this is a label given for classifying the functions, and its hardware configuration and the type of software to be introduced are not limited, and are arbitrary. It can be any type of computer.

As an embodiment, the server device 10 can be implemented as package software or online software by installing a video processing program that realizes a function corresponding to the above video generation service in a desired computer. For example, the server device 10 may be mounted on-premises as a server that provides the above-described video generation service, or may be implemented as a cloud that provides the above-mentioned video generation service by outsourcing.

The client terminal 30 corresponds to an example of a computer that receives the above-described video generation service.

As one embodiment, the client terminal 30 corresponds to any computer used by a user who receives the above-described video generation service. For example, the client terminal 30 corresponds to a desktop computer such as a personal computer or a workstation. The computer is not limited to such a desktop computer, and may be any computer such as a rack-top computer, a mobile terminal device, and a wearable terminal.

[Video generation]
The above free viewpoint video is generated by executing four processes of (1) foreground/background separation, (2) foreground model generation, (3) background model generation, and (4) rendering, as one aspect. ..

(1) Foreground/background separation The above-mentioned “foreground/background separation” refers to a process of separating the foreground and the background from the camera image for each camera image corresponding to each viewpoint. This foreground/background separation can be executed in parallel for each camera image of the same frame, or can be executed in order by a predetermined number.

The “foreground” referred to here corresponds to a subject of interest in photographing among objects existing in a three-dimensional space within the photographing range of the camera 5. For example, in watching sports, subjects such as players and balls correspond to the foreground. Further, in the case of motor sports, vehicles used for athletes, such as automobiles and motorcycles, are included in the foreground category.

On the other hand, the “background” corresponds to the subject existing behind the subject corresponding to the foreground. For example, the subjects corresponding to the background include those whose position or orientation does not change. Hereinafter, a subject whose position or orientation does not change among the backgrounds may be referred to as a “static background”. For example, taking a sports watching as an example, a static background includes a structure such as a stadium where the sports are watched and facilities such as seats of the spectators. In addition to such a static background, the background may include a background whose position or orientation changes. Hereinafter, a subject whose position or orientation changes among the backgrounds may be referred to as a “dynamic background”. For example, a spectator watching a spectator seat in a stadium corresponds to the dynamic background. Because the spectator's position and posture change as the spectator sits in the spectator's seat to watch the game, or leans forward from the spectator's seat to watch the game, or stands up and watches the game. Is.

An example of foreground/background separation will be described with reference to FIGS. 2A and 2B. FIG. 2A is a diagram showing an example of a camera image. FIG. 2B is a diagram showing an example of a silhouette image. 2A shows a camera image 200 corresponding to a certain viewpoint p1, and FIG. 2B shows a silhouette image 210 generated from the camera image 200 of the viewpoint p1. For foreground/background separation, as an example, so-called background difference is applied to the camera image 200, or a two-dimensional graph cut is applied to the camera image 200. By applying any algorithm including these background differences and graph cuts, a silhouette image 210 in which each pixel is assigned a binary label of the foreground or background is generated from the camera image 200 in which each pixel has a pixel value. It In this silhouette image 210, as shown in FIG. 2B, the silhouette of the subject 3fg included in the camera image 200 is extracted after being separated from the background.

(2) Generation of Foreground Model In the above “generation of foreground model”, for example, a technique called Visual-Hull is used. For example, in Visual Hull, a Cone (visual volume) formed by connecting the optical center of the camera 5 and the silhouette on the silhouette image is generated, and the area in the three-dimensional space where the Cone overlaps is the three-dimensional object 3fg. It is calculated as a shape.

FIG. 3 is a diagram illustrating an example of the Visual Hull. FIG. 3 shows an example in which silhouette images 210A to 210C of the three cameras 5A to 5C are used to calculate the Visual Hull. As shown in FIG. 3, the silhouettes SA to SC are projected in the three-dimensional space for each silhouette image 210A to 210C corresponding to each viewpoint of the cameras 5A to 5C. For example, when the silhouette SA is projected, a visual volume CA connecting the optical center of the camera 5A and the silhouette SA on the silhouette image 210A is obtained. Further, when the silhouette SB is projected, a visual volume CB connecting the optical center of the camera 5B and the silhouette SB on the silhouette image 210B is obtained. Furthermore, when the silhouette SC is projected, a visual volume CC connecting the optical center of the camera 5C and the silhouette SC on the silhouette image 210C is obtained. A Visual Hull region in which these visual volumes CA to CC overlap, that is, a three-dimensional model with black filling shown in FIG. 3 is calculated as the three-dimensional shape of the subject 3.

(3) Generation of Background Model For the above-mentioned “generation of background model”, computer graphics, three-dimensional distance measurement, etc. are used as an example. For example, a background model is generated by modeling a static background by 3DCG (3 Dimensional Computer Graphics). In addition, the depth corresponding to each pixel on the camera image 200 is measured by the three-dimensional laser sensor for each viewpoint corresponding to each camera 5. As a result, a depth image in which the depth is associated with each pixel is obtained for each viewpoint of the camera 5.

(4) Rendering The above-mentioned “rendering” refers to a process of generating a camera image corresponding to a virtual viewpoint, that is, a so-called free viewpoint video, using camera images of multiple viewpoints. The “virtual viewpoint” mentioned here refers to a viewpoint given to the virtual camera, for example, a position or orientation in which the virtual camera is arranged in a three-dimensional space. This virtual viewpoint may be designated by accepting a user input from the client terminal 30, or may be designated by user setting via the client terminal 30 or system setting registered in the server device 10. I don't care.

FIG. 4 is a diagram showing an example of rendering. FIG. 4 shows an example in which the position of the virtual camera Vc is set between the cameras 5B and 5C, and a straight line passing through the optical center of the virtual camera Vc and the pixel intersects with the foreground model 3Mfg corresponding to the subject 3fg. ing. As shown in FIG. 4, the three-dimensional position of the intersection of the straight line passing through the optical center of the virtual camera Vc and the pixel and the foreground model 3Mfg is obtained (S1). Then, according to the camera parameters in which the external parameters such as the position and orientation of the camera 5 and the internal parameters such as the angle of view of the camera 5 and the distortion of the lens are set, the above intersections are projected on the camera image corresponding to each viewpoint. It Here, as an example, the intersections are projected onto camera images of a predetermined number of cameras 5 close to the virtual camera Vc, that is, two camera images 200B and 200C of the cameras 5B and C (S2B and S2C). .. Thereby, the pixel of the camera 5B and the pixel of the camera 5C corresponding to the pixel of the virtual camera Vc are identified as the texture coordinates.

After that, in the camera image 200B captured by the camera 5B, the pixel value of the pixel corresponding to the pixel of the virtual camera Vc is referred to (S3B). Further, the pixel value of the pixel corresponding to the pixel of the virtual camera Vc in the camera image 200C captured by the camera 5C is referred to (S3C). The pixel values referred to in S3B and S3C are mapped to the pixels of the virtual camera Vc. For example, a statistical value of a pixel value of a pixel on the camera image 200B corresponding to a pixel of the virtual camera Vc and a pixel value of a pixel on the camera image 200C, such as an arithmetic mean or a weighted average using a distance from the virtual camera Vc, is calculated. It is determined as the pixel value of the pixel of the virtual camera Vc.

In this way, the free viewpoint video corresponding to the virtual viewpoint is rendered by mapping the texture of the camera image 200B or the camera image 200C for each pixel of the virtual camera Vc. Note that, here, as an example, the case where the free viewpoint video is rendered by using the camera images of the plurality of cameras 5 has been illustrated, but the camera images of the nearest camera 5 having the closest distance to the virtual camera Vc are narrowed down. It can also be used to render free-viewpoint video.

[One aspect of the issue]
As described in the section of the background art above, the use of the background model in which the static background is modeled for rendering the free viewpoint video cannot deal with the case where the camera image includes the dynamic background. This is because, when a dynamic background is included, there is a gap between the depth from the virtual viewpoint to the background model and the depth from the virtual viewpoint to the dynamic background. This shift in depth is one of the causes, and as a result of mapping the pixel value of the pixel of the wrong texture coordinate in the camera image used as the texture, the image quality of the free viewpoint image is deteriorated.

FIG. 5: is a figure which shows an example of the cross section of a stadium. FIG. 5 shows a cross-sectional view of the stadium in the direction from the center to the outside, that is, in the row direction of the stands as a cutting plane. In the cross-sectional view shown in FIG. 5, a background model 3Mbgs in which a stand portion of a stadium is modeled is shown as an example of a static background. Further, in the cross-sectional view shown in FIG. 5, as an example of a subject corresponding to the dynamic background, a spectator 3bgd who is watching a sport at a stadium stand is shown.

As shown in FIG. 5, there is a gap between the depth from the virtual viewpoint Vc to the background model 3Mbgs (the portion indicated by the solid line arrow) and the depth from the virtual viewpoint Vc to the audience 3bgd (the portion indicated by the alternate long and short dash line). Nevertheless, when the background model 3Mbgs in which the static background is modeled is used for rendering, the texture coordinates corresponding to the three-dimensional position of the background model 3Mbgs of the static background are not the three-dimensional position of the audience 3bgd of the dynamic background. Is used for texture mapping. That is, the Ray passing through the optical center of the virtual viewpoint Vc is not the three-dimensional position of the intersection O2 intersecting the spectator 3bgd, but the three-dimensional position of the intersection O1 where Ray intersecting the optical center of the virtual viewpoint Vc intersects with the background model 3Mbgs is the camera image 200B. And projected on a texture such as 200C. As described above, as a result of using pixels with wrong texture coordinates in the camera images 200B and 200C for texture mapping, the image quality of the free-viewpoint video deteriorates.

From the aspect corresponding to such a dynamic background, the free-viewpoint video generation device mentioned in the section of the background art has been proposed. The free-viewpoint video generation device first generates a temporary free-viewpoint image of each frame from the reference image and the depth map. Then, the free viewpoint video generation device extracts the background image and its depth value to be stored in the curved background buffer from the reference image and the depth map as the background area. Then, the free viewpoint video generation device complements the temporary free viewpoint image with the background image stored in the curved background buffer and its depth value.

However, in the above-described free-viewpoint image generation device, it may still be impossible to reduce the depth shift of the background model and the dynamic background.

That is, in the above free-viewpoint image generation device, the background area is extracted by using the depth corresponding to the minimum value in the Gaussian distribution in which the depth distribution of the reference image is smoothed for dividing the foreground and the background. However, when the depth value is used to extract the background region, it becomes difficult to divide the foreground and the background as the depth values become closer to each other. Therefore, in the curved background buffer, the subject corresponding to the foreground is erroneously stored as the background, resulting in an increase in the depth shift.

[One aspect of approach to problem solving]
Therefore, the server device 10 according to the present embodiment calculates the depth image corresponding to each viewpoint in a predetermined frame from the side surface corresponding to the dynamic background. For example, the depth image may be calculated by stereo matching from two or more camera images, or may be measured by a depth camera such as a three-dimensional laser sensor.

Then, the server device 10 according to the present embodiment corrects the depth of each pixel of the depth image using the depth of the pixel separated into the background by the foreground/background separation for the camera image, and the corrected depth image is used as a background model. To generate.

As described above, by using the separation result of the foreground and the background, even when the depth of the foreground subject and the depth of the background subject are close to each other, the depth image can be corrected by distinguishing between them. Further, even at the boundary between the foreground subject and the background subject, it is possible to correct the depth variation in the depth image without mixing both. As a result of the background model being generated from the depth image thus corrected, the accuracy of the background model can be improved.

Therefore, according to the server device 10 according to the present embodiment, it becomes possible to reduce the depth shift of the background model and the dynamic background.

[Configuration of Server Device 10]
Next, the functional configuration of the server device 10 according to the present embodiment will be described. FIG. 6 is a block diagram illustrating a functional configuration of the server device 10 according to the first embodiment. As shown in FIG. 6, the server device 10 includes a communication I/F (InterFace) unit 11, a storage unit 13, and a control unit 15. It should be noted that FIG. 11 only shows an excerpt of the functional units related to the above-described video generation service, and the functional units other than those shown in the figure, for example, the functional units that an existing computer equips by default or as an option, are servers. It does not prevent the device 10 from being equipped. For example, when multi-view camera images are propagated from the camera 5 to the server device 10 via broadcast waves or satellite waves, a broadcast wave or satellite wave receiving unit may be further provided.

The communication I/F unit 11 is an interface that controls communication with other devices.

As an embodiment, the communication I/F unit 11 corresponds to a network interface card such as a LAN (Local Area Network) card. For example, the communication I/F unit 11 receives a camera image from each camera 5, or sends an instruction relating to image pickup control, for example, turning on/off the power, and also an instruction such as pan and tilt to the camera 5. To do.

The storage unit 13 corresponds to hardware that stores data used for various programs such as an OS (Operating System) executed by the control unit 15 and the above-described image generation program.

As one embodiment, the storage unit 13 corresponds to the auxiliary storage device in the server device 10. For example, an HDD (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), or the like corresponds to the auxiliary storage device. In addition, a flash memory such as an EPROM (Erasable Programmable Read Only Memory) also corresponds to the auxiliary storage device.

The storage unit 13 stores a silhouette image 210 and a corrected depth image 230 as an example of data used for a program executed by the control unit 15. In addition to the silhouette image 210 and the corrected depth image 230, the storage unit 13 can store various data related to the technique of free viewpoint video. For example, the storage unit 13 stores time-series data of camera images transmitted from the camera 5, in addition to camera parameters including external parameters such as the position and orientation of the camera 5 and internal parameters such as the angle of view of the camera 5 and lens distortion. Etc. can be saved for each viewpoint. The silhouette image 210 and the corrected depth image 230 will be described together with the description of the control unit 15 that registers or refers to each data.

The control unit 15 is a processing unit that controls the entire server device 10.

As one embodiment, the control unit 15 can be implemented by a hardware processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit). Here, the CPU and the MPU are illustrated as an example of the processor, but the processor can be implemented by any processor regardless of general-purpose type and specialized type. In addition, the control unit 15 may be realized by a hard-wired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 15 virtually realizes the following processing unit by expanding the above-mentioned video generation program on a work area of a RAM such as a DRAM (Dynamic Random Access Memory) implemented as a main storage device (not shown). To do. In addition, here, the example in which the video generation program in which the function corresponding to the above-described video generation service is packaged is executed has been described, but the present invention is not limited to this. For example, among the functions provided by the above video generation service, a program module is executed in units such as a background model generation function that generates a background model from a corrected depth image in which the depth image of each viewpoint is corrected, or a library is referenced. It doesn't matter if they are done.

As shown in FIG. 6, the control unit 15 includes an acquisition unit 15A, a calculation unit 15B, a separation unit 15C, a correction unit 15D, a foreground generation unit 15E, a background generation unit 15F, and a rendering unit 15G. ..

The acquisition unit 15A is a processing unit that acquires a camera image of each viewpoint.

As one embodiment, the acquisition unit 15A can acquire the camera image of each viewpoint transmitted from the cameras 5A to 5N in frame units. Here, the information source from which the acquisition unit 15A acquires the camera image may be any information source and is not limited to the camera 5. For example, the acquisition unit 15A can also acquire multi-view camera images from an auxiliary storage device such as a hard disk or an optical disc that stores camera images of each viewpoint, or a removable medium such as a memory card or a USB (Universal Serial Bus) memory. .. In addition, the acquisition unit 15A can also acquire a camera image of each viewpoint from an external device other than the camera 5 via the network NW.

After the camera images of the respective viewpoints are acquired in this way, a silhouette image used for generating the foreground model and a corrected depth image used for generating the background model are generated. The silhouette image and the corrected depth image can be generated for each viewpoint of the camera 5 by inputting the camera image corresponding to the viewpoint and performing the processing by the calculation unit 15B, the separation unit 15C, and the correction unit 15D.

Hereinafter, as an example, the calculation of the depth image by the calculation unit 15B, the foreground/background separation by the separation unit 15C, and the correction of the depth image by the correction unit 15D will be described as a single thread. However, the present invention is not limited to this. Not done. For example, the calculation of the depth image by the calculation unit 15B, the foreground/background separation by the separation unit 15C, and the correction of the depth image by the correction unit 15D may be performed in parallel for each viewpoint of the camera 5. When multithreaded parallel processing is performed in this way, the calculation unit 15B, the separation unit 15C, and the correction unit 15D can operate in parallel up to the number of threads corresponding to the number of viewpoints of the camera 5.

FIG. 7 is a diagram illustrating an example of data transmitted and received between the functional units according to the first embodiment. In FIG. 7, as an example, when the depth image corresponding to the viewpoint of the camera 5A among the N viewpoints of the cameras 5A to 5N is calculated, the foreground and background are separated, and the depth image is corrected, the calculation unit 15B, An example of data exchanged between the separation unit 15C and the correction unit 15D is shown.

Hereinafter, among the N viewpoints, a viewpoint selected as a processing target of depth image calculation, foreground/background separation, and depth image correction may be referred to as a “reference viewpoint”. Note that, in the following, as an example, the case where the reference viewpoint is the viewpoint of the camera 5A is excerpted and illustrated. However, even when the viewpoint of another camera 5 is selected as the reference viewpoint, the processing is performed only by changing the camera image. The contents are the same.

As shown in FIG. 7, among the camera images 200A to 200N acquired by the acquisition unit 15A, the camera image 200A corresponding to the reference viewpoint is input to the calculation unit 15B. Furthermore, as an example only, from the side of calculating the depth image corresponding to the reference viewpoint by stereo matching, a viewpoint that can obtain a parallax with the camera image 200A, for example, the viewpoint of the camera 5B adjacent to the reference viewpoint is Selected as the reference viewpoint. The camera image 200B corresponding to the reference viewpoint thus selected is also input to the calculation unit 15B.

When the camera image 200A and the camera image 200B are input, the calculation unit 15B calculates the depth image 220A corresponding to the reference viewpoint by stereo matching. For example, the calculation unit 15B converts the parallax map of the camera image 200B with respect to the camera image 200A into the depth image 220A corresponding to the reference viewpoint according to the camera parameters of the cameras 5A and 5B. The depth image 220A thus obtained is input from the calculation unit 15B to the correction unit 15D.

Here, as an example, the depth image 220A corresponding to the reference viewpoint is calculated by stereo matching, but the present invention is not limited to this. For example, the depth image 220A corresponding to the reference viewpoint may be acquired by measuring with a depth camera such as a three-dimensional laser sensor.

On the other hand, the camera image 200A corresponding to the reference viewpoint is input to the separation unit 15C as well as the calculation unit 15B. When the camera image 200A is input, the separating unit 15C separates the subject included in the camera image 200A into the foreground and the background.

By way of example only, the separating unit 15C can extract the silhouette corresponding to the foreground from the camera image 200A based on the so-called background difference. For example, of the camera images 200A acquired in time series, an image of a frame in which the foreground is not likely to be observed is saved as a background image. For example, as the background image, an image of a frame in which a difference between frames is not detected over a predetermined number of frames can be used. Under such a background image being stored, the separation unit 15C determines whether the foreground is different depending on whether the difference in pixel value between the camera image 200A corresponding to the latest frame and the background image is equal to or more than a predetermined threshold value. Alternatively, a background label is assigned for each pixel. As a result, a silhouette image 210A to which a foreground or background label is assigned for each pixel is obtained. Here, as an example, the foreground/background separation is performed based on the background difference, but the foreground/background separation may be performed by a graph cut, for example, a two-dimensional graph cut.

The silhouette image 210A thus obtained is input to the correction unit 15D from the separation unit 15C from the side used for correcting the depth image 220A, and is also stored in the storage unit 13 from the side used for generating the foreground model. ..

When the depth image 220A and the silhouette image 210A are input, the correction unit 15D corrects the depth image 220A using the silhouette image 210A. When correcting the depth image 220A, the pixel values of the pixels of the depth image 220A to which the background label is assigned in the silhouette image 210A are made valid, and (1) spatial filtering and (2) temporal filtering can be performed. It suffices that at least one of these spatial filtering and temporal filtering is executed, and both do not necessarily have to be executed.

1 and 7 exemplify the case where the silhouette image and the corrected depth image are stored in the storage unit 13, but the silhouette image and the corrected depth image may not necessarily be stored in the storage such as the storage unit 13. Absent.

(1) Spatial Filtering The correction unit 15D applies a filter for each pixel included in the depth image 220A by convolving the depth value of the pixel with the depth value of the peripheral pixels. Examples of such a filter include a Gaussian filter, an edge-preserving filter that refers to edges of an input image, and a smoothing filter such as a bilateral filter.

Here, when the filter is applied, the correction unit 15D validates the depth value of the pixel of the depth image 220A to which the background label is assigned in the silhouette image 210A, and executes the filter convolution operation.

FIG. 8A is a diagram showing an example of a silhouette image 210A. FIG. 8B is a diagram showing an example of the depth image 220A. 8A and 8B show an example in which a Gaussian filter having a filter size of 3×3 is applied, and three pixels of pixel (a), pixel (b), and pixel (c) are shown as an example. The scene where the Gaussian filter is applied is shown. Here, for convenience of explanation, a case where the filter size is 3×3 is illustrated, but it goes without saying that the filter size may be any filter size.

Hereinafter, a pixel that is overlapped with the origin of the filter when the filter is applied may be referred to as a “target pixel”, and a pixel located in the periphery of the target pixel, for example, a pixel in the vicinity of 8 may be referred to as a “peripheral pixel”. ..

FIG. 9A is a diagram showing an example of an image ID (IDentification). As shown in FIG. 9A, at the time of the convolution calculation of the filter, the pixel of interest is identified as “p ₄ ”as an example. Further, among the peripheral pixels of the pixel of interest, the upper left pixel is “p ₀ ”, the pixel immediately above is “p ₁ ”, the upper right pixel is “p ₂ ”, the left pixel is “p ₃ ”, the right pixel Is identified as “p ₅ ”, the lower left pixel is identified as “p ₆ ”, the pixel immediately below is identified as “p ₇ ”, and the lower right pixel is identified as “p ₈ ”.

Under such identification, the correction unit 15D calculates the correction depth D _{i of the} pixel of interest according to the following formula (1) or formula (2). Here, “i” in Expressions (1) and (2) indicates a pixel ID, and includes, for example, eight pixels from p ₀ to p ₈ . In addition, “l _i ”in Expression (1) refers to a label value of the foreground or the background, which is given to the pixel p _i of the pixels of the silhouette image 210A. Here, as an example, the following description will be made assuming that “1” is given to the background label and “0” is given to the foreground label. Further, “k _i ”in Expression (1) refers to the filter coefficient applied to the pixel p _i in the array of 3×3 filter coefficients. In addition, “d _i ”in Expression (1) refers to the depth value of the pixel p _i of the pixels of the depth image 220A.

D _i =(Σl _i ×k _i ×d _i )÷(Σl _i ×k _i )If l ₄ =1...(1)
D _i =foreground If l ₄ =0 (2)

That is, the correction unit 15D, when the pixel of interest labeled l ₄ is "1", that is, when the label of the background to the target pixel is assigned to calculate the corrected depth using Equation (1). On the other hand, if the target pixel label l ₄ is "0", i.e., if the foreground label is applied to the pixel of interest, the pixel of interest is identified as foreground by equation (2). In this case, the correction unit 15D invalidates the depth of the target pixel by setting an invalid value, for example, a NULL value, in the correction depth D _i of the target pixel from the viewpoint of suppressing the depth of the target pixel from being used for generating the background model.

For example, when the Gaussian filter is applied to the pixel (a), the correction unit 15D executes the convolution calculation shown in FIG. 9B. FIG. 9B is a diagram showing an example of the convolution operation of the filter. In FIG. 9B, pixels for which the depth is valid at the time of the convolution calculation are shown by hatching. That is, as shown in the silhouette image 210A of FIG. 8A, the background label is given to the pixel of interest (a) and its eight neighboring pixels. In this case, as shown in FIG. 9B, “1” is set to all label values l _{0 to} l ₈ of the label matrix. With such a label matrix, all depth values d _{0 to} d _{8 of the} depth matrix are used in the convolution operation. In addition, among the kernel, "1/16" as the filter coefficient _{k 0} of the upper left pixel _{p 0,} "2/16" as the filter coefficients _{k 1} directly above the pixel _{p 1,} the filter coefficients in the upper right of the pixel _{p 2} k "1/16" is used as a _2. Furthermore, "2/16" as the filter coefficient _{k 3} of the left pixel _{p 3,} "4/16" as the filter coefficient _{k 4} of the pixel of interest _{p 4,} as the filter coefficient _{k 5} of the right pixel _{p 5} "2/16 Is used. Furthermore, "1/16" as the filter coefficient _{k 6} of the lower left pixel _{p 6,} "2/16" as the filter coefficients _{k 7} pixel _{p 7} beneath, "1 as the filter coefficient _{k 8} pixels _{p 8} lower right /16” is used.

Under these label matrix, kernel, and depth matrix, the correction depth D _i of the pixel of interest (a) is calculated according to Expression (1). For example, the calculation of the upper left pixel p ₀ is 1×(1/16)×d ₀ . Moreover, the calculation of the pixel p ₁ immediately above is 1×(2/16)×d ₁ . The calculation of the upper left pixel p ₂ is 1×(1/16)×d ₂ . Further, the calculation of the left pixel p ₃ is 1×(2/16)×d ₃ . Further, the calculation of the pixel p ₄ of the target pixel is 1×(4/16)×d ₄ . Further, the calculation of the right pixel p ₅ is 1×(2/16)×d ₅ . Further, the calculation of the lower left pixel p ₆ is 1×(1/16)×d ₆ . Further, the calculation of the pixel p ₇ immediately below is 1×(2/16)×d ₇ . Further, the calculation of the lower right pixel p ₈ is 1×(1/16)×d ₈ . The sum of these is calculated as the correction depth D _i of the target pixel (a).

Next, when the Gaussian filter is applied to the pixel (b), the correction unit 15D executes the convolution operation shown in FIG. 9C. FIG. 9C is a diagram showing an example of the convolution operation of the filter. Also in FIG. 9C, the pixels for which the depth is valid at the time of the convolution operation are shown by hatching, while the pixels for which the depth is invalid at the time of the convolution operation are shown as plain. That is, as shown in the silhouette image 210A of FIG. 8A, although the target pixel (b) is labeled with the background, some of the peripheral pixels in the eight neighboring pixels, that is, the left pixel and The foreground label is attached to the lower left pixel. In this case, as shown in FIG. 9C, “0” is set to the label value l ₃ of the left pixel and the label value l ₆ of the lower left pixel of the label matrix. With such a label matrix, the depth value d ₃ of the left pixel and the depth value d ₆ of the lower left pixel of all the depth values d _{0 to} d ₈ of the depth matrix are invalidated.

Under these label matrix, kernel, and depth matrix, the correction depth D _i of the pixel of interest (b) is calculated according to Expression (1). For example, the calculation of the upper left pixel p ₀ is 1×(1/16)×d ₀ . Moreover, the calculation of the pixel p ₁ immediately above is 1×(2/16)×d ₁ . The calculation of the upper left pixel p ₂ is 1×(1/16)×d ₂ . Further, the calculation of the left pixel p ₃ is 0×(2/16)×d ₃ . Further, the calculation of the pixel p ₄ of the target pixel is 1×(4/16)×d ₄ . Further, the calculation of the right pixel p ₅ is 1×(2/16)×d ₅ . Further, the calculation of the lower left pixel p ₆ is 0×(1/16)×d ₆ . Further, the calculation of the pixel p ₇ immediately below is 1×(2/16)×d ₇ . Further, the calculation of the lower right pixel p ₈ is 1×(1/16)×d ₈ . The sum of these is calculated as the correction depth D _i of the pixel of interest (b).

Further, when the Gaussian filter is applied to the pixel (C), the foreground label is given to the pixel (C), so the correction unit 15D sets the NULL value to the correction depth D _i of the pixel (C) of interest. Set.

As described above, the correction unit 15D validates the depth value of the pixel to which the background label is assigned among the pixel of interest and the peripheral pixels for each pixel of the depth image 220A, and sets the depth value of the pixel to which the foreground label is assigned. Perform spatial filtering to apply the filter as invalid. As a result, it is possible to correct the depth variation in the depth image even in the pixels at the boundary between the foreground subject and the background subject without the depths of both being mixed. Therefore, it is possible to suppress the variation in depth between pixels of the depth image, or to interpolate the depth value even when the depth value of the pixel to which the background label is assigned in the depth image has a defect.

(2) Temporal Filtering The correction unit 15D applies, for each pixel included in the depth image 220A, a depth value of the pixel that is convolved with the depth value of the pixel existing at the same position retroactively to a predetermined number of past frames. ..

Also in this temporal filtering, when the filter is applied, the correction unit 15D validates the depth value of the pixel of the depth image 220A to which the background label is assigned in the silhouette image 210A, and executes the filter convolution operation.

FIG. 10A is a diagram showing an example of a pixel of interest in temporal filtering. As shown in FIG. 10A, at the time of the convolution calculation of the filter, the pixel of interest is identified as “i” as an example. Furthermore, the index t is used to identify the frame of the depth image 220A, and the target frame for which the corrected depth D _i of the target pixel _i is calculated is identified as “t=T”. Further, among the past frames of the target frame T, the past frame immediately before the target frame is identified as “t=T−1”, and the N past frames of the target frame among the past frames of the target frame T are denoted by “t. =T−N”.

For example, the correction unit 15D calculates the correction depth D _i,T of the target pixel p _i,T of the target frame T according to the following formula (3) or the following formula (4). Here, “N” in Expression (3) and Expression (4) indicates the size of the kernel. In addition, “l _i,t ” in Expression (3) refers to the label value of the foreground or background that is given to the pixel p _i,t of the silhouette image 210A of the frame t. Here, as an example, the following description will be made assuming that “1” is given to the background label and “0” is given to the foreground label. Further, “k _t ”in Expression (3) refers to a filter coefficient applied to frame t in the kernel. In addition, “d _i,t ” in Expression (3) refers to the depth value of the pixel p _i,t of the depth image 220A of the frame t.

D _i,T =foreground ground If l _i,T =0 (4)

That is, the correction unit 15D includes the pixel of interest p _i of the frame of interest _T, when _T label l _{i, T} is "1", that is, when the label of the background to the target pixel of the frame of interest has been granted, the formula ( Calculate the corrected depth using 3). On the other hand, the pixel of interest p _i of the frame of interest _T, when _T label l _{i, T} is "0", i.e., if the foreground label to the target pixel is assigned a pixel of interest is the foreground by the formula (4) To be identified. In this case, the correction unit 15D includes the pixel of interest p _i of the frame of interest _T, from the side depth of _T is suppressed from being used to generate the background model, the pixel of interest p _i of the frame of interest _{T, T} of the correction depth D _{i , T} as null values and invalidate.

FIG. 10B is a diagram illustrating an example of the convolution operation of the filter. In FIG. 10B, the target pixel p _{i,t of the} frame t for which the depth is valid during the convolution calculation is shown by hatching, while the target pixel p _{i,t of the} frame t for which the depth is invalid during the convolution calculation is shown. Shown in solid color. Note that FIG. 10B shows an example in which the kernel size N is “4”, but the kernel size N may be any value of 2 or more.

As shown in FIG. 10B, regarding the pixel of interest i, a background frame is given to the frame of interest T, the previous frame T-1 that is one before, and the past frame T-3 that is three before, but there are two. The foreground label is given to the previous past frame T-2. In this case, as shown in FIG. 10B, “0” is set to the label value l _i,T-2 of the past frame T-2 that is two frames before in the label matrix. Such label matrix, the depth value _d i of the depth matrix, _{T to d i,} 2 preceding the past frame T-2 of the depth values _{d i} of the _T-4, the _T-2 is invalidated. Further, in the kernel, the filter coefficient k _T of the frame of interest _T is “20/64”, the filter coefficient k _T-1 of the previous frame _T-1 is “15/64”, and the previous frame is two previous frames. “6/64” is used as the filter coefficient k _T-2 of _T-2 . Furthermore, “1/64” is used as the filter coefficient k _T-3 of the past frame T-3 that is three frames before.

Under these label matrix, kernel and depth matrix _, the correction depth D _{i,T of the} target pixel p _i,T is calculated according to the equation (3). For example, the calculation of the pixel p _i,T of the frame of interest T is 1×(20/64)×d _i,T . In addition, the calculation of the pixel p _i,T-1 of the immediately previous frame T-1 is 1×(15/64)×d _i,T-1 . Further, the calculation of the pixel p _i,T-2 of the previous frame T-2 two frames before is 0×(6/64)×d _i,T-2 . Further, the calculation of the pixel p _i,T-3 of the past frame T-3 three frames before is 1×(1/64)×d _i,T-3 . The sum of these is calculated as the correction depth D _i,T of the pixel of interest p _i,T .

As described above, the correction unit 15D validates the depth value of the pixel to which the background label is assigned among the pixels of interest in the target frame and the past frame for each pixel of the depth image 220A, and determines the depth value of the pixel to which the foreground label is assigned. Apply the filter by invalidating the depth value. By such temporal filtering, even if the label of the pixel of interest varies in the foreground or background in the past frame, it is possible to correct the depth variation between the frames of the depth image without mixing the depths of the both. For this reason, it is possible to suppress variation in depth between frames of the depth image.

The corrected depth image 230A obtained by the spatial filtering and the temporal filtering is stored in the storage unit 13 from the aspect used for generating the background model.

Returning to the description of FIG. 6, the foreground generation unit 15E is a processing unit that generates a foreground model.

As one embodiment, the foreground generation unit 15E can generate the foreground model 3Mfg using the silhouette image 210 stored in the storage unit 13 for each viewpoint of the camera 5. The Visual-Hull described above with reference to FIG. 3 can be applied to the generation of this foreground model. In this Visual Hull, a Cone formed by connecting the optical center of the camera 5 and the silhouette on the silhouette image is generated, and the area in the three-dimensional space where the Cone overlaps is the three-dimensional shape of the subject 3fg corresponding to the foreground. Is calculated as For example, as shown in FIG. 3, the foreground generation unit 15E projects the silhouettes SA to SC in the three-dimensional space for each of the silhouette images 210A to 210C corresponding to the viewpoints of the cameras 5A to 5C. For example, when the silhouette SA is projected, a visual volume CA connecting the optical center of the camera 5A and the silhouette SA on the silhouette image 210A is obtained. Further, when the silhouette SB is projected, a visual volume CB connecting the optical center of the camera 5B and the silhouette SB on the silhouette image 210B is obtained. Furthermore, when the silhouette SC is projected, a visual volume CC connecting the optical center of the camera 5C and the silhouette SC on the silhouette image 210C is obtained. A Visual Hull region in which these visual volumes CA to CC overlap, that is, a three-dimensional shape with black filling shown in FIG. 3 is calculated as the foreground model 3Mfg.

The background generation unit 15F is a processing unit that generates a background model.

As an embodiment, the background generation unit 15F can generate the background model 3Mbg using the corrected depth image 230 stored in the storage unit 13 for each viewpoint of the camera 5. For example, the background generation unit 15F generates the background model 3Mbg by combining the corrected depth images 230 of the respective viewpoints. Note that here, as an example, the corrected depth image is combined to generate the three-dimensional background model, but the three-dimensional background model may not necessarily be generated. For example, the corrected depth image of each viewpoint may be directly input to the rendering unit 15G without being combined with the corrected depth image obtained for each viewpoint of the camera 5.

The rendering unit 15G is a processing unit that renders free viewpoint video.

As one embodiment, the rendering unit 15G can specify a virtual viewpoint by receiving a user input from the client terminal 30. In addition, the rendering unit 15G can specify a virtual viewpoint by user setting via the client terminal 30 or system setting registered in the server device 10. In this way, after the virtual viewpoint is designated, the rendering unit 15G renders the free viewpoint video corresponding to the virtual viewpoint, as described with reference to FIG. That is, the rendering unit 15G calculates the three-dimensional position of the intersection of the straight line passing through the optical center of the virtual camera Vc and the pixel and the foreground model 3Mfg or the background model 3Mbg (S1). Then, the rendering unit 15G corresponds the above intersections to each viewpoint according to the external parameters such as the position and orientation of the camera 5 and the internal parameters such as the angle of view of the camera 5 and the distortion of the lens. Project to camera image. In the example shown in FIG. 4, the intersection is projected on the camera images of a predetermined number of cameras 5 that are close to the virtual camera Vc, that is, the two camera images 200B and 200C of the cameras 5B and C (S2B). And S2C). Thereby, the pixel of the camera 5B and the pixel of the camera 5C corresponding to the pixel of the virtual camera Vc are identified as the texture coordinates. After that, the rendering unit 15G refers to the pixel value of the pixel corresponding to the pixel of the virtual camera Vc in the camera image 200B captured by the camera 5B (S3B). Further, the rendering unit 15G refers to the pixel value of the pixel corresponding to the pixel of the virtual camera Vc in the camera image 200C captured by the camera 5C (S3C). Then, the rendering unit 15G maps the pixel values referred to in S3B and S3C to the pixels of the virtual camera Vc. For example, a statistical value of a pixel value of a pixel on the camera image 200B corresponding to a pixel of the virtual camera Vc and a pixel value of a pixel on the camera image 200C, such as an arithmetic mean or a weighted average using a distance from the virtual camera Vc, is calculated. It is determined as the pixel value of the pixel of the virtual camera Vc.

[Process flow]
FIG. 11 is a flowchart illustrating the procedure of the video generation process according to the first embodiment. As an example, this processing is executed when a camera image is acquired from each camera 5, that is, when a multi-view camera image is obtained.

As illustrated in FIG. 11, when the camera images of the respective viewpoints are acquired from the cameras 5A to 5N (step S101), the calculation unit 15B uses the unselected viewpoints among the N viewpoints of the cameras 5A to 5N as a reference. A viewpoint is selected (step S102). Subsequently, the calculation unit 15B selects, as a reference viewpoint, a viewpoint that can obtain parallax with the camera image corresponding to the reference viewpoint, for example, the viewpoint of the camera 5 adjacent to the reference viewpoint (step S103).

Then, the calculation unit 15B calculates the depth image corresponding to the standard viewpoint from the camera image corresponding to the standard viewpoint selected in step S102 and the camera image corresponding to the reference viewpoint selected in step S103 by stereo matching. (Step S104).

Further, the separating unit 15C separates the subject included in the camera image corresponding to the reference viewpoint selected in step S102 into the foreground and the background (step S105). By separating the foreground and the background in this way, a silhouette image in which the label of the foreground or the background is assigned to each pixel is obtained.

Then, the correction unit 15D uses the silhouette image obtained in step S105 to correct the depth image obtained in step S104 (step S106). A corrected depth image is obtained by the correction of the depth image.

Then, until all the pixels included in the camera image are selected (No in step S107), the processes from step S102 to step S106 are repeatedly executed.

After that, when all the pixels included in the camera image are selected (Yes in step S107), the foreground generation unit 15E generates the foreground model using the silhouette image of each viewpoint obtained by repeating step S105 (step S108). ). The background generation unit 15F also generates a background model using the corrected depth image of each viewpoint obtained by repeating step S106 (step S109).

Then, the rendering unit 15G uses the camera image of each viewpoint acquired in step S101 and the foreground model and background model generated in steps S108 and S109 to generate a camera image corresponding to a virtual viewpoint, a so-called free viewpoint video. Is generated (step S110), and the process ends.

Although the flowchart of FIG. 11 shows an example in which the foreground/background separation of step S105 is executed after the processing of step S104 is executed, the foreground/background separation of step S105 selects the reference viewpoint in step S102. It is possible to start from the designated stage. Therefore, the foreground/background separation in step S105 may be executed before the processing in steps S103 and S104, or may be executed in parallel with the processing in steps S103 and S104. Even if such an order change or parallel processing is performed, the processing content of the foreground/background separation in step S105 remains unchanged.

[One side of effect]
As described above, the server device 10 according to the present embodiment corrects the depth image corresponding to each viewpoint using the foreground/background separation result obtained by the foreground/background separation performed from the side that generates the foreground model. Then, a background model is generated from the corrected depth image. As described above, by using the separation result of the foreground and the background, even when the depth of the foreground subject and the depth of the background subject are close to each other, the depth image can be corrected by distinguishing between them. Further, even at the boundary between the foreground subject and the background subject, it is possible to correct the depth variation in the depth image without mixing both. As a result of the background model being generated from the depth image thus corrected, the accuracy of the background model can be improved. Therefore, according to the server device 10 according to the present embodiment, it becomes possible to reduce the depth shift of the background model and the dynamic background.

Although the embodiments of the disclosed device have been described above, the present invention may be implemented in various different forms other than the embodiments described above. Therefore, other embodiments included in the present invention will be described below.

[Application example 1 of foreground/background separation]
For example, the server device 10 can perform foreground/background separation by further using the depth image obtained by stereo matching or the like.

That is, as described in the first embodiment, when the foreground/background separation is realized by the background difference, the foreground and the background are separated based on the color information represented by the pixel value. In this case, when the foreground subject and the background subject are similar in color, the foreground subject may be separated as the background, or the background subject may be separated as the foreground, so that sufficient separation accuracy may not be achieved. For example, when a state of watching a sports game is captured as a camera image, since the player who is the foreground and the spectator who is the dynamic background are both humans, it is not necessary to separate the foreground and the background from the camera image using only color information. Have difficulty. Note that here, the case where the background difference is used for the foreground/background separation has been described as an example, but the present invention is not limited to this example. For example, in the case where a histogram of colors corresponding to the foreground and a histogram of colors corresponding to the background are generated in advance and the colors of the pixels of the camera image acquired by the acquisition unit 15A are separated based on the histograms of these colors. Similar challenges arise.

Therefore, in the application example 1, by performing the foreground/background separation by further using the depth information in addition to the color information, the robust foreground/background separation is achieved even when the colors of the foreground subject and the background subject are similar. This will be achieved, and the accuracy of separating the foreground and the background will be improved.

From the aspect of realizing such foreground/background separation, in Application Example 1, an example of using a two-dimensional graph cut for foreground/background separation will be described. For example, a labeling problem of assigning a foreground or background label to pixels included in a camera image is formulated as a problem of minimizing the energy function shown in the following formula (5).

E=ΣEd(p)+λΣEs(p,q) (5)

The energy function E shown in the above equation (5) includes the "data term" of the first term on the right side and the "smoothing term" of the second term on the right side. “Λ” in the equation (5) indicates a coefficient of weight given to the smoothing term. Further, “p” in Expression (5) indicates a pixel to which a label of the foreground or the background is assigned. Further, “q” in the equation (5) indicates a pixel adjacent to the pixel p, and, for example, pixels near 8 or around 4 around the pixel p can be set as adjacent pixels.

Here, the data term is, as shown in the following formula (6), the first foreground likelihood obtained from the color information and the energy E _color based on the first background likelihood, and the second foreground obtained from the depth value. It is formulated by the likelihood and the energy E _depth based on the second background likelihood. In addition, "w _color " in Formula (6) points out the weighting factor given to E _color, and "w _depth " in Formula (6) points out the weighting factor given to E _depth .

Ed(p)=w _color ×E _color +w _depth ×E _depth (6)

Further, as the smoothing term, a penalty function that makes the label smooth between adjacent pixels is defined as in the following Expression (7). It should be noted that “Cp” in Expression (7) indicates the pixel value of the pixel p, and “Cq” in Expression (7) indicates the pixel value of the adjacent pixel q.

ΣEs(p,q)=exp(|Cp−Cq|) (7)

With such a data term, the action of maintaining the tendency between the first foreground likelihood and the first background likelihood and the second foreground likelihood and the second background likelihood is exerted in the label allocation. You can Further, the smoothing can exert the effect of suppressing the label variation for each pixel in the label allocation.

The label of the foreground or the background can be assigned to each pixel by calculating the set of labels that minimizes the energy function E including the data term and the smoothing term according to the maximum flow minimum cut theorem.

From the aspect of realizing the graph cut as described above, in the application example 1, functional units such as the first likelihood calculation unit 21 and the second likelihood calculation unit 22 are added. Furthermore, in the application example 1, instead of the separation unit 15C shown in the above-described first embodiment, a separation unit 23 that realizes foreground/background separation by graph cutting is added.

FIG. 12 is a diagram illustrating an example of data exchanged between the functional units in the application example 1. FIG. 12 shows, as an example, an example of data exchanged between the functional units when the viewpoint of the camera 5A is selected as the reference viewpoint among the N viewpoints of the cameras 5A to 5N. There is.

As shown in FIG. 12, among the camera images 200A to 200N acquired by the acquisition unit 15A, the camera image 200A corresponding to the reference viewpoint is input to the calculation unit 15B. Furthermore, as an example only, from the side of calculating the depth image corresponding to the reference viewpoint by stereo matching, a viewpoint that can obtain a parallax with the camera image 200A, for example, the viewpoint of the camera 5B adjacent to the reference viewpoint is Selected as the reference viewpoint. The camera image 200B corresponding to the reference viewpoint thus selected is also input to the calculation unit 15B.

When the camera image 200A and the camera image 200B are input, the calculation unit 15B calculates the depth image 220A corresponding to the reference viewpoint by stereo matching. For example, the calculation unit 15B converts the parallax map of the camera image 200B with respect to the camera image 200A into the depth image 220A corresponding to the reference viewpoint according to the camera parameters of the cameras 5A and 5B.

Up to this point, there is no difference between FIG. 12 and FIG. 7, but it is different from here. That is, the depth image 220A obtained by stereo matching or the like is not only input from the calculation unit 15B to the correction unit 15D, but also input from the calculation unit 15B to the first likelihood calculation unit 21.

On the other hand, the camera image 200A corresponding to the reference viewpoint is input to the first likelihood calculation unit 21 as well as the calculation unit 15B. When the camera image 200A is input, the first likelihood calculation unit 21 calculates the first foreground likelihood and the first background likelihood by using the pixel value of the pixel for each pixel included in the camera image 200A. To do. The first foreground likelihood and the first background likelihood can be calculated as follows. For example, for each label of the foreground and background, a frequency distribution in which the color corresponds to the label, for example, a histogram or a probability distribution, for example, a Gaussian mixture distribution is calculated in advance. Here, as an example, a case where a mixed Gaussian distribution including K Gaussian distributions is prepared for each foreground and background label is illustrated. The first foreground likelihood and the first background likelihood of the pixel p are calculated by comparing the mixed Gaussian distribution and the pixel value I _p of the pixel p of the camera image 200A. For example, the first likelihood calculating unit 21 obtains the first foreground likelihood or the first background likelihood from the pixel value I _p of the pixel p of the camera image 200A according to the following formula (8). Here, “w _k ”in Expression (8) refers to the weight of the k-th Gaussian distribution. Further, “N(I _p |μ _k , Σ _k )” in Expression (8) indicates the kth Gaussian distribution. By such an equation (8), one Gaussian distribution is selected from K Gaussian distributions for each label of the foreground and the background. The first foreground likelihood and the first background likelihood calculated in this way are input from the first likelihood calculating unit 21 to the separating unit 23.

P _color (p|l)=Σw _k ·N(I _p |μ _k , Σ _k )... (8)

Further, the second likelihood calculation unit 22 to which the depth image 220A is input calculates the second foreground likelihood and the second background likelihood for each pixel of the depth image 220A using the depth value of the pixel. The second foreground likelihood and the second background likelihood can be calculated as follows. First, the presence area of the foreground and the presence area of the background are set in advance in the three-dimensional space. For example, taking sports watching as an example, a field surface in which a player competes in the stadium, a height at which the player or the ball can move, and the like are set as the foreground existence region. In addition, an area other than the area where the foreground exists in the stadium is set as the area where the background exists.

Under these settings of the foreground existing area and the background existing area, the second likelihood calculating unit 22 calculates an evaluation value in the depth direction of the pixel p of the depth image 220A corresponding to the reference viewpoint. For example, the second likelihood calculation unit 22 projects Ray, which passes through the pixel p for which the likelihood is to be calculated, from the optical center of the camera 5A of the standard viewpoint to the depth image 220B of the camera 5B of the standard viewpoint. As a result, an epipolar line is drawn on the depth image 220B. Then, the second likelihood calculating unit 22 calculates an evaluation value, for example, SAD (Sum of Absolute Difference) between the pixel p of the depth image 220A and each pixel existing on the epipolar line of the depth image 220B. ..

FIG. 13 is a diagram showing an example of a graph of the evaluation value and the depth. FIG. 13 shows a graph in which the vertical axis represents SAD and the horizontal axis represents depth. The “depth” here means the distance in the depth direction from the optical center of the camera 5A as the origin. Further, FIG. 13 shows the existence region of the foreground and the existence region of the background in an overlapping manner on the graph. As shown in FIG. 13, the second likelihood calculating unit 22 extracts the minimum point j1 at which the minimum value of SAD is observed among the SADs corresponding to the depth of the existing area of the foreground. Then, the second likelihood calculating unit 22 sets the SAD measured at the minimum point j1 as the representative evaluation value r(l), and uses this representative evaluation value r(l) as the second value according to the following equation (9). To the foreground likelihood of. Further, the second likelihood calculating unit 22 extracts the minimum point j2 at which the minimum value of SAD is observed among the SADs corresponding to the depth of the background existing area. Then, the second likelihood calculating unit 22 sets the SAD measured at the minimum point j2 as the representative evaluation value r(l), and uses this representative evaluation value r(l) as the second evaluation value according to the following equation (9). To the background likelihood of. The second foreground likelihood and the second background likelihood calculated in this way are input from the second likelihood calculating unit 22 to the separating unit 23.

P _depth (p|l)=exp(-r(l)) (9)

Then, the separation unit 23 calculates a set of labels that minimizes the energy function shown in the above equation (5) according to the maximum flow minimum cut theorem. By such a two-dimensional graph cut, a silhouette image 210A' in which a label of a foreground or a background is assigned to each pixel is obtained.

The silhouette image 210A′ thus obtained is input to the correction unit 15D from the separation unit 23 from the side used for correcting the depth image 220A, and is stored in the storage unit 13 from the side used also for generating the foreground model. It

When the depth image 220A and the silhouette image 210A' are input, the correction unit 15D corrects the depth image 220A using the silhouette image 210A'. At the time of correcting the depth image 220A, the correction unit 15D validates the pixel value of the pixel of the depth image 220A to which the background label is assigned in the silhouette image 210A', and executes at least one of spatial filtering and temporal filtering. .. As a result, a corrected depth image 230A' obtained by correcting the depth image 220A is obtained. As described above, even when the foreground subject and the background subject are similar in color, the accuracy of the background model can be improved by using the silhouette image 210A′ subjected to the robust foreground/background separation for the correction of the depth image. it can.

On the other hand, the silhouette image 210A' stored in the storage unit 13 is used by the foreground generation unit 15E together with other silhouette images 210 to generate a foreground model. As described above, by using the silhouette image 210A' for generating the foreground model, the accuracy of the foreground model can be improved.

FIG. 14 is a flowchart illustrating the procedure of the video generation process according to the application example 1. As an example, this processing is executed when a camera image is acquired from each camera 5, that is, when a multi-view camera image is obtained.

As illustrated in FIG. 14, when the camera images of the respective viewpoints are acquired from the cameras 5A to 5N (step S101), the calculation unit 15B uses the unselected viewpoints among the N viewpoints of the cameras 5A to 5N as a reference. A viewpoint is selected (step S102). Subsequently, the calculation unit 15B selects, as a reference viewpoint, a viewpoint that can obtain parallax with the camera image corresponding to the reference viewpoint, for example, the viewpoint of the camera 5 adjacent to the reference viewpoint (step S103).

Then, the calculation unit 15B calculates the depth image corresponding to the standard viewpoint from the camera image corresponding to the standard viewpoint selected in step S102 and the camera image corresponding to the reference viewpoint selected in step S103 by stereo matching. (Step S104).

Subsequently, the first likelihood calculating unit 21 calculates the first foreground likelihood and the first background likelihood of each pixel based on the color information of the camera image corresponding to the reference viewpoint (step S201). Further, the second likelihood calculating unit 22 calculates the second foreground likelihood and the second background likelihood of each pixel using the depth image calculated in step S104 (step S202).

Then, the separating unit 23 minimizes the energy function in which the first foreground likelihood and the first background likelihood, and the second foreground likelihood and the second background likelihood are incorporated in the data term. A set of labels to be calculated is calculated according to the maximum flow minimum cut theorem (step S203). With such a two-dimensional graph cut, a silhouette image 210A in which a foreground or background label is assigned to each pixel is obtained.

Then, the correction unit 15D corrects the depth image obtained in step S104 using the silhouette image obtained in step S203 (step S106). A corrected depth image is obtained by the correction of the depth image.

Then, until all the pixels included in the camera image are selected (No in step S107), the processes from step S102 to step S106 are repeatedly executed.

After that, when all the pixels included in the camera image are selected (Yes in step S107), the foreground generation unit 15E generates the foreground model using the silhouette image of each viewpoint obtained by repeating step S203 (step S108). ). The background generation unit 15F also generates a background model using the corrected depth image of each viewpoint obtained by repeating step S106 (step S109).

Then, the rendering unit 15G uses the camera image of each viewpoint acquired in step S101 and the foreground model and background model generated in steps S108 and S109 to generate a camera image corresponding to a virtual viewpoint, a so-called free viewpoint video. Is generated (step S110), and the process ends.

Note that the flowchart of FIG. 14 illustrates an example in which the calculation of the first foreground likelihood and the first background likelihood in step S201 is executed after the processing in step S104 is executed. The process can be started from the stage when the reference viewpoint is selected in step S102. Therefore, the foreground/background separation in step S203 may be executed before the processing in steps S103 and S104, or may be executed in parallel with the processing in steps S103 and S104. Even when such order replacement and parallel processing are performed, the processing content of step S201 remains the same. Also, an example is shown in which the calculation of the second foreground likelihood and the second background likelihood in step S202 is executed after the processing in step S201 is executed, but the processing in step S202 is executed in step S104. It can start from the stage when the depth image is calculated. Therefore, the process of step S202 may be executed before the process of step S201, or may be executed in parallel with the process of step S201. Even when such order replacement and parallel processing are performed, the processing content of step S202 does not change.

[Application example 2 of foreground/background separation]
In the above-described first embodiment, the example in which the pixels included in the camera image are separated into at least two categories of the foreground and the background has been described, but the pixels may be separated into three or more categories. For example, the separating unit 15C and the separating unit 23 can further separate the pixels, which are separated from the background among the pixels included in the camera image, into the background subcategory group into which the background category is further divided. For example, in the case of watching sports, the background category can be further divided into a background subcategory 1 “spectator” and a background subcategory 2 “field”. When there are three or more categories in this way, when performing graph cut, instead of the binary labels of the foreground and the background, for each of the multivalued labels corresponding to the foreground category, the background subcategory 1 and the background subcategory 2, , A first foreground likelihood and a first background likelihood, and a second foreground likelihood and a second background likelihood. For example, when the first foreground likelihood and the first background likelihood are calculated, a mixed Gaussian distribution may be prepared for each foreground category, background subcategory 1 and background subcategory 2. Further, when calculating the second foreground likelihood and the second background likelihood, the presence area of the foreground category, the presence area of the background subcategory 1 and the presence area of the background subcategory 2 may be set. Then, the separating unit 15C and the separating unit 23 assign the multivalued labels of the foreground category, the background subcategory 1 and the background subcategory 2 to each pixel by the multivalued graph cut. Then, the correction unit 15D may correct the depth of the pixels included in the depth image using the depth of the pixels separated into the same background subcategory as the background subcategory in which the pixels are separated. For example, the convolution calculation may be performed by narrowing down the depth value of the peripheral pixels and the depth value of the past frame separated into the same background subcategory as the background subcategory of the pixel of interest.

[Distribution and integration]
In addition, each component of each illustrated device may not necessarily be physically configured as illustrated. That is, the specific form of distribution/integration of each device is not limited to that shown in the figure, and all or part of the device may be functionally or physically distributed/arranged in arbitrary units according to various loads and usage conditions. It can be integrated and configured. For example, the acquisition unit 15A, the calculation unit 15B, the separation unit 15C, the correction unit 15D, the foreground generation unit 15E, the background generation unit 15F or the rendering unit 15G may be connected as an external device of the server device 10 via a network. Further, another device has the acquisition unit 15A, the calculation unit 15B, the separation unit 15C, the correction unit 15D, the foreground generation unit 15E, the background generation unit 15F, or the rendering unit 15G, and by being network-connected and cooperating, You may make it implement|achieve the function of the said server apparatus 10.

[Background model generation program]
Further, the various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a background model generation program having the same functions as those in the above embodiments will be described with reference to FIG.

FIG. 15 is a diagram illustrating a hardware configuration example of a computer that executes the background model generation program according to the first and second embodiments. As shown in FIG. 15, the computer 100 includes an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 has a CPU 150, a ROM 160, an HDD 170, and a RAM 180. Each unit of these 110 to 180 is connected via a bus 140.

As shown in FIG. 15, the HDD 170 is similar to the acquisition unit 15A, the calculation unit 15B, the separation unit 15C, the correction unit 15D, the foreground generation unit 15E, the background generation unit 15F, and the rendering unit 15G described in the first embodiment. A background model generation program 170a that exhibits the function of is stored. This background model generation program 170a is integrated like each component of the acquisition unit 15A, the calculation unit 15B, the separation unit 15C, the correction unit 15D, the foreground generation unit 15E, the background generation unit 15F, or the rendering unit 15G shown in FIG. Or you may separate. In other words, the HDD 170 does not necessarily need to store all the data described in the first embodiment, and the data used for the processing may be stored in the HDD 170.

Under such an environment, the CPU 150 reads out the background model generation program 170a from the HDD 170 and expands it in the RAM 180. As a result, the background model generation program 170a functions as the background model generation process 180a, as shown in FIG. The background model generation process 180a expands various data read from the HDD 170 in the area allocated to the background model generation process 180a in the storage area of the RAM 180, and executes various processes using the expanded data. .. For example, as an example of the process executed by the background model generation process 180a, the processes shown in FIGS. 11 and 14 are included. Note that in the CPU 150, not all the processing units described in the above-described first embodiment may operate, and the processing unit corresponding to the processing to be executed may be virtually realized.

Note that the background model generation program 170a described above does not necessarily have to be stored in the HDD 170 or the ROM 160 from the beginning. For example, the background model generation program 170a is stored in a “portable physical medium” such as a flexible disk, a so-called FD, a CD-ROM, a DVD disk, a magneto-optical disk, an IC card, which is inserted into the computer 100. Then, the computer 100 may acquire the background model generation program 170a from these portable physical media and execute it. Further, the background model generation program 170a is stored in another computer or a server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 executes the background model generation program 170a from these. It may be acquired and executed.

The following supplementary notes will be further disclosed regarding the embodiments including the above-described examples.

(Supplementary Note 1) An acquisition unit that acquires a camera image captured by a camera from a predetermined imaging position,
A calculation unit that calculates a depth image corresponding to the imaging position,
A separation unit that separates a plurality of pixels included in the camera image into a foreground and a background;
A correction unit that corrects the depth value of each pixel of the depth image by using the depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image,
Using a new depth image based on the depth value of each pixel of the depth image corrected by the correction unit, a background generation unit that generates a background model related to the background,
A background model generation device comprising:

(Supplementary Note 2) The background model generation according to Supplementary note 1, wherein the correction unit sets an invalid value to a depth value of each pixel separated into the foreground among a plurality of pixels included in the depth image. apparatus.

(Supplementary Note 3) The correction unit sets, for each of a plurality of pixels included in the depth image, a depth value of a pixel separated into the background out of the depth value and the depth values of surrounding pixels to a predetermined filter. The background model generation device according to appendix 2, wherein the depth value of each pixel separated into the background is corrected by convolving based on a coefficient.

(Supplementary Note 4) The correction unit sets, for each of a plurality of pixels included in the depth image, a depth value of a pixel separated in the background, among depth values in a target frame to be corrected and depth values in a past frame. 3. The background model generation device according to attachment 2, wherein the depth value of each pixel separated into the background is corrected by convolving based on a filter coefficient set in a predetermined filter.

(Supplementary Note 5) The separation unit separates a plurality of pixels included in the camera image into a foreground and a background based on the depth image,
The background model according to appendix 1, wherein the correction unit corrects the depth value of each pixel of the depth image by using the depth value of each pixel separated into the background based on the depth image. Generator.

(Supplementary Note 6) The separation unit separates each pixel separated into the background among a plurality of pixels included in the camera image into a background subcategory group into which the background category is further divided,
The correction unit, the depth value of the first pixel included in the depth image, the depth value of the second pixel separated into the same background sub-category as the background sub-category from which the first pixel is separated. The background model generation device according to appendix 1, wherein the background model generation device is configured to perform correction.

(Supplementary note 7) The background model according to Supplementary note 1, wherein the calculation unit calculates the depth image based on a parallax between the camera image and another camera image whose imaging position is different from that of the camera image. Generator.

(Supplementary Note 8) A camera image captured by the camera is acquired from a predetermined image capturing position,
Calculating a depth image corresponding to the imaging position,
Separating a plurality of pixels included in the camera image into a foreground and a background,
Using the depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image, the depth value of each pixel of the depth image is corrected,
A background model relating to the background is generated using a new depth image based on the depth value of each pixel of the corrected depth image,
A background model generation method characterized in that a computer executes the processing.

(Supplementary note 9) The background model according to Supplementary note 8, wherein in the correction process, an invalid value is set as a depth value of each pixel separated into the foreground among a plurality of pixels included in the depth image. Generation method.

(Supplementary Note 10) In the correction process, for each of a plurality of pixels included in the depth image, the depth value of the pixel separated into the background among the depth value and the depth values of the peripheral pixels is set in a predetermined filter. 10. The background model generation method according to appendix 9, wherein the depth value of each pixel separated into the background is corrected by convolving based on a filter coefficient.

(Supplementary Note 11) In the correction processing, for each of a plurality of pixels included in the depth image, the depth value of the pixel separated into the background among the depth value in the attention frame to be corrected and the depth values in the past frames 10. The background model generation method according to supplementary note 9, wherein the depth value of each pixel separated into the background is corrected by convoluting the above with a filter coefficient set in a predetermined filter.

(Supplementary Note 12) In the separation processing, a plurality of pixels included in the camera image are separated into a foreground and a background based on the depth image,
The background according to appendix 8, wherein the correction process corrects the depth value of each pixel of the depth image by using the depth value of each pixel separated into the background based on the depth image. Model generation method.

(Supplementary Note 13) In the separation processing, each pixel of the plurality of pixels included in the camera image, which is separated into the background, is separated into a background subcategory group into which the background category is further divided,
The correction process, the depth value of the first pixel included in the depth image, the depth value of the second pixel separated into the same background sub-category as the background sub-category from which the first pixel is separated The background model generation method according to Supplementary Note 8, wherein the background model generation method is characterized in that

(Supplementary Note 14) The background according to Supplementary Note 8, wherein the calculation process calculates the depth image based on a parallax between the camera image and another camera image whose imaging position is different from that of the camera image. Model generation method.

(Supplementary Note 15) A camera image captured by a camera is acquired from a predetermined image capturing position,
Calculating a depth image corresponding to the imaging position,
Separating a plurality of pixels included in the camera image into a foreground and a background,
Using the depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image, the depth value of each pixel of the depth image is corrected,
A background model relating to the background is generated using a new depth image based on the depth value of each pixel of the corrected depth image,
A background model generation program characterized by causing a computer to execute processing.

(Supplementary note 16) The background model according to Supplementary note 15, wherein in the correction process, an invalid value is set as a depth value of each pixel separated into the foreground among a plurality of pixels included in the depth image. Generator.

(Supplementary Note 17) In the correction process, for each of a plurality of pixels included in the depth image, the depth value of the pixels separated into the background among the depth value and the depth values of the peripheral pixels is set in a predetermined filter. 17. The background model generation program according to appendix 16, wherein the depth value of each pixel separated into the background is corrected by convolving based on a filter coefficient.

(Supplementary Note 18) In the correction process, for each of a plurality of pixels included in the depth image, the depth value of the pixel separated into the background among the depth value in the target frame to be corrected and the depth values in the past frames. 17. The background model generation program according to appendix 16, wherein the depth value of each pixel separated into the background is corrected by convoluting the above with a filter coefficient set in a predetermined filter.

(Supplementary Note 19) The separation processing separates a plurality of pixels included in the camera image into a foreground and a background based on the depth image,
16. The background according to appendix 15, wherein the correction process corrects the depth value of each pixel of the depth image by using the depth value of each pixel separated into the background based on the depth image. Model generator.

(Supplementary Note 20) In the separating process, each pixel of the plurality of pixels included in the camera image, which is separated into the background, is separated into a background subcategory group into which the background category is further divided,
The correction process, the depth value of the first pixel included in the depth image, the depth value of the second pixel separated into the same background sub-category as the background sub-category from which the first pixel is separated The background model generation program according to attachment 15, wherein the background model generation program is corrected by using

1 image generation system 3fg, 3bgs, 3bgd subject 5A to 5N camera 10 server device 11 communication I/F unit 13 storage unit 15 control unit 15A acquisition unit 15B calculation unit 15C separation unit 15D correction unit 15E foreground generation unit 15F background generation unit 15G Rendering unit 30 Client terminal

Claims (8)

An acquisition unit that acquires a camera image captured by the camera from a predetermined imaging position,
A calculation unit that calculates a depth image corresponding to the imaging position,
A separation unit that separates a plurality of pixels included in the camera image into a foreground and a background;
A correction unit that corrects the depth value of each pixel of the depth image by using the depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image,
Using a new depth image based on the depth value of each pixel of the depth image corrected by the correction unit, a background generation unit that generates a background model related to the background,
A background model generation device comprising: The background model generation device according to claim 1, wherein the correction unit sets an invalid value to a depth value of each pixel separated into the foreground among a plurality of pixels included in the depth image. The correction unit, for each of a plurality of pixels included in the depth image, the depth value of the pixels separated into the background among the depth values and the depth values of peripheral pixels based on a filter coefficient set in a predetermined filter. The background model generation device according to claim 1, wherein the depth value of each pixel separated into the background is corrected by convolution. The correction unit sets, for each of a plurality of pixels included in the depth image, a depth value of a pixel separated into the background among depth values in a target frame to be corrected and depth values in a past frame to a predetermined filter. The background model generation device according to claim 1, wherein the depth value of each pixel separated into the background is corrected by performing convolution based on the set filter coefficient. The separation unit separates a plurality of pixels included in the camera image into a foreground and a background based on the depth image,
5. The correction unit corrects the depth value of each pixel of the depth image by using the depth value of each pixel separated into the background based on the depth image. The background model generation device according to any one of the above. The separating unit separates each pixel separated into the background among a plurality of pixels included in the camera image into a background subcategory group into which the background category is further divided,
The correction unit, the depth value of the first pixel included in the depth image, the depth value of the second pixel separated into the same background sub-category as the background sub-category from which the first pixel is separated. The background model generation apparatus according to claim 1, wherein the background model generation apparatus corrects the background model. Acquire the camera image taken by the camera from the predetermined imaging position,
Calculating a depth image corresponding to the imaging position,
Separating a plurality of pixels included in the camera image into a foreground and a background,
Using the depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image, the depth value of each pixel of the depth image is corrected,
A background model relating to the background is generated using a new depth image based on the depth value of each pixel of the corrected depth image,
A background model generation method characterized in that a computer executes the processing. Acquire the camera image taken by the camera from the predetermined imaging position,
Calculating a depth image corresponding to the imaging position,
Separating a plurality of pixels included in the camera image into a foreground and a background,
Using the depth value of each pixel of the depth image corresponding to each pixel separated into the background in the camera image, the depth value of each pixel of the depth image is corrected,
A background model relating to the background is generated using a new depth image based on the depth value of each pixel of the corrected depth image,
A background model generation program characterized by causing a computer to execute processing.