JP2023127450A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide a technology for improving the accuracy of interpolating a distance information image.SOLUTION: An image processing device obtains a photographed image obtained by a photographing device at a first frame rate, and a distance information image generated at a second frame rate lower than the first frame rate and formed of a pixel value based on a distance to a subject. The image processing device generates a distance information image corresponding to a to-be-interpolated frame time which is a frame time of the first frame rate but is not a frame time of the second frame rate, on the basis of the obtained distance information image and a photographed image corresponding to the to-be-interpolated frame time.SELECTED DRAWING: Figure 1B

Description

The present disclosure relates to an image processing device, an image processing method, and a program.

BACKGROUND ART Recently, a technology that generates an image from a virtual viewpoint (hereinafter referred to as a virtual viewpoint image) using a plurality of images obtained by synchronously capturing images from a plurality of cameras from different positions has been attracting attention. According to this technology, for example, highlight scenes of soccer or basketball games can be viewed and viewed from various angles, so it is possible to provide the user with a higher sense of realism compared to normal images. A virtual viewpoint image based on a plurality of images is generated by collecting images taken by a plurality of cameras in an image processing unit such as a server, and performing processing such as rendering in the image processing unit. The generated virtual viewpoint image is transmitted to the user terminal and viewed.

JP2017-092545A

Various techniques have been developed for generating such virtual viewpoint images. For example, there is a technology that separates the foreground and background from a captured image, calculates three-dimensional coordinates from a plurality of captured images, and generates a virtual viewpoint image. Since the accuracy of foreground image separation is related to the image quality of the virtual viewpoint image, improving the separation accuracy of the foreground image is one of the important issues in generating the virtual viewpoint image. In order to improve the separation accuracy of foreground images, the use of distance information images that are composed of pixel values based on the distance to the subject is being considered. The distance information image is, for example, a distance image (depth image) whose pixel value is the distance to the subject, a foreground image based on distance information representing a foreground region extracted from the depth image, or the like. However, since the frame rate of the distance information image is generally lower than the frame rate of the photographed image, it is not possible to use the distance information at the same time as the luminance information at the time when there is a shortage of frames of distance information. Patent Document 1 discloses a technique of interpolating a frame of missing distance information at a time using past distance information based on a frame rate ratio. However, the technique of Patent Document 1 does not estimate block movement like block matching, and cannot obtain satisfactory interpolation accuracy.

An object of the present disclosure is to provide a technique that improves the interpolation accuracy of distance information images.

An image processing device according to an aspect of the present disclosure includes a first acquisition unit that acquires a photographed image photographed by a photographing device at a first frame rate, and a second frame rate that is slower than the first frame rate. a distance information image acquired by the second acquisition means, and a frame time of the first frame rate; and a generation means for generating a distance information image corresponding to the interpolation target frame time based on a photographed image corresponding to an interpolation target frame time which is not a frame time of the second frame rate.

According to the present disclosure, the interpolation accuracy of distance information images is improved.

1A is a diagram showing an example of the configuration of an image processing system according to the first embodiment, and FIG. 1B is a block diagram showing an example of the hardware configuration of the image processing apparatus according to the first embodiment. FIG. 1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a first embodiment. FIG. 3 is a diagram illustrating frame rate complementation of the distance information foreground mask according to the first embodiment. FIG. 3 is a diagram illustrating a method for determining an interpolated image according to the first embodiment. 5 is a flowchart showing a frame rate conversion procedure according to the first embodiment. FIG. 3 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. FIG. 7 is a diagram illustrating frame rate complementation of distance information according to the second embodiment. FIG. 7 is a diagram illustrating a motion vector determination method according to the second embodiment. 7 is a flowchart showing a frame rate conversion procedure according to the second embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

[First embodiment]
Hereinafter, a configuration will be described in which distance information images are interpolated with high precision using a method (block matching method) of detecting corresponding positions between images in units of blocks and estimating a movement destination. Furthermore, in interpolation of distance information images, it is difficult to distinguish between objects with similar shapes, which may reduce the accuracy of block matching in interpolation. For example, in an image in which a plurality of objects with similar shapes move at periodic intervals, the destination of a block may be estimated incorrectly, resulting in a decrease in interpolation accuracy. In this embodiment, a configuration that can interpolate distance information images with high accuracy even in such a situation will be described. Note that in the present disclosure, a distance information image refers to an image configured of pixel values based on the distance to a subject. Examples of such distance information images include a depth image in which each pixel value represents the distance to a subject, a foreground image (a foreground image based on distance information) representing a foreground region extracted from the depth image, and the like. In the first embodiment, the interpolation accuracy is improved for a foreground image (hereinafter referred to as a distance information foreground mask) in which the foreground area is represented by 1 and other areas are represented by 0, which is an example of a foreground image based on distance information. Explain the technology.

FIG. 1A (a) is a diagram showing a configuration example of an image processing system according to the first embodiment. The camera group 190 is composed of a plurality of cameras that capture images based on brightness information. Note that when an image representing brightness information of a single color is photographed, the photographed image is a monochrome image. For example, when an image representing brightness information corresponding to three colors of RGB is photographed, the photographed image is a color image. The distance measuring device group 191 is composed of a plurality of distance measuring devices that acquire depth images. The image processing device 100 generates a virtual viewpoint image based on the photographed image taken by the camera group 190 and the depth image acquired by the distance measuring device group 191. The display device 192 includes, for example, a liquid crystal display, and displays the virtual viewpoint image generated by the image processing device 100.

The hardware configuration of the image processing device 100 will be explained using (b) of FIG. 1A. The image processing device 100 includes a CPU 111 , a ROM 112 , a RAM 113 , an auxiliary storage device 114 , a display control section 115 , an operation section 116 , a communication I/F 117 , and a bus 118 .

The CPU 111 implements each function of the image processing apparatus 100 by controlling the entire image processing apparatus 100 using computer programs and data stored in the ROM 112 and RAM 113. Note that the image processing device 100 may include one or more dedicated hardware different from the CPU 111, and the dedicated hardware may execute at least part of the processing by the CPU 111. Examples of specialized hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 112 stores programs that do not require modification. The RAM 113 temporarily stores programs and data supplied from the auxiliary storage device 114, data supplied from the outside via the communication I/F 117, and the like. The auxiliary storage device 114 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The display control unit 115 is connected to the display device 192 and controls the display device 192 to display a GUI (Graphical User Interface) for a user to operate the image processing device 100, a virtual viewpoint image, and the like. The operation unit 116 includes, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 111 in response to user operations. The CPU 111 operates as a display control unit that controls the display control unit 115 and an operation control unit that controls the operation unit 116.

The communication I/F 117 is used for communication with an external device of the image processing apparatus 100. For example, when the image processing apparatus 100 is connected to an external device by wire, a communication cable is connected to the communication I/F 117. When the image processing device 100 has a function of wirelessly communicating with an external device, the communication I/F 117 includes an antenna. In this embodiment, each camera of the camera group 190 and the distance measuring device group 191 and the image processing device 100 are connected through the communication I/F 117. A bus 118 connects each part of the image processing apparatus 100 and transmits information.

Note that in this embodiment, it is assumed that the operation unit 116 exists inside the image processing apparatus 100, but the operation unit 116 may exist outside the image processing apparatus 100 as a separate device. Further, the display device 192 may be located inside the image processing device 100.

<Function description>
FIG. 1B is a block diagram showing an example of the functional configuration of the image processing apparatus 100 according to the first embodiment. Each functional unit may be realized by the CPU 111 as a processor executing a predetermined program, may be realized by dedicated hardware, or may be realized by cooperation between a processor that executes a program and dedicated hardware. It may also be realized by working.

The image processing device 100 includes a storage section 101, a background generation section 102, a first separation section 103, a video generation section 104, a storage section 105, a second separation section 106, an interpolation section 107, a determination section 108, a correction section 109, and a control section. 110. Note that image data in which a captured image from the camera group 190 and a depth image from the distance measuring device group 191 are paired is input to the image processing device 100. The photographed image is an image composed of, for example, RGB color information (luminance information). A depth image is an image whose pixel values are distance information representing the distance to the subject. Depth image acquisition methods that can be used by the distance measuring device group 191 include, but are not limited to, methods using LiDAR (TOF), infrared dot pattern irradiation, ultrasound, and the like. Note that in this embodiment, the photographed image and the depth image are acquired from separate devices, but a camera that can acquire both the photographed image and the depth image may be used.

The storage unit 101 stores captured images input from the camera group 190 in a frame memory. The background generation unit 102 generates a background image from the photographed images stored in the frame memory using a sequential background update method. Specifically, the background generation unit 102 identifies a moving subject as a foreground and a stationary area as a background from a plurality of captured images, and generates a background image by extracting only the background. The first separation unit 103 uses the photographed image and the background image generated by the background generation unit 102 to generate a binary image for specifying the foreground area by a background subtraction method. Hereinafter, this binary image will be referred to as a color information foreground mask. At this time, the first separation unit 103 generates a color information foreground mask by, for example, binarizing the difference image obtained by subtracting the background image from the photographed image using a predetermined threshold. In the color information foreground mask according to this embodiment, "1" indicates a foreground region and "0" indicates a background region, but the reverse may be possible.

The storage unit 105 stores the depth image input from the distance measuring device group 191 in a frame memory. The second separation unit 106 uses the depth image to generate a binary image (hereinafter referred to as a distance information foreground mask) in which the foreground region is identified as "1" and the background region as "0". The second separation unit 106, for example, obtains a background depth image in advance, and generates a distance information foreground mask by binarizing the difference between the input depth image and the background depth image using a predetermined threshold. . Note that the method for acquiring the background depth image is not limited to the above method. For example, it may be obtained by applying a sequential background update method to the depth image. Note that if the resolution of the depth image is different from the resolution of the captured image, resolution conversion is performed on the depth image to match the resolution of the captured image. This resolution conversion may be performed on the depth image or may be performed on the distance information foreground mask. As an interpolation method used for resolution conversion, filter calculations such as nearest neighbor, bilinear, bicubic, etc. can be used.

The interpolation unit 107 converts the frame rate of the distance information foreground mask by interpolating the distance information foreground mask. Details of frame rate conversion by the interpolation unit 107 will be described later using FIG. 2. The determination unit 108 determines an error that occurs when the interpolation unit 107 converts the rate of the distance information foreground mask using the photographed image and the background image. The correction unit 109 corrects the error determined by the determination unit 108 in the distance information foreground mask. The details of the processing by the determination unit 108 and the details of the processing by the correction unit 109 will be described later using FIG. 3. The control unit 110 optimizes the binarization threshold used in the background subtraction method executed by the first separation unit 103 using a distance information foreground mask having the same frame rate as the captured image. Specifically, the control unit 110 lowers the binarization threshold for identifying the foreground region using the background subtraction method in the region of the distance information foreground mask. In this way, by lowering the binarization threshold for the region estimated to be the foreground from the distance information, it is possible to prevent the foreground region from being missing without detecting noise in the background region. For example, when the pixel values of the foreground region and the background region are close, this optimization can prevent the foreground region from being missing.

The video generation unit 104 generates a virtual viewpoint image using the photographed image, the background image, and the color information foreground mask. For example, the video generation unit 104 projects the background image generated by the background generation unit 102 onto a two-dimensional plane, thereby generating images necessary for generating a virtual viewpoint image corresponding to a viewpoint (virtual viewpoint) specified by the user. Generate a background. For example, the video generation unit 104 extracts from the captured image a region of the captured image that corresponds to the foreground region (pixel group with a pixel value of “1”) of the color information foreground mask generated by the first separation unit 103 as a foreground region. . Then, the video generation unit 104 generates three-dimensional shape data (three-dimensional model) of the foreground based on the extracted foreground region. The video generation unit 104 generates a virtual viewpoint image by mapping a texture corresponding to a viewpoint specified by the user to the generated three-dimensional model and projecting the texture onto the two-dimensional plane.

<Overview of frame rate conversion>
A method of converting the frame rate of distance information in this embodiment will be explained using FIG. 2. In this embodiment, the frame rate of the captured image is 30 Hz, and the frame rate of the depth image is 15 Hz. However, the frame rate ratio in this embodiment is just an example, and other ratios may be used.

In FIG. 2, 2a, 2b, and 2c are photographed images input from the camera group 190, and are three frames of photographed images acquired at 30Hz. The subject 201 is moving from left to right within the captured images over three frames of captured images 2a, 2b, and 2c. 2d and 2e are depth images input from the distance measuring device group 191, and two frames acquired at 15 Hz are shown. Since the frame rate ratio of the photographed image and the depth image is 2:1, there is no depth image corresponding to frame time F(n+1) of the photographed image 2b. Note that the subject 202 is moving within the image from left to right over two frames of the depth images 2d and 2e.

2f and 2g are distance information foreground masks in which the foreground region is "1" and the background region is "0", and are generated by the second separation unit 106 from the depth images 2d and 2e. The distance information foreground mask 2f includes a foreground region 203, and the distance information foreground mask 2g includes a foreground region 204. The control unit 110 uses these distance information foreground masks 2f and 2g to specify the foreground region of the photographed image to be controlled by the binarization threshold. However, since there is no distance information foreground mask corresponding to the captured image 2b at frame time F(n+1), the control unit 110 cannot specify the foreground region at frame time F(n+1) from the distance information. Therefore, it is necessary to interpolate the distance information foreground mask corresponding to frame time F(n+1). The frame time (F+1) is a frame time at which the distance information foreground mask is interpolated, and is one of the interpolation target frame times.

2h is an interpolated image corresponding to the distance information foreground mask at frame time F(n+1), which is generated by the interpolation unit 107 by interpolating using the distance information foreground mask obtained from the plurality of input depth images. . By using the interpolated image 2h, the frame rate of the distance information foreground mask becomes 30 Hz, which is the same as that of the captured image. The interpolation unit 107 generates an interpolated image by detecting a motion vector indicating in which direction and by how much the foreground region moves between frames. As a general motion vector detection method, for example, there is a block matching method. An example of processing by the interpolation unit 107 by applying the block matching method will be described. First, the distance information foreground mask 2f (reference image) of the frame time F(n) to be processed is divided into predetermined sizes to generate reference blocks. Next, in the distance information foreground mask 2g of frame time F(n+2) temporally later than frame time F(n), a reference block of the same size as the reference block is extracted from the position corresponding to the reference block. do. Next, the interpolation unit 107 calculates the sum of absolute differences (SAD) for all corresponding pixels of the standard block and the reference block. The interpolation unit 107 repeatedly performs this SAD calculation while moving the reference block pixel by pixel within a predetermined search range set in the distance information foreground mask 2g. Then, the interpolation unit 107 detects the coordinates of the point where the SAD shows a minimum (minimum SAD) with respect to the starting point as a motion vector. The interpolation unit 107 uses the detected motion vector to obtain an interpolated distance information foreground mask (interpolated image 2h) at frame time F(n+1). Note that, among the SADs indicating a minimum value, an SAD having a value smaller than a threshold value may be used as the minimum SAD so that an unnecessarily large number of minimum SADs are not detected.

The interpolated image 2h includes a foreground region 205 interpolated from the foreground region 203 of the distance information foreground mask 2f and the foreground region 204 of the distance information foreground mask 2g using a block matching method. However, in the motion vector detection method using the block matching method, when a periodic pattern exists in the displayed image, if the size of the reference block is smaller than the periodic pattern, multiple positions where the SAD is minimum are detected. Put it away. When multiple positions are detected in this way, a motion vector cannot be detected correctly and an accurate interpolated image cannot be generated. A method for solving this problem will be explained next.

<Overview of interpolated image determination and correction method>
A method for determining validity and correcting a foreground region in an interpolated image generated by frame rate conversion in the first embodiment will be described with reference to FIG. 3.

In FIG. 3, 3a, 3b, and 3c are captured images input from the camera group 190, and are three-frame images acquired at 30Hz. The subject image 301 of the photographed image 3a is an image of subject A moving from left to right within the image over three frames. The subject image 302 of the photographed image 3b is an image in which the subject A has moved from the photographed image 3a in one frame period. The subject image 303 of the photographed image 3c is an image in which the subject A has moved from the photographed image 3a in a period of two frames. Further, the subject image 304 of the photographed image 3c is an image of subject B moving at a constant distance from subject A in a periodic pattern.

3d and 3e are distance information foreground masks generated by the second separation unit 106. In the distance information foreground mask 3d, a foreground region 311 corresponds to the subject image 301 of the photographed image 3a. In the distance information foreground mask 3d at frame time F(n), the reference block 314 is one of the reference blocks obtained by dividing the distance information foreground mask 3d at frame time F(n) into predetermined sizes, and includes the subject A. In the distance information foreground mask 3e at frame time F(n+2), the foreground region 312 corresponds to the subject image 303 of the photographed image 3c, and the foreground region 313 corresponds to the subject image 304 of the photographed image 3c. In the distance information foreground mask 3e, the reference block 315 is one of the reference blocks searched for the reference block 314 in the distance information foreground mask 3e and has the minimum SAD, and includes the subject B. Further, the reference block 316 of the distance information foreground mask 3e is one of the reference blocks searched for with respect to the reference block 314 and has the minimum SAD, and includes the subject A. In this way, in the example of FIG. 3, two reference blocks with patterns similar to the reference block 314 appear. That is, two motion vectors are detected: a motion vector indicating a slight movement to the left and a motion vector indicating a large movement to the right.

3f is an interpolated image of the distance information foreground mask corresponding to frame time F(n+1), generated by the interpolation unit 107. Frame time F(n+1) is a frame time when no distance information foreground mask exists, and is an interpolation target frame time. In this embodiment, an interpolated image is generated using all of the multiple detected motion vectors. Therefore, in the interpolated image 3f, the foreground region 321 is a foreground region where the subject A has moved to the right due to the detected rightward motion vector. Furthermore, in the interpolated image 3f, the foreground region 322 is a foreground region where the subject A has moved to the left due to the detected leftward motion vector.

3g, 3h, and 3i are background images generated by the background generation unit 102, respectively. 3j is a determination image indicating determination information used by the determination unit 108 to determine the validity of a plurality of generated foreground masks using a captured image and a background image. The determination unit 108 superimposes the captured image 3b at frame time F(n+1), which is the same time as the interpolation target frame time, and the interpolated image 3f, which is the interpolated distance information foreground mask, and determines the foreground area 321 of the interpolated image 3f. , 322 of the photographed image 3b are extracted. A pixel group 331 of the determination image 3j is a pixel group of the photographed image 3b that exists in an area overlapping with the foreground area 321 of the interpolated image 3f. Similarly, a pixel group 332 of the determination image 3j is a pixel group of the photographed image 3b that exists in an area overlapping with the foreground area 322 of the interpolated image 3f. Next, the determination unit 108 superimposes the background image 3h at frame time F(n+1) and the interpolated image 3f, and extracts a group of pixels of the background image 3h that overlap the foreground regions 321 and 322. The pixel group 334 is a pixel group of the background image 3h that exists in an area overlapping with the foreground area 321 of the interpolated image 3f. Furthermore, the pixel group 335 is a pixel group of the background image 3h that exists in an area overlapping with the foreground area 322 of the interpolated image 3f.

Next, the determination unit 108 compares the values of each pixel in the pixel groups 331 and 332 of the determination image 3j and the pixel groups 334 and 335 of the background image 3h. As a result of this comparison, the determining unit 108 determines that pixels whose pixel values have a difference exceeding a certain value are pixels in the foreground region in the corrected image 3h, and pixels whose pixel values have a difference below a certain value are not pixels in the foreground region. It is determined that In the example of FIG. 3, the pixel group of the foreground area 321 of the interpolated image 3f interpolated by the rightward motion vector remains, and the pixel group that constituted the foreground area 322 is determined to be not a mask, and the foreground area 322 is Excluded.

Here, the equation used by the determining unit 108 to determine whether to leave each pixel as a pixel in the foreground area is shown in the following equation (1). If the inequality in equation (1) is false, it is determined to be a "false detection (the pixel does not constitute the foreground region)", and if it is true, it is determined to be "correct (the pixel constitutes the foreground region)". .

・Ｂｇ（ｘ，ｙ）は、座標（ｘ，ｙ）に対応した背景画像の画素値を示す。
・Ｉｎ（ｘ，ｙ）は、座標（ｘ，ｙ）に対応した撮影画像の画素値を示す。
・Ｘは、後述するＭ（ｘ，ｙ）の画素値が「１」即ち前景である時のｘ座標値。
・Ｙは、後述するＭ（ｘ，ｙ）の画素値が「１」即ち前景である時のｙ座標値。
・Ｍ（ｘ，ｙ）は、座標（ｘ，ｙ）に対応した補間画像の画素値（二値）。
・αは、背景画像と撮影画像との類似の程度を示すパラメータであって、例えば０～５の値が設定される。

-Bg(x,y) indicates the pixel value of the background image corresponding to the coordinates (x,y).
-In(x,y) indicates the pixel value of the captured image corresponding to the coordinates (x,y).
-X is the x coordinate value when the pixel value of M(x, y), which will be described later, is "1", that is, the foreground.
- Y is the y coordinate value when the pixel value of M(x, y), which will be described later, is "1", that is, the foreground.
- M(x, y) is the pixel value (binary) of the interpolated image corresponding to the coordinates (x, y).
- α is a parameter indicating the degree of similarity between the background image and the photographed image, and is set to a value of 0 to 5, for example.

3k is a corrected interpolated image obtained as a result of the correction unit 109 correcting the interpolated image 3f based on the determination result by the determining unit 108. The correction unit 109 deletes the foreground area determined to be erroneously detected by the determination unit 108. In this way, the interpolated image 3k after correction becomes an image having only the foreground area 341. In addition, in the above interpolation process, if no movement of the subject is detected, an interpolated image is not generated using the motion vector, and one of the depth images before and after the interpolation target frame time is used as the interpolated image. It's okay. The case where the movement of the subject is not detected is, for example, the case where the amount of movement of the block is smaller than the threshold value.

<Processing procedure>
Next, the processing procedure of the image processing apparatus 100 according to the first embodiment will be described with reference to the flowchart in FIG. 4. The image processing device 100 starts the processing shown in FIG. 4 by receiving a captured image from the outside.

In S101, the background generation unit 102 generates a background image from the captured image using a sequential background update method. In S102, the image processing apparatus 100 determines whether a depth image corresponding to the frame time to be processed has been received. If a depth image corresponding to the frame time to be processed has been received (YES in S102), in S103, the second separation unit 106 generates a distance information foreground mask using the received depth image. The distance information foreground mask is a binary image that identifies the foreground region as "1" and the background region as "0". On the other hand, if the depth image corresponding to the frame time to be processed has not been received (NO in S102), in S104, the interpolation unit 107 interpolates the distance information foreground mask corresponding to the frame time (frame time to be interpolated). Generate an image. As a result, the frame rate of the distance information foreground mask is converted to the frame rate of the photographed image. Once the interpolated image is generated, the determining unit 108 and the correcting unit 109 perform correction processing on the interpolated image (S105 to S107).

In the loop that performs the processes of S105 to S107, a process of correcting the interpolated image generated in S104 based on the captured image at the interpolation target frame time and the background image is executed. First, in S105, the determination unit 108 determines whether the pixel value of the interpolated image to be processed is "1", that is, whether it is a pixel in the foreground area. If it is determined that the pixel is not in the foreground area (NO in S105), the process proceeds to the next pixel. If it is determined that the pixel is in the foreground area (YES in S105), the process proceeds to S106. In S106, the determination unit 108 determines whether the pixel is appropriate as a foreground region using the above-mentioned equation (1). If formula (1) is satisfied (YES in S106), the pixel is determined to be valid in the foreground region, and the process proceeds to the next pixel. On the other hand, if equation (1) is not satisfied (NO in S106), the pixel is determined to be inappropriate as a foreground region, and the process proceeds to S107. In S107, the correction unit 109 sets the value of the pixel currently being processed in the interpolated image to "0". The processes of S105 to S107 are repeated for all pixels in the foreground area existing in the interpolated image, and a corrected interpolated image is obtained.

In S108, the control unit 110 uses the distance information foreground mask corresponding to the frame time of the photographed image or the distance information foreground mask that is the corrected interpolated image obtained from the correction unit 109 to be used in the first separation unit 103. , optimize the binarization threshold of the background subtraction method. The first separation unit 103 uses the optimized binarization threshold to generate a binary image (color information foreground mask) for specifying a foreground region from the photographed image and the background image by a background subtraction method. In S109, the video generation unit 104 generates a virtual viewpoint image from an arbitrarily designated virtual viewpoint using the photographed image, the background image, and the color information foreground mask.

As described above, according to the first embodiment, a distance information mask corresponding to each frame time of a captured image can be obtained by detecting a motion vector and interpolating the distance information mask. Further, the validity of the interpolated distance information foreground mask is determined based on the photographed image corresponding to the interpolation target frame time and the background image generated from the photographed image, and is corrected as necessary. Therefore, even when the block matching method cannot accurately specify the destination of a block using distance information alone, the destination can be specified more accurately. Therefore, for example, even if a plurality of objects with similar shapes are moving at periodic intervals, it is possible to accurately estimate the destination of the distance information mask.

[Second embodiment]
In the first embodiment, the frame rate conversion of the distance information foreground mask is performed by interpolating the distance information foreground mask. However, the target of frame rate conversion is not limited to the distance information foreground mask, but may also be distance information (depth image) input from a distance measuring device. The values of the pixels forming the distance information foreground mask are binary values, whereas the values of the pixels of the depth image forming the distance information are multi-bit values. Therefore, by using distance information as the object of frame rate conversion processing, it can be expected that the search accuracy when searching for similar patterns using the block matching method will be improved. Further, in the first embodiment, the validity is determined for the interpolated distance information foreground mask, but the validity may be determined for the motion vector calculation result. In the second embodiment, a configuration will be described in which a determination is made on a motion vector and frame rate conversion is performed on distance information.

<Summary explanation of functions>
FIG. 5 is a schematic configuration diagram of an image processing apparatus 100 according to the second embodiment. In FIG. 5, functional blocks similar to those of the first embodiment (FIG. 1B) are given common reference numbers. In the image processing device 100 of the second embodiment, the interpolation section 500 is arranged between the storage section 105 and the second separation section 106. The interpolation unit 500 performs frame rate conversion on the depth image. Details of the frame rate conversion performed by the interpolation unit 500 will be described later using FIG. 6. The interpolation unit 500 includes a calculation unit 501, a motion vector generation unit 502, a determination unit 503, a correction unit 504, and a frame interpolation unit 505.

The calculation unit 501 performs the following processing using the block matching method. The calculation unit 501 divides the depth image (reference image) at frame time F(n) to be processed into predetermined sizes to generate reference blocks. Next, in the depth image (reference image) at a frame time F(n+2) temporally later than the reference image, the calculation unit 501 calculates a reference block having the same size as the reference block, starting from the position corresponding to the reference block. Extract. Next, the calculation unit 501 calculates the sum of absolute differences (SAD) for all corresponding pixels of the standard block and the reference block. The calculation unit 501 repeats this SAD calculation while moving the reference block pixel by pixel within a predetermined search range set in the reference image. The calculation unit 501 acquires coordinates with respect to the starting point of the reference block whose calculated SAD is minimum (minimum SAD). If there are multiple coordinates of the starting point of the reference block indicating the minimum SAD (hereinafter referred to as coordinates of the minimum SAD) due to the presence of a periodic pattern in the displayed image, the calculation unit 501 calculates the coordinates of the multiple coordinates. get. Note that, among the SADs indicating a minimum value, an SAD having a value smaller than a threshold value may be used as the minimum SAD so that an unnecessarily large number of minimum SADs are not detected.

The motion vector generation unit 502 generates a motion vector of the reference block in the interpolated image corresponding to frame time F(n+1), where the coordinates of the reference block are the starting point and the coordinates of the acquired minimum SAD are the end point. . This motion vector is generated for each reference block. The determination unit 503 determines the validity of the motion vector using a captured image and a background image obtained from the captured image. Details of the validity determination by the determination unit 503 will be described later using FIG. The correction unit 504 selects a correct vector from the motion vectors generated by the motion vector generation unit 502 based on the determination result of the determination unit 503. The frame interpolation unit 505 generates an interpolated image at frame time F(n+1) using the reference image at frame time F(n), the reference image at frame time F(n+2), and the selected motion vector.

<Overview of frame rate conversion>
A method of converting the frame rate of an image of distance information in the second embodiment will be described using FIG. 6.

In FIG. 6, 6a, 6b, and 6c are photographed images obtained from the camera group 190, and are three frame images obtained at 30Hz. The subject moves within the image from left to right over three frames of captured images 6a to 6c. 6d and 6e are two-frame depth images input at 15 Hz from the distance measuring device group 191. The frame time F(n+1) at which no depth image exists is the interpolation target frame time. 6f is an interpolated image generated by the interpolation unit 500 using two frames of depth images. The interpolated image 6f is an interpolated image corresponding to frame time F(n+1), which is the interpolation target frame time. By generating a distance information foreground mask from the acquired depth image and interpolated image, the frame rate of the distance information foreground mask becomes 30 Hz, which is the same as that of the captured image. 6g, 6h, and 6i are distance information foreground masks generated by the second separation unit 106. The distance information foreground masks 6g and 6i are generated from the depth images 6d and 6e, and the distance information foreground mask 6h is generated from the interpolated image 6f. The distance information foreground mask is a binary image in which the subject, that is, the foreground region is set to "1" and the background region is set to "0". Note that if no movement of the subject is detected, the interpolation image may not be generated using the motion vector, and either depth images before or after the interpolation target frame time may be used as the interpolation image. Whether or not the subject has moved can be determined, for example, based on motion vectors detected from two depth images. For example, if the amount of movement represented by the motion vector is smaller than the threshold value, it is determined that no movement of the subject has been detected.

<Overview of motion vector correction>
An overview of correction of a plurality of generated motion vectors in the second embodiment will be described using FIG. 7.

7a, 7b, and 7c are three frames of captured images input from the camera group 190 at 30 Hz. The photographed images 7a to 7c are the same as the photographed images (FIG. 3) exemplified in the first embodiment. 7d and 7e are two-frame depth images input from the distance measuring device group 191 at 15 Hz. The reference block 701 in the depth image 7d is an example of a reference block set in the reference image for motion vector calculation. A reference block 702 and a reference block 703 shown in the depth image 7e are examples of reference blocks on the reference image in which minimal SAD is detected with respect to the reference block 701. These standard blocks and reference blocks are calculated and generated by the calculation unit 501.

The motion vector information 7f is information in which the motion vectors generated by the motion vector generation unit 502 are arranged at two-dimensional coordinates of the interpolated image. A position 711 indicates the position of the reference block 701 in the depth image 7d. The motion vector 712 represents the direction and amount of movement starting from the position 711, and indicates the movement direction and amount of movement of the reference block 701 at frame time F(n+1) when the reference block 701 moves to the position of the reference block 702 in the depth image 7e. Indicates the estimated destination. The reference area 713 indicates a state in which the reference block 701 has been moved by the motion vector 712. Similarly, the motion vector 714 represents the direction and amount of movement starting from the position 711, and the frame time F(n+1) of the reference block 701 when the reference block 701 moves to the reference block 703 of the depth image 7e. Indicates the estimated destination. A reference area 715 indicates a state in which the reference block 701 has been moved by the motion vector 714. In the illustrated example, two movement destinations (reference areas 713 and 715) are obtained for one reference block 701.

7g, 7h, and 7i are background images generated by the background generation unit 102. Background images 7g, 7h, and 7i are background images obtained from photographed images 7a, 7b, and 7c, respectively. In the background image 7h corresponding to frame time F(n+1), a pixel group 721 indicates a pixel group existing in the reference area 713, and a pixel group 722 indicates a pixel group existing in the reference area 715.

The determination information 7j is an illustration of information used by the determination unit 503 to determine the validity of the motion vector generated by the motion vector generation unit 502. The determination unit 503 superimposes the captured image 7b at frame time F(n+1), which is the interpolation target frame time, and the motion vector information 7f at frame time F(n+1), and determines whether the captured image 7b exists within the reference areas 713, 715. , specify the pixel group of the photographed image 7b. A pixel group 731 of the determination information 7j is a pixel group of the photographed image 7b that exists within the range of the reference area 713. A pixel group 732 of the determination information 7j is a pixel group of the photographed image 7b that exists within the reference area 715. Next, the determination unit 503 calculates the degree of similarity between the pixel group of the captured image 7b and the pixel group of the background image 7h in the detected area. For example, the determination unit 503 calculates the similarity between the pixel group 731 and the pixel group 721 and the similarity between the pixel group 732 and the pixel group 722. If a plurality of motion vectors exist for one reference block, the determination unit 503 determines the motion vector corresponding to the reference region for which the lowest degree of similarity has been calculated to be an appropriate motion vector (correct motion vector). . For example, since the pixel group 731 of the determination information 7j and the pixel group 721 of the background image 7h almost match, a high degree of similarity is calculated. On the other hand, since the pixel group 732 of the determination information 7j and the pixel group 722 of the background image 7h hardly match, a low degree of similarity is calculated. Therefore, the determining unit 503 determines that the rightward motion vector 714 is correct.

Here, an example of a method for calculating the degree of similarity by the determination unit 503 is shown in equation (2). The smaller the value of S calculated by Equation (2), the higher the degree of similarity. The determination unit 503 calculates the similarity S as described below for the reference blocks corresponding to the plurality of detected motion vectors, and determines the motion vector corresponding to the reference area with the largest calculated value, that is, the lowest similarity, as the correct answer. It is determined that the motion vector is

・Ｓは、背景画像と撮影画像に存在する双方の参照ブロックの類似度を示す。計算される値Ｓが大きい程、類似度が低い。
・Ｂｇ（Ｘｎ，Ｙｎ）は、座標（Ｘｎ，Ｙｎ）に対応した背景画像の画素値を示す。
・Ｉｎ（Ｘｎ，Ｙｎ）は、座標（Ｘｎ，Ｙｎ）に対応した撮影画像の画素値を示す。
・Ｘｎは、参照領域の範囲に存在するｎ番目の画素のＸ座標値である。
・Ｙｎは、参照領域の範囲に存在するｎ番目の画素のＹ座標値である。
・ｎは、参照領域の範囲に存在する画素の番号を示す。値は、０～Ｎｍａｘ－１（参照領域を構成する全画素数－１）の値をとる。例えば、参照領域が１６×１６のサイズの場合、Ｎｍａｘの値は２５６となり、ｎの値は０～２５５となる。

- S indicates the degree of similarity between both reference blocks existing in the background image and the photographed image. The larger the calculated value S, the lower the degree of similarity.
-Bg (Xn, Yn) indicates the pixel value of the background image corresponding to the coordinates (Xn, Yn).
-In(Xn, Yn) indicates the pixel value of the captured image corresponding to the coordinates (Xn, Yn).
-Xn is the X coordinate value of the n-th pixel existing in the range of the reference area.
- Yn is the Y coordinate value of the nth pixel existing in the range of the reference area.
-n indicates the number of pixels existing in the range of the reference area. The value takes a value from 0 to Nmax-1 (total number of pixels constituting the reference area-1). For example, if the reference area has a size of 16×16, the value of Nmax is 256, and the value of n is 0 to 255.

The corrected motion vector information 7k indicates the motion vector corrected by the correction unit 504. Here, the motion vectors are visualized and shown by arranging them on a two-dimensional coordinate diagram. The correction unit 504 obtains a corrected motion vector by selecting the motion vector determined to be correct by the determination unit 503 from among the motion vectors generated by the motion vector generation unit 502. The motion vector 741 of the corrected motion vector information 7k is a motion vector selected by the correction unit 504, and corresponds to the motion vector 714 of the motion vector information 7f.

<Processing procedure>
Next, the processing procedure of the image processing apparatus 100 according to the second embodiment will be described with reference to the flowchart in FIG. 8. The image processing device 100 starts processing by receiving a captured image from the outside. S101, S102, S108, and S109 are the same processes as in the first embodiment (FIG. 4).

In S201, the interpolation unit 500 determines whether to generate an interpolated image of distance information (depth image) for the current frame time to be processed. If a depth image exists at the frame time to be processed, it is determined not to generate an interpolated image (NO in S201), and the process proceeds to S103. If no depth image exists at the frame time to be processed, it is determined that the frame time is the interpolation target frame time (YES in S201), and the process proceeds to S202. In S202, the calculation unit 501 detects the coordinates of the minimum SAD from the depth images before and after the frame time to be processed. In S203, the motion vector generation unit 502 generates motion vectors for all reference blocks set in the reference image.

In S204, the determination unit 503 determines whether a plurality of motion vectors have been generated for each reference block. If it is determined that a plurality of motion blocks have been generated (YES in S204), the process proceeds to S205 in order to correct the generated motion vector. If it is determined that only one motion vector has been generated (NO in S204), the process proceeds to the next reference block. The processing in S205 to S209 is a process of correcting a motion vector using the captured image at the interpolation target frame time and the background image. In S205, the determination unit 503 extracts a group of pixels of the background image existing in a range corresponding to the reference area after moving the reference block according to the motion vector. In S206, the determination unit 503 extracts a group of pixels of the photographed image existing in a range corresponding to the reference area. In S207, the determination unit 503 calculates the degree of similarity between the pixel group of the background image extracted in S205 and the pixel group of the captured image extracted in S206 using equation (2). The determination unit 503 repeats the processes of S205 to S207 described above for all reference blocks corresponding to the motion vector generated in S203. When the processing for all reference blocks is completed, the processing advances to S208.

In S208, the determination unit 503 determines the reference region from which the smallest degree of similarity is obtained among the degrees of similarity calculated by the above process. In S209, the correction unit 504 selects the motion vector with the smallest degree of similarity as the correct motion vector of the reference block being processed. The processing of S204 to S209 is repeated for all the set reference blocks, and when the processing for all the reference blocks is completed, the processing proceeds to S210. In S210, the frame interpolation unit 505 represents the depth image at frame time F(n+1) using the depth image at frame time F(n), the depth image at frame time F(n+2), and the corrected motion vector. Generate an interpolated image. In S103, the second separation unit 106 generates a distance information foreground mask from the depth image or the interpolated image.

As described above, according to the second embodiment, when interpolating a depth image, the validity of a motion vector is determined using an image taken at the same time as the interpolation target frame time, thereby improving the accuracy of the motion vector. The motion vector can be estimated correctly. Thereby, even when a plurality of objects with similar shapes are moving at periodic intervals, it is possible to accurately interpolate the depth image. Note that in the second embodiment, the motion vector is corrected in the interpolation of the depth image, but the motion vector may be corrected as described above in the interpolation of the distance information foreground mask. That is, instead of the pixel-by-pixel correction of the distance information foreground mask described in the first embodiment, the above-described motion vector correction may be applied to the interpolation of the distance information foreground mask.

(Other embodiments)
The present disclosure provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: Storage unit, 102: Background generation unit, 103: First separation unit, 104: Video generation unit, 105: Storage unit, 106: Second separation unit, 107: Interpolation unit, 108: Determination unit, 109: Correction unit , 110: Control unit, 500: Interpolation unit, 501: Calculation unit, 502: Motion vector generation unit, 503: Determination unit, 504: Correction unit, 505: Frame interpolation unit

Claims (16)

a first acquisition means for acquiring a photographed image photographed at a first frame rate by the photographing device;
a second acquisition unit that acquires a distance information image that is generated at a second frame rate that is slower than the first frame rate and is composed of pixel values based on the distance to the subject;
Based on a distance information image acquired by the second acquisition means and a photographed image corresponding to an interpolation target frame time that is a frame time of the first frame rate but not a frame time of the second frame rate. An image processing apparatus comprising: generating means for generating a distance information image corresponding to the interpolation target frame time. The image processing apparatus according to claim 1, wherein the distance information image is a foreground image representing a foreground region extracted from a depth image in which each pixel represents a distance to the subject. The generating means generates a foreground image at the interpolation target frame time from foreground images before and after the interpolation target frame time, and corrects the generated foreground image based on the captured image at the interpolation target frame time. The image processing device according to claim 2. further comprising background generation means for generating a background image for separating a foreground region from the photographed image based on the photographed image,
The generation means generates a pixel value between a photographed image corresponding to the interpolation target frame time and a background image in an area corresponding to a foreground area represented by the distance information image at the interpolation target frame time. 4. The image processing apparatus according to claim 3, wherein the foreground region represented by the foreground image is corrected. 5. The image processing apparatus according to claim 4, wherein the generating means excludes a region of pixels in which the difference is smaller than a threshold value from a foreground region represented by the generated foreground image. The generating means generates a motion vector for each area based on foreground images before and after the interpolation target frame time, and uses the generated motion vectors to generate a foreground image at the interpolation target frame time. The image processing apparatus according to any one of claims 2 to 5. The generating means is
Generating a motion vector for each area based on foreground images before and after the interpolation target frame time, and using the generated motion vectors, generating a foreground image at the interpolation target frame time;
10. When a plurality of motion vectors are generated for one region, one motion vector is selected from the plurality of motion vectors based on a captured image corresponding to the interpolation target frame time. 2. The image processing device according to 2. When the amount of movement indicated by the motion vector of each region in the foreground images before and after the interpolation target frame time is smaller than a threshold value, the generating means generates one of the foreground images before and after the interpolation target frame time. 8. The image processing apparatus according to claim 6, wherein the image processing apparatus is used as a foreground image at an interpolation target frame time. The image processing apparatus according to claim 1, wherein the distance information image is a depth image in which each pixel represents a distance to the subject. The generating means is
Generate a motion vector for each region from depth images before and after the interpolation target frame time, and use the generated motion vectors to generate a depth image at the interpolation target frame time;
10. When a plurality of motion vectors are generated for one region, one motion vector is selected from the plurality of motion vectors based on a captured image corresponding to the interpolation target frame time. 9. The image processing device according to 9. further comprising background generation means for generating a background image for separating a foreground region from the photographed image based on the photographed image,
The generating means calculates, for each of the plurality of motion vectors, a difference in pixel values of a group of pixels in an area associated with the motion vector in the photographed image and the background image corresponding to the interpolation target frame time, and calculates the difference in pixel values of a pixel group in an area associated with the motion vector. The image processing apparatus according to claim 10, wherein the motion vector for which the difference has been calculated is selected from the plurality of motion vectors. When the amount of movement indicated by the motion vector of each region in the depth images before and after the interpolation target frame time is smaller than a threshold value, the generating means generates one of the depth images before and after the interpolation target frame time. The image processing device according to claim 10 or 11, wherein the image processing device is used as a depth image of an interpolation target frame time. By generating a foreground image representing a foreground region from each of the depth image acquired by the second acquisition means and the depth image generated by the generation means, a foreground image corresponding to each frame time of the first frame rate is generated. The image processing apparatus according to any one of claims 9 to 12, further comprising a third acquisition means for acquiring a foreground image. further comprising a separating means for separating a foreground region from the photographed image by comparing a difference between the photographed image and a background image generated based on the photographed image with a threshold value;
Claims 2 to 8, wherein the separation means sets the threshold value differently between a foreground region indicated by a foreground image acquired at a frame time of a photographed image to be processed and another region. , 13. The image processing device according to any one of . a first acquisition step of acquiring a photographed image photographed at a first frame rate by a photographing device;
a second acquisition step of acquiring a distance information image composed of pixel values based on the distance to the subject, generated at a second frame rate slower than the first frame rate;
Based on the distance information image acquired in the second acquisition step and a photographed image corresponding to an interpolation target frame time that is a frame time of the first frame rate and not a frame time of the second frame rate. An image processing method comprising: a generation step of generating a distance information image corresponding to the interpolation target frame time. A program for causing a computer to function as each means of the image processing apparatus according to claim 1.

Images