JP7322235B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing apparatus, image processing method, and program, and is particularly suitable for use in generating a virtual viewpoint image.

When observing a subject (for example, an object such as a person) from a virtual viewpoint (an arbitrary viewpoint including a viewpoint at which no imaging device actually exists) based on images obtained by imaging the subject (for example, an object such as a person) with a plurality of imaging devices. Techniques for reconstructing (generating) obtained images (virtual viewpoint images) are known. Patent Document 1 discloses the following method. First, a three-dimensional model of the subject is generated using captured images of the subject captured by a plurality of cameras and position information of the cameras. Next, a texture image (blended texture image) at each position on the three-dimensional model is generated by blending texture images appearing in a plurality of captured images. Finally, the image from the virtual viewpoint is reconstructed by texture mapping the blended texture image onto the 3D model.

Japanese Patent No. 5011224

W.Matusik, C.Buehler, R.Raskar, S.Gortler, L.McMillan,"ImageBased Visual Hulls", ACM SIGGRAPH2000,pp.369-374, 2000

However, with the technique described in Patent Document 1, if the subject cannot be appropriately extracted from the captured image , there is a risk that the subject will be rendered as a large shape that is different from the actual shape . That is, the technique described in Patent Document 1 has a problem that it is not easy to appropriately generate a virtual viewpoint image in a captured image.
SUMMARY OF THE INVENTION The present invention has been made in view of such problems, and an object of the present invention is to appropriately generate a virtual viewpoint image in a captured image.

The image processing apparatus of the present invention extracts a region of a subject by a first extraction method from a captured image captured by any one of a plurality of imaging devices that capture images of an imaging region from a plurality of directions . and a region of the subject extracted from the captured image captured by the imaging device by a second extraction method different from the first extraction method, a three-dimensional image of the subject located within the imaging region . A first acquisition means for acquiring shape information that is information for specifying a shape, a second acquisition means for acquiring viewpoint information that is information for specifying a virtual viewpoint, the shape information, and the viewpoint and generating means for generating a virtual viewpoint image based on information, wherein the first extraction method includes, based on the captured image and a background image corresponding to the captured image, a method of extracting the subject, wherein the second extraction method is based on the captured image and another captured image captured at a timing different from the timing at which the captured image was captured by the imaging device; , and a method for extracting the subject .

According to the present invention, it is possible to appropriately generate a virtual viewpoint image in a captured image.

1 is a diagram showing the configuration of an image processing system; FIG. 2 is a diagram showing the hardware configuration of an image processing apparatus; FIG. 1 is a diagram illustrating a first example of a functional configuration of an image processing device; FIG. 4 is a flowchart for explaining a first example of an image processing method; FIG. 4 is a diagram for explaining a first example of the content of image processing; FIG. 4 is a diagram showing a second example of the functional configuration of the image processing device; 9 is a flowchart for explaining a second example of an image processing method; FIG. 10 is a diagram illustrating a second example of the content of image processing; FIG. 13 is a diagram showing a third example of the functional configuration of the image processing device; FIG. 11 is a flow chart illustrating a third example of an image processing method; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
<First Embodiment>
In the present embodiment, motion information of a subject (for example, an object such as a person) is acquired from a captured image, and based on the speed and imaging conditions, an area in which blurring occurs due to the movement of the subject in the captured image and a blurred area are detected. Identify areas that are not Then, an image when the subject is observed from the virtual viewpoint is generated (reconfigured) using a plurality of captured images so that the blurring area becomes translucent. The image processing system of this embodiment can be applied to a plurality of image data obtained by imaging the same subject from different viewpoints. In the following description, an image obtained by observing an object from a virtual viewpoint will be referred to as a virtual viewpoint image as necessary. In addition, blurring that occurs in at least a partial area of the subject in the captured image due to the movement of the subject is referred to as motion blurring as necessary.

FIG. 1 is a schematic diagram showing an example of the configuration of an image processing system. The image processing system has multiple cameras 101 , an image processing device 102 , a display device 103 and an input device 104 . The camera 101 captures images of the subject 105 from a plurality of viewpoints surrounding the subject 105 arranged in an area on a schematic plane. A display device 103 and an input device 104 are connected to the image processing device 102 . A user performs an input operation to the image processing apparatus 102 using the display device 103 and the input device 104 . Through this input operation, the user sets imaging conditions, confirms the result of processing the image data obtained by imaging with the camera 101, and the like.

FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus 102. As shown in FIG. The image processing apparatus 102 includes a CPU 201 , a RAM 202 , a ROM 203 , a storage section 204 , an input interface 205 , an output interface 206 and a system bus 207 . An external memory 208 is connected to the input interface 205 . A display device 103 is connected to the output interface 206 .
A CPU 201 is a processor that comprehensively controls each component of the image processing apparatus 102 . A RAM 202 is a memory that functions as a main memory and a work area for the CPU 201 . A ROM 203 is a memory that stores programs and the like used for processing in the image processing apparatus 102 . The CPU 201 uses the RAM 202 as a work area and executes programs stored in the ROM 203 to perform various processes described later.

A storage unit 204 is a storage device that stores image data used for processing in the image processing apparatus 102, parameters (that is, setting values) for the processing, and the like. An HDD, an optical disk drive, a flash memory, or the like can be used as the storage unit 204 .
The input interface 205 is, for example, a serial bus interface such as USB or IEEE1394. The image processing apparatus 102 can acquire image data to be processed from an external memory 208 (eg, hard disk, memory card, CF card, SD card, USB memory) via the input interface 205 . The output interface 206 is, for example, a video output terminal such as DVI or HDMI (registered trademark). The image processing apparatus 102 can output image data processed by the image processing apparatus 102 to a display device 103 (an image display device such as a liquid crystal display) via an output interface 206 . Note that the image processing apparatus 102 may include components other than those described above, but they are not the focus of the present invention, and detailed description thereof will be omitted.

An example of image processing in the image processing apparatus 102 of this embodiment will be described below with reference to FIGS. 3, 4, and 5. FIG. FIG. 3 is a block diagram showing an example of the functional configuration of the image processing apparatus 102 of this embodiment. FIG. 4 is a flowchart for explaining an example of the image processing method of this embodiment. FIG. 5 is a schematic diagram illustrating an example of the content of image processing according to this embodiment.
In this embodiment, the CPU 201 functions as each block shown in FIG. 3 by executing the program stored in the ROM 203, and executes the processing shown in the flowchart of FIG. Note that the CPU 201 does not necessarily have to execute all the functions of the image processing apparatus 102, and a processing circuit corresponding to each function may be provided in the image processing apparatus 102 and the processing circuit may execute the function.

FIG. 5 shows the details of the image processing, taking as an example a case where an image of a subject whose left hand is moving quickly is captured. Images representing the actual movement of the subject are obtained in the order of image 501A, image 501B, and image 501C. However, when the exposure time (Tv) of the camera 101 is long, the camera 101 captures an image such as an image 502 in which the left hand portion is motion-blurred.
In S<b>401 , the moving image data acquisition unit 301 acquires a plurality of pieces of moving image data from the external memory 208 via the input interface 205 and stores them in the RAM 202 . Each of the plurality of moving image data is image data obtained by imaging the same subject from different viewpoints with each camera 101, that is, moving image data representing the same subject from different viewpoints. In FIG. 5, an image 502 represents an example of moving image data.

Next, in S<b>402 , the background image acquisition unit 302 acquires a plurality of background image data corresponding to the moving image data acquired in S<b>401 from the external memory 208 via the input interface 205 and stores them in the RAM 202 . In FIG. 5, an image 503 represents an example of background image data (captured by a certain camera 101). It is assumed that the background image is an image captured by each camera 101 in a state where the object 105 does not exist, and is stored in the external memory 208 in advance. The position and posture of each camera 101 when capturing the background image is preferably the same as when the subject 105 exists.

Next, in S<b>403 , the first foreground/background separation unit 303 separates the moving image data into a foreground image and a background image based on the difference between the moving image data and the background image data stored in the RAM 202 and stores them in the RAM 202 . do. The first foreground/background separation unit 303 generates, for example, image data having the same size as the moving image data and the background image data, in which each pixel has a binary value, as the foreground/background image data. Then, when the absolute value of the difference between the pixel values of the pixels corresponding to each other between the moving image data and the background image data exceeds a threshold value, the first foreground/background separation unit 303 assigns white (1) to the pixel, and otherwise assigns white (1) to the pixel. For each pixel, black (0) is assigned to the pixel if it is the case. In this case, the area assigned white (1) is the foreground area, and the area assigned black (0) is the background area. The first foreground/background separation unit 303 stores such foreground/background image data in the RAM 202 together with the moving image data and the background image data. In FIG. 5, image 504 represents an example of foreground background image data.

Next, in S<b>404 , the moving object map calculation unit 304 calculates a moving object map of the moving image from the moving image data stored in the RAM 202 and stores the calculated moving object map in the RAM 202 . A moving object map is a map in which the amount of movement of the x and y coordinates of an object in each frame image with respect to the previous or next frame image is stored in the form of a map for each pixel. In order to reduce the amount of calculation, the moving body map calculating unit 304 may calculate the moving body map of only the foreground area by calculating the movement of only the foreground area stored in the RAM 202 .

Next, in S405, the motion blur amount calculation unit 305 calculates the motion blur amount based on the moving object map, the exposure time (Tv) [sec], and the frequency [fps (frame/sec)]. Save to Note that the frequency is the frame rate and is the frequency corresponding to the imaging cycle. In this embodiment, the amount of motion blur is represented by the number of pixels of the blur of the subject in the image. As a method for calculating the amount of motion blur, for example, when the amount of movement in each pixel of the moving body map is (x, y) [pixel/frame], the amount of motion blur calculation unit 305 calculates the amount of motion blur [pixel] as follows. It can be calculated by the formula (1).

In addition, the motion blur amount calculation unit 305 may store, in the RAM 202, a motion blur amount (numerical value) assigned in a map format to each pixel of an image for each camera 101, for example. Hereinafter, the motion blur amount saved in this map format will be referred to as a motion blur amount map as needed. In FIG. 5, in the motion blur amount map 505, the value corresponding to the fast moving left hand portion indicates a large motion blur amount.

Next, in S<b>406 , the second foreground/background separation unit 306 extracts a non-motion-blurring foreground area from the foreground/background image data, and stores it in the RAM 202 . Specifically, for example, the second foreground/background separation unit 306 is a foreground area (white area) in the foreground/background image data (image 504), and the value of the corresponding motion blur amount map 505 is A region (black region) that is equal to or less than a certain threshold value may be set as a non-motion-blurring foreground region. In FIG. 5, image 506 represents an example of a non-motion blurred foreground region. Thus, the non-motion-blurred foreground area is a foreground area in which motion blur does not occur. Among the foreground areas, areas other than the non-motion-blurred foreground area are areas in which motion blur occurs, where the foreground and the background are mixed and cannot be distinguished (motion-blurred area).

Next, in S407, the first shape estimation unit 307 estimates the shape of the foreground region. Also, the second shape estimation unit 308 estimates the shape of the motion-blurred foreground region. For example, camera position and orientation parameters including information indicating the position and orientation of the camera 101 are used for shape estimation. A method of estimating the shape includes, for example, a method using the Visual Hull method described in Non-Patent Document 1. For example, the first shape estimation unit 307 uses the Visual Hull method to project the silhouettes of the foreground region onto the real space, and estimates the portion where the silhouettes overlap as the shape of the foreground. The first shape estimating unit 307 and the second shape estimating unit 308 store in the RAM 202, for example, each pixel of the image for each camera 101, in which a distance (as information representing shape) is assigned in a map format. Just do it. Here, the distance refers to the distance from the output viewpoint to the subject captured in the pixel of interest. In the following description, distances stored in this map format will be referred to as distance maps as needed. An output viewpoint refers to a virtual viewpoint.

A method of generating a distance map based on images of a subject obtained by a plurality of cameras 101 is well known, and any method can be adopted. For example, a three-dimensional model of the subject can be generated using the visual volume tolerance method or the stereo matching method described in US Pat. Based on the relationship between the virtual viewpoint and the three-dimensional model of the subject, the distance from the virtual viewpoint to the corresponding subject is derived for each pixel of the virtual viewpoint image and stored in the distance map. The method of generating the distance map is not limited to the method based on the captured image of the subject, and a three-dimensional model of the subject may be generated using some tracker or the like, and the distance map may be generated based on this three-dimensional model. Alternatively, the distance from the virtual viewpoint to the corresponding subject may be measured in advance by a range sensor or the like, and a distance map may be acquired.

Next, in S408, the first rendering unit 309 renders the shape of the foreground area to generate a foreground virtual viewpoint image. Also, the second rendering unit 310 renders the shape of the non-motion-blurred foreground region to generate a non-motion-blurred virtual viewpoint image. For rendering, virtual viewpoint parameters including, for example, the position of the virtual viewpoint and the direction of the line of sight are used.
An example of the outline of the processing of the first rendering unit 309 and the second rendering unit 310 will be described below.
The processing performed by the first rendering unit 309 and the second rendering unit 310 corresponds to processing for specifying the position of the subject existing in the direction of interest based on the distance map and extracting the color information of the subject from the captured image. In other words, the first rendering unit 309 and the second rendering unit 310 identify the position of the subject appearing in the pixel of interest in the virtual viewpoint image based on the distance map, and obtain the color information of the subject appearing in the pixel of interest. Extract from the captured image. Specifically, the first rendering unit 309 and the second rendering unit 310 render images in the direction of interest based on the distance from the virtual viewpoint to the subject present in the direction of interest and the relationship between the positions and orientations of the virtual viewpoint and the camera 101 . A pixel on the captured image corresponding to the existing subject is specified. Then, the first rendering unit 309 and the second rendering unit 310 acquire the color information of the specified pixel as the color information of the subject present in the viewing direction from the virtual viewpoint.

This processing can be performed, for example, as follows. In the following description, the coordinates of the pixel of interest in the virtual viewpoint image are assumed to be (u ₀ , v ₀ ). The position of the subject captured in the pixel of interest can be represented by the coordinates of the camera coordinate system at the output viewpoint according to the following equation (2).

In Equation (2), (x ₀ , y ₀ , z ₀ ) represent the coordinates of the subject in the camera coordinate system. d ₀ (u ₀ , v ₀ ) represents the distance from the output viewpoint to the subject in the pixel of interest shown in the distance map. f ₀ represents the focal length of the output viewpoint, and c _x0 and c _y0 represent the principal point positions of the output viewpoint.
Next, the coordinates of the camera coordinate system at the output viewpoint for the subject captured in the pixel of interest can be converted into the coordinates of the world coordinate system according to the following equation (3).

In Equation (3), (X ₀ , Y ₀ , Z ₀ ) represent the coordinates of the subject in the world coordinate system. R ₀ represents the optical axis direction of the output viewpoint. (X _output , Y _output , Z _output ) represent the coordinates of the output viewpoint in the world coordinate system.
Next, the coordinates of the captured image from the input viewpoint, in which the subject exists at the coordinates (X ₀ , Y ₀ , Z ₀ ) of the subject's world coordinate system, are calculated according to the following equation (5). can be done. An input viewpoint refers to the viewpoint of the camera 101 .

In Equation (4), R _i represents the optical axis direction of input viewpoint i (input viewpoint i is the i-th input viewpoint among a plurality of input viewpoints). (X _cam,i , Y _cam,i , Z _cam,i ) represent the coordinates of the world coordinate system of the camera 101 at the input viewpoint i. f _i represents the focal length of the input viewpoint i, and c _xi and c _yi represent the principal point positions of the input viewpoint i. Also, t represents a constant. Solving equation (4) for (u _i , v _i ) yields equation (5).

According to equation (5), the constant t can be calculated first, and (u _i , v _i ) can be calculated using the obtained constant t. In this way, the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image can be converted to the coordinates (u _i , v _i ) of the pixel in the captured image. It is highly likely that the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image and the coordinates (u _i , v _i ) of the pixel in the captured image correspond to the same object. Therefore, the pixel value (color information) at the coordinates (u _i , v _i ) of the pixel in the captured image is used as the pixel value (color information) at the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image. be able to.

However, due to the difference in line-of-sight direction, the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image and the coordinates (u _i , v _i ) of the pixel in the captured image do not necessarily correspond to the same subject. do not have. In addition, even if these images correspond to the same object, there is a possibility that the captured images have different colors due to the influence of the direction of the light source and the like. For this reason, in the present embodiment, the first rendering unit 309 and the second rendering unit 310 extract the captured image corresponding to the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image from the plurality of captured images. Pixel coordinates (u _i , v _i ) (i=1 to N: N is the number of cameras 101) are specified. Then, the first rendering unit 309 and the second rendering unit 310 weight-synthesize the pixel values of the specified pixels. Here, a captured image in which the subject corresponding to the pixel of interest is not captured because the subject is out of the imaging range can be excluded from the synthesis target. The pixel value obtained by such weighted synthesis is used as the pixel value of the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image.

At this time, at the same time, the value of the motion blur amount map at the coordinates (u ₀ , v ₀ ) of the pixel of interest in the virtual viewpoint image is also generated by weighted synthesis of the motion blur amount map at the real viewpoint in the same way as the pixel value. can do.
In FIG. 5, an image 507 represents an example of the result of rendering by the first rendering unit 309 (foreground virtual viewpoint image), and an image 508 represents an example of the result of rendering by the second rendering unit 310 (non-motion-blurred virtual viewpoint image). represents an example. In image 507, the fast moving left hand portion is rendered as a large opaque blob. In image 508, the image is rendered with the fast moving left hand portion removed. Here, the picture that should be seen from the virtual viewpoint is a picture that can be seen semi-transparently because the left hand part moves and blurs. As described above, the image 507 (foreground virtual viewpoint image) is more blurred in at least a partial area of the subject on the image due to the motion of the subject than the image 508 (non-motion-blurred virtual viewpoint image).

Returning to the description of FIG. 4, in S409, the α-blending unit 311 α-blends the foreground virtual viewpoint image and the non-motion-blurred virtual viewpoint image according to the amount of motion blur to generate a motion-blurred mixed virtual viewpoint image. In FIG. 5, an image 509 represents an example of a motion-blurred mixed virtual viewpoint image. According to the motion blur amount map 505, the images 507 (foreground virtual viewpoint image) and 508 (non-motion blur virtual viewpoint image) are α-blended. By doing so, it is possible to generate an image 509 in which the fast-moving left hand part is translucent.

α is an example of a parameter for determining the synthesis ratio of the pixel values of mutually corresponding pixels in the foreground virtual viewpoint image and the non-motion-blurred virtual viewpoint image. For example, if the value of the motion blurring amount map 505 is x [pixel], α is represented by the following equation (6). Let [R1, G1, B1] be the RGB values obtained as a result of rendering (foreground virtual viewpoint image) by the first rendering unit 309 . Let [R2, G2, B2] be the RGB values obtained as the result of rendering by the second rendering unit 310 (non-motion-blurred virtual viewpoint image). Then, these RGB values [R1, G1, B1] and [R2, G2, B2] are combined using α as shown in the following equation (7) to determine the RGB values of the output image. .

As described above, in the present embodiment, the image processing apparatus 102 acquires subject motion information from a captured image, divides the subject area of the captured image into a motion blur area and a non-motion blurring foreground area, and renders α blend. Therefore, it is possible to generate a virtual viewpoint image in which the foreground area where motion blur occurs is naturally translucent. Therefore, it is possible to appropriately generate a virtual viewpoint image even when motion blur occurs in the captured image.

In this embodiment, an example has been shown in which rendering is divided into two areas, the motion blurred area and the non-motion blurred foreground area. However, shape estimation and rendering may be performed by dividing into three or more regions depending on the magnitude of motion blur.
Also, in order to reduce computational resources, the second shape estimation unit 308 and the second rendering unit 310 may render only the shape of the background without estimating and rendering the shape of the foreground. In this case, [R2, G2, B2] in Equation (7) is the result of rendering the background image, and the amount of blur in the foreground area ([R1, G1, B2]) (the value of the motion blur amount map 505) is It is possible to change only the transparency.

<Second embodiment>
Next, a second embodiment will be described. In this embodiment, an example of creating an image by combining a short-second exposure camera and a long-second exposure camera will be described. Here, short-second exposure and long-second exposure refer to moving images with relatively long and short exposure times. For example, in a moving image of 60 [fps], it is assumed that the exposure time is 1/100 [sec] for long seconds and 1/1000 [sec] for short seconds. . It is assumed that these cameras have the same frequency [fps] and their shooting timings are synchronized. It is also assumed that a plurality of cameras 101 are set in advance as cameras 101 that perform short-second exposure and cameras 101 that perform long-second exposure. As will be described later, a virtual viewpoint image is generated based on an image captured by the camera 101 that performs short exposure, and a virtual viewpoint image is generated based on an image captured by the camera 101 that performs long exposure. It is preferable to disperse the camera 101 for short-second exposure and the camera 101 for long-second exposure so that each virtual viewpoint image is appropriately generated. For example, in FIG. 1, the cameras 101 that perform short-time exposure and the cameras 101 that perform long-time exposure can be alternately arranged. The number of cameras 101 that perform short-second exposure and the number of cameras 101 that perform long-second exposure may be the same or different.

In the first embodiment described above, a motion blurring region is determined by estimating a time-series moving body map of images from one camera. In this way, it takes a relatively long time to calculate the moving body map. Therefore, in the present embodiment, both a group of images with relatively little motion blur with short-second exposure and a group of images with relatively large motion blur with long-second exposure are used without calculating a moving body map. generates a virtual viewpoint image of a scene with motion blur. As described above, the main difference between the present embodiment and the first embodiment is the processing for determining an area in which motion blur occurs. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS. 1 to 5, and detailed description thereof is omitted.

An example of image processing in the image processing apparatus 102 of the present embodiment will be described below with reference to FIGS. 6, 7, and 8. FIG. FIG. 6 is a block diagram showing an example of the functional configuration of the image processing apparatus 102 of this embodiment. FIG. 7 is a flowchart for explaining an example of the image processing method of this embodiment. FIG. 8 is a schematic diagram illustrating an example of the content of image processing according to this embodiment.
Also in this embodiment, the CPU 201 functions as each block shown in FIG. 6 by executing the program stored in the ROM 203, and executes the processing according to the flowchart of FIG. Further, the CPU 201 does not necessarily have to execute all the functions of the image processing apparatus 102, and a processing circuit corresponding to each function may be provided in the image processing apparatus 102 and the processing circuit may execute the function.

FIG. 8 shows the details of the image processing, taking as an example a case where an image of a subject whose left hand is moving quickly is captured. Images representing the actual movement of the subject are obtained in the order of image 801A, image 801B, and image 801C. However, when the exposure time (Tv) of the camera 101 is long, the camera 101 captures an image such as an image 802 in which the left hand portion is motion-blurred. On the other hand, when the exposure time (Tv) of the camera 101 is short, the camera 101 captures an image such as an image 803 in which the left hand portion is not blurred.

Hereinafter, an image captured by a camera 101 with a relatively long exposure time (Tv) is referred to as a long Tv image as necessary, and an image captured by a camera 101 with a relatively short exposure time (Tv) is It is called a short Tv image if necessary.
In S701, the long Tv image acquisition unit 601 acquires long Tv image data. The short Tv image acquisition unit 602 acquires short Tv image data. For example, when a person quickly waves his/her left hand in the order of image 801A→image 801B→image 801C, the long Tv image becomes image 802, and the short Tv image becomes image 803.

Next, in S702, the long-Tv background image acquisition unit 603 acquires long-Tv background image data. The short Tv background image acquisition unit 604 acquires short Tv background image data. The long-Tv background image is an image captured with a relatively long exposure time (Tv) by each camera 101 without the object 105 present, and is stored in the external memory 208 in advance. The short Tv background image is an image captured by each camera 101 with a relatively short exposure time (Tv) in a state where the object 105 does not exist, and is stored in the external memory 208 in advance. The exposure time for capturing the long-Tv image and the exposure time for capturing the long-Tv background image are preferably the same. Similarly, the exposure time for capturing the short Tv image and the exposure time for capturing the short Tv background image are preferably the same. Also, the position and posture of each camera 101 when capturing the long Tv background image and the short Tv background image are preferably the same as when the subject 105 exists. In FIG. 8, for example, a background image such as image 804 is obtained.

Next, in S703, the first foreground image separation unit 605 separates the long Tv image data into a long Tv foreground region and a long Tv background region. For example, the first foreground image separation unit 605 determines whether the absolute value of the difference in at least one of color and texture in corresponding pixels of the image (long Tv image) 802 and the image (background image) 804 exceeds a threshold. determine whether or not The first foreground image separating unit 605 regards the area whose absolute value exceeds the threshold as the foreground area, assigns white (1) to the pixels of this area, and treats the other area as the background area, and assigns black (0) to the pixels of this area. ) is assigned to each pixel. In this case, the area assigned white (1) is the foreground area, and the area assigned black (0) is the background area. The foreground region thus defined is the long-Tv foreground region, and the background region is defined as the long-Tv background region. As a result, an image such as the image 805 shown in FIG. 8 is obtained.

Next, in S704, the second foreground image separation unit 606 separates the short Tv image data into a short Tv foreground region and a short Tv background region. For example, the second foreground image separation unit 606 determines whether the absolute value of the difference in at least one of the color and texture in the corresponding pixels of the image (short Tv image) 803 and the image (background image) 804 exceeds a threshold. determine whether or not The second foreground image separation unit 606 regards the area whose absolute value exceeds the threshold as the foreground area, assigns white (1) to the pixels of the area, and treats the other area as the background area, and assigns black (0) to the pixels of the area. ) is assigned to each pixel. In this case, the area assigned white (1) is the foreground area, and the area assigned black (0) is the background area. The foreground area thus determined is the short-Tv foreground area, and the background area thus determined is the short-Tv background area. As a result, an image such as the image 806 shown in FIG. 8 is obtained.

Next, in S705, the first shape estimating unit 607 calculates the image from the overlap region of the multi-viewpoint long Tv foreground region (the long Tv foreground region obtained by each camera 101) with a relatively long exposure time. Estimate the shape of the foreground region. In the following description, this shape will be referred to as the long Tv shape as appropriate.
Next, in S706, the second shape estimating unit 608 captures an image with a relatively short exposure time from the overlap region of the multi-viewpoint short Tv foreground regions (short Tv foreground regions obtained at the same timing by each camera 101). Estimate the shape of the foreground region when In the following description, this shape will be referred to as the short Tv shape as appropriate.

Next, in S707, the first rendering unit 609 renders the long Tv shape to generate a virtual viewpoint image obtained when it is assumed that the image is captured from the virtual viewpoint with a relatively long exposure time. In the following description, this virtual viewpoint image will be referred to as a long-Tv virtual viewpoint image as required. Also, the second rendering unit 610 renders the short Tv shape to generate a virtual viewpoint image obtained when it is assumed that the image is captured from the virtual viewpoint with a relatively short exposure time. In the following description, this virtual viewpoint image will be referred to as a short-Tv virtual viewpoint image as required.

Here, the input images (long Tv image, short Tv image) used to generate the respective foreground region images are not necessarily included in the textures used to generate the virtual viewpoint images (long Tv virtual viewpoint image, short Tv virtual viewpoint image). Tv image) may not be used. For example, if the exposure time (Tv) is different, the color tone may also change. Therefore, only the long Tv image may be used as the texture used when generating the virtual viewpoint images (the long Tv virtual viewpoint image and the short Tv virtual viewpoint image). In FIG. 8, for example, the image 807 is the long-Tv virtual viewpoint image, and in the long-Tv virtual viewpoint image, moving parts appear as large opaque masses like the image 807 . Also, an image 808 is a short Tv virtual viewpoint image, and in the short Tv virtual viewpoint image, like the image 808, the shape of a hand at a certain moment appears. As described above, the image 807 (long Tv virtual viewpoint image) is more blurred than the image 808 (short Tv virtual viewpoint image) in at least a partial area of the subject on the image due to the movement of the subject.

Next, in S708, the motion blur amount calculation unit 611 calculates the motion blur amount from the absolute value of the difference between the pixel values of the corresponding pixels of the long-Tv virtual viewpoint image and the short-Tv virtual viewpoint image. At this time, a long Tv shape and a short Tv shape may be used instead of the long Tv virtual viewpoint image and the short Tv virtual viewpoint image.
Next, in S709, the α-blending unit 612 α-blends the long-Tv virtual viewpoint image and the short-Tv virtual viewpoint image according to the amount of motion blur to generate a motion-blur mixed virtual viewpoint image. For example, the α blend unit 612 sets the RGB values of the long-Tv virtual viewpoint image to [R1, G1, B1] and sets the RGB values of the short-Tv virtual viewpoint image to [R2, G2, B2] in Equation (7), A long-Tv virtual viewpoint image and a short-Tv virtual viewpoint image can be combined according to equation (7). At this time, the α-blending unit 612, for example, makes the α-blending value (=α) of the long-Tv virtual viewpoint image smaller as the amount of motion blur increases (that is, the blending ratio of the long-Tv virtual viewpoint image is set to make low). In FIG. 8, for example, the result of α-blending is an image such as image 809 in which the blurred portion due to the movement of the hand is translucent. The image will look like it was taken with

As described above, in the present embodiment, the image processing apparatus 102 renders images using both short-exposure images with small motion blur and long-exposure images with large motion blur, and performs α-blending. do. Therefore, a virtual viewpoint image of a scene with motion blur can be generated without calculating a motion map. Therefore, in addition to the effect described in the first embodiment, the effect of being able to reduce the processing time can be obtained.

<Third Embodiment>
Next, a third embodiment will be described. In this embodiment, an example of switching the α-blending ratio or simplifying the processing depending on the difference between the virtual viewpoint and the real viewpoint of the camera (actual viewpoint) will be described. In the first and second embodiments, a problem arises when the virtual viewpoint image of the portion that becomes translucent due to motion blur is generated when the virtual viewpoint is far from the real viewpoint of the camera. If the virtual viewpoint and the camera's real viewpoint are sufficiently close, the camera's image is close to what it looks like from the virtual viewpoint. becomes a picture. Therefore, in the present embodiment, switching between whether or not to perform α blending, and control of the α blending value (=α) when performing α blending, according to the closeness between the virtual viewpoint and the real viewpoint of the camera. Here is an example of doing As described above, the main difference between the present embodiment and the first and second embodiments is the processing related to α-blending. Therefore, in the description of this embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals as those in FIGS. 1 to 8, and detailed description thereof is omitted.

An example of image processing in the image processing apparatus 102 of the present embodiment will be described below with reference to FIGS. 9 and 10. FIG. FIG. 9 is a block diagram showing an example of the functional configuration of the image processing apparatus 102 of this embodiment. FIG. 10 is a flowchart for explaining an example of the image processing method of this embodiment.
The image processing apparatus 102 shown in FIG. 9 further includes a viewpoint dependent processing setting unit 912 in contrast to the image processing apparatus 102 shown in FIG. 901-911 in FIG. 9 are the same as blocks 301-311 in FIG. 3, respectively. However, the moving object map calculation unit 904 and the first foreground/background separation unit 903 in this embodiment change the processing according to the processing switching setting output as a result of the viewpoint dependent processing setting unit 912 . Also, when the virtual viewpoint and the real viewpoint of the camera 101 are sufficiently close, the moving image data acquisition unit 901 does not send the moving image data to the moving body map calculation unit 904 . In this case, the image processing apparatus 102 does not execute processing by processing blocks after the moving object map calculation unit 904 . Similarly, the first foreground/background separator 903 does not send the foreground image data to the second foreground/background separator 906 . In this case, the image processing apparatus 102 does not execute processing by processing blocks after the second foreground/background separation unit 906 .

Also in this embodiment, the CPU 201 functions as each block shown in FIG. 9 by executing the program stored in the ROM 203, and executes the processing according to the flowchart of FIG. Further, the CPU 201 does not necessarily have to execute all the functions of the image processing apparatus 102, and a processing circuit corresponding to each function may be provided in the image processing apparatus 102 and the processing circuit may execute the function.
Since S1001 to S1003 in FIG. 10 are the same as S401 to S403 in FIG. 4, detailed description thereof will be omitted.

In S1004, the viewpoint-dependent processing setting unit 912 determines whether the virtual viewpoint and the real viewpoint of the camera 101 are sufficiently close. As the real viewpoint of the camera 101 used for this determination, the real viewpoint of the camera 101 representative of the plurality of cameras 101 is used when generating the virtual viewpoint image. For example, the real viewpoint of the camera 101 that captures an image that serves as a texture can be used when generating a virtual viewpoint image. Alternatively, the real viewpoint of the camera 101 closest to the virtual viewpoint may be adopted. As a result of this determination, if the virtual viewpoint and the real viewpoint of the camera 101 are sufficiently close, the process proceeds to S1011; otherwise, the process proceeds to S1005. The index for evaluating the closeness of viewpoints includes, for example, at least one of the position of each viewpoint and the orientation of each viewpoint (the angle between a virtual line connecting the viewpoint and the subject and a reference line (for example, a horizontal plane)). includes one. Here, for example, it is considered that the closer the direction from the input viewpoint to the subject is to the direction from the output viewpoint to the subject, the closer the subject image captured in the captured image is to the subject image from the virtual viewpoint. Therefore, the closeness of viewpoints can be evaluated by the closeness between the direction of the direction vector indicating the direction from the input viewpoint to the subject and the direction of the direction vector indicating the direction from the output viewpoint to the subject. Specifically, the angle formed by a direction vector (of any magnitude) indicating the direction from the virtual viewpoint to the subject and a direction vector (of any magnitude) indicating the direction from the output viewpoint to the subject is smaller than a threshold. The closeness of the viewpoint can be evaluated by whether or not.

In addition to such a direction, the position of the subject positioned in the direction of interest within the field of view of the camera 101 may be further considered to evaluate the proximity of the viewpoint. For example, if the position of the subject is close to the outside of the field of view of the camera 101, the closeness of viewpoints may be evaluated such that the viewpoint difference is large. In this case, for example, even if the direction from the input viewpoint (real viewpoint of the camera 101) to the subject is close to the direction from the output viewpoint (virtual viewpoint) to the subject, if the subject is not included in the field of view of the camera 101, the virtual It can be evaluated that the closeness between the viewpoint and the actual viewpoint of the camera 101 is not close. In this way, the index for evaluating the closeness of viewpoints includes, for example, the field of view of each viewpoint. In the following description, the closeness between the virtual viewpoint and the real viewpoint of the camera 101 will be referred to as a virtual viewpoint difference as needed.
As described above, if it is determined in S1004 that the virtual viewpoint difference is large, the process proceeds to S1005. Since the processing of S1005 to S1009 is the same as the processing of S404 to S408 in FIG. 4, detailed description of these processing will be omitted.

Then, the process proceeds to S1010. In S1010, the α-blending unit 911 generates a motion-blur mixed virtual viewpoint image by α-blending the foreground virtual viewpoint image and the non-motion-blurred virtual viewpoint image according to the amount of motion blur and the virtual viewpoint difference. At this time, the larger the amount of motion blur, the smaller the blending rate of the foreground virtual viewpoint image (the value of α-blending (=α) when performing α-blending). Also, the smaller the virtual viewpoint difference is, the larger the blending rate of the foreground virtual viewpoint image (the α blend value (=α) when α blending is performed) is made. This completes the processing when the virtual viewpoint difference is large.

On the other hand, if it is determined in S1004 that the virtual viewpoint difference is small, the process proceeds to S1011. In S1011, the first shape estimation unit 907 estimates the shape of the foreground region. Since the content of this process is the same as that of S407, its detailed description is omitted.
Next, in S1012, the first rendering unit 909 renders the shape of the foreground area to generate a virtual viewpoint image. Since the content of this process is the same as that of S408, its detailed description is omitted. Here, the virtual viewpoint image to be output is not an image that is α-blended, but an image that is created only by rendering a shape including motion blur. Even without α-blending, an image with natural motion blur can be rendered when the virtual viewpoint and the real viewpoint of the camera 101 are sufficiently close.

As described above, in the present embodiment, the image processing apparatus 102 switches between generating a motion blur amount map and a non-motion blurring virtual viewpoint image depending on whether the virtual viewpoint and the real viewpoint of the camera 101 are sufficiently close. . In addition, the image processing device 102 controls the blending ratio in α-blending when generating a non-motion-blurred virtual viewpoint image according to the closeness between the virtual viewpoint and the real viewpoint of the camera 101 . Therefore, it is possible to render an image with natural motion blur and reduce the processing time.
The technique of this embodiment can also be applied to the second embodiment. In this case, for example, when the virtual viewpoint and the real viewpoint of the camera 101 are sufficiently close, the processing for generating the short-Tv virtual viewpoint image is omitted.

It should be noted that the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

<Other Examples>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

101: camera, 102: image processing device, 103: display device, 104: input device, 105: subject

Claims (13)

A region of a subject extracted by a first extraction method from an image captured by any one of a plurality of imaging devices that capture images of an imaging region from a plurality of directions, and an image captured by the imaging device information for specifying the three-dimensional shape of a subject located within the imaging area based on the area of the subject extracted from the captured image by a second extraction method different from the first extraction method; a first acquisition means for acquiring certain shape information;
a second acquisition means for acquiring viewpoint information, which is information for specifying a virtual viewpoint;
generating means for generating a virtual viewpoint image based on the shape information and the viewpoint information ;
The first extraction method includes a method of extracting the subject based on the captured image and a background image corresponding to the captured image,
The second extraction method includes a method of extracting the subject based on the captured image and another captured image captured at a timing different from the timing at which the captured image was captured by the imaging device.
An image processing apparatus characterized by: The imaging device captures a moving image including a plurality of frames,
2. The image processing apparatus according to claim 1 , wherein the captured image and another captured image captured at a timing different from the timing at which the captured image was captured are included in the moving image as one frame. . The shape information includes first shape information for specifying the three-dimensional shape of the subject based on the region of the subject extracted from the captured image by the first extraction method, and the second extraction. 3. The image according to claim 1, further comprising second shape information for specifying the three-dimensional shape of the subject based on the region of the subject extracted from the captured image by the method. processing equipment. The generating means generates a first virtual viewpoint image based on the viewpoint information and the first shape information , and generates a second virtual viewpoint image based on the viewpoint information and the second shape information. 4. The image processing apparatus according to claim 3 , wherein the virtual viewpoint image is generated by generating the first virtual viewpoint image and the second virtual viewpoint image, and synthesizing the first virtual viewpoint image and the second virtual viewpoint image. The generating means synthesizes the first virtual viewpoint image and the second virtual viewpoint image at a synthesis ratio determined based on the amount of movement of a subject positioned within the imaging area. 5. The image processing apparatus according to claim 4 , wherein the virtual viewpoint image is generated. The generation means uses a synthesis ratio determined based on the position of one of the plurality of imaging devices and the position of the virtual viewpoint specified based on the viewpoint information , 6. The image processing apparatus according to claim 4 , wherein the virtual viewpoint image is generated by synthesizing the first virtual viewpoint image and the second virtual viewpoint image. 7. The generating means generates the virtual viewpoint image by alpha-blending the first virtual viewpoint image and the second virtual viewpoint image. The image processing device according to . whether to generate the second virtual viewpoint image based on the position of one of the plurality of imaging devices and the position of the virtual viewpoint specified based on the viewpoint information; a determination means for determining
When the determination means determines not to generate the second virtual viewpoint image, the first virtual viewpoint image is output , and the determination means outputs the second virtual viewpoint image. output means for outputting a virtual viewpoint image generated by synthesizing the first virtual viewpoint image and the second virtual viewpoint image by the generating means when it is determined to generate a virtual viewpoint image; 8. The image processing apparatus according to any one of claims 4 to 7, comprising : 9. The image processing apparatus according to claim 3, wherein said first shape information and said second shape information are information representing a three-dimensional shape of the same subject . The image processing apparatus according to any one of claims 1 to 9, wherein the subject is a person or a part of a person. A region of a subject extracted by a first extraction method from an image captured by any one of a plurality of imaging devices that capture images of an imaging region from a plurality of directions, and an image captured by the imaging device information for specifying the three-dimensional shape of a subject located within the imaging area based on the area of the subject extracted from the captured image by a second extraction method different from the first extraction method; a first acquisition step of acquiring certain shape information;
a second acquisition step of acquiring viewpoint information, which is information for specifying a virtual viewpoint;
a generating step of generating a virtual viewpoint image based on the shape information and the viewpoint information;
The first extraction method includes a method of extracting the subject based on the captured image and a background image corresponding to the captured image,
The second extraction method includes a method of extracting the subject based on the captured image and another captured image captured at a timing different from the timing at which the captured image was captured by the imaging device.
An image processing method characterized by: The imaging device captures a moving image including a plurality of frames,
12. The image processing method according to claim 11, wherein the captured image and another captured image captured at a timing different from the timing at which the captured image was captured are included in the moving image as one frame. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 10.

Images