JP2014120093A - Device, method and program for generating virtual viewpoint image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device, a method and a program capable of appropriately generating a virtual viewpoint image from a virtual viewpoint.SOLUTION: The virtual viewpoint image generation device 1 includes an object separation unit 10, a camera space interpolation unit 10 and a frame space interpolation unit 30. The object separation unit 10 separates each of a plurality of objects forming occlusion and obtains texture of each of the plurality of objects. The camera space interpolation unit 20 obtains texture of objects containing a region hidden by the occlusion, out of a plurality of viewpoint images other than viewpoint images where the occlusion occurs. The frame space interpolation unit 30 obtains texture of objects containing the region hidden by the occlusion, out of a frame preceding or following a frame where the occlusion occurs.

Description

  The present invention relates to a virtual viewpoint video generation device, a virtual viewpoint video generation method, and a program for generating a virtual viewpoint video.

  Conventionally, a method for generating a video at a virtual viewpoint from a plurality of viewpoint videos has been proposed. As this technique, for example, in Patent Document 1, a texture of each object is generated from a video in which an event performed in a large space is captured by a plurality of cameras, and the generated texture is stored in a three-dimensional space for each object. A method of mapping to a plane arranged in FIG.

  However, in the above-described method disclosed in Patent Document 1, if an occlusion in which a plurality of objects overlap is generated, a plurality of objects are included in one texture. For this reason, there is a possibility that the video at the virtual viewpoint cannot be appropriately generated.

  Therefore, for example, in Non-Patent Document 1, using another viewpoint video having a different viewpoint from the viewpoint video in which occlusion occurs, rough three-dimensional information is restored, and an object existing in the foreground is separated to create a texture. The method of acquisition is shown. In this method, a plane parallel to the image where the occlusion occurs is set near the object position in the three-dimensional space, and the area of the object existing in the foreground extracted from other viewpoint images is projected onto this plane. The texture area of the object in the foreground is acquired, and the texture of the object in the foreground is acquired. Further, since there is a portion hidden behind the object in front, the texture is acquired from the viewpoint video close to the viewpoint.

JP 2011-170487 A

Y. Ohta, I. Kitahara, Y. Kameda, H. Ishikawa, and T. Koyama. Live 3D video in soccer stadium. IJCV, 75 (1): 173-187, 2007.

  Non-Patent Document 1 discloses a method of using both object separation processing and inter-camera interpolation processing when occlusion occurs in a free viewpoint video generation process.

  The separation process is effective when the occlusion is small. However, when the occlusion is large, the loss of texture becomes large and the visual quality is degraded.

  Therefore, the separation process is applied to the object that is present in the foreground by determining that no texture loss occurs. On the other hand, since there is a possibility that a texture loss may occur with respect to an object existing behind the object existing in the foreground, the inter-camera interpolation process is uniformly applied instead of the separation process. In the inter-camera interpolation process, a texture is acquired from a viewpoint video that is close to the viewpoint.

  However, if occlusion occurs even in viewpoint videos that are close to the viewpoint, if the inter-camera interpolation process is applied, the texture is acquired with the occlusion still occurring, so the visual quality of the virtual viewpoint video There was a risk that it would fall.

  Also, as the difference between the direction of the virtual viewpoint video and the direction of the viewpoint video from which the texture was acquired in the inter-camera interpolation process becomes larger, the acquired texture direction is far from the original object direction. . For this reason, even when the difference between these directions is large, there is a possibility that the visual quality of the virtual viewpoint video is deteriorated when the inter-camera interpolation process is applied.

  Accordingly, the present invention has been made in view of the above-described problems, and an object thereof is to provide a virtual viewpoint video generation device, a virtual viewpoint video generation method, and a program that can improve the visual quality of a virtual viewpoint video. .

The present invention proposes the following matters in order to solve the above problems.
(1) The present invention is a virtual viewpoint video generation device (e.g., equivalent to the virtual viewpoint video generation device 1 in FIG. 1) that generates a video at a virtual viewpoint from a plurality of viewpoint videos. When occlusion occurs in any of the first texture acquisition means (for example, the object separation unit 10 in FIG. 1) that separates each of the plurality of objects forming the occlusion and obtains the texture of each of the plurality of objects. And when the occlusion occurs in any one of the plurality of viewpoint videos, the part of the plurality of viewpoint videos excluding the viewpoint video in which the occlusion occurs includes a portion hidden by the occlusion. Second texture acquisition means for acquiring the texture of the object (for example, the camera of FIG. When the occlusion occurs in any of the plurality of viewpoint videos, the object including the portion hidden by the occlusion from the frame before or after the frame where the occlusion occurs is equivalent. Third texture acquisition means (for example, corresponding to the inter-frame interpolation unit 30 in FIG. 1) for acquiring textures, texture acquired by the first texture acquisition means, and second texture acquisition means Determination means (for example, the determination unit in FIG. 1) that determines the texture to be used as the texture of the object including the portion hidden by the occlusion from the texture obtained by the third texture acquisition means 40) and the text determined by the determination means With turbocharger, said image generating means for generating an image of a virtual viewpoint (e.g., corresponding to the image generation unit 50 of FIG. 1) proposes a virtual viewpoint video generation device characterized by comprising a, a.

  According to the present invention, a virtual viewpoint video generation device that generates a video at a virtual viewpoint from a plurality of viewpoint videos includes a first texture acquisition unit, a second texture acquisition unit, a third texture acquisition unit, a determination unit, And video generation means. Then, when occlusion occurs in any of the plurality of viewpoint videos by the first texture acquisition means, the plurality of objects forming the occlusion are separated and the respective textures of the plurality of objects are acquired. It was. Further, when occlusion occurs in any one of the plurality of viewpoint videos by the second texture acquisition unit, the occlusion is hidden from the viewpoint videos except for the viewpoint video in which the occlusion occurs among the plurality of viewpoint videos. The texture of the object including the part was acquired. Further, when occlusion occurs in any of a plurality of viewpoint videos by the third texture acquisition means, an object including a portion hidden by this occlusion from the frame before or after the frame where the occlusion occurs is displayed. I decided to get a texture. Further, the occlusion from the texture acquired by the first texture acquisition means, the texture acquired by the second texture acquisition means, and the texture acquired by the third texture acquisition means by the determination means. It was decided to determine what to use as the texture of the object including the hidden part. In addition, the video generation unit generates the video at the virtual viewpoint using the texture determined by the determination unit.

  For this reason, the first texture acquisition unit performs a separation process for separating an object and acquires a texture, and the second texture acquisition unit performs an inter-camera interpolation process for acquiring a texture from another viewpoint video. At the same time, the third texture acquisition means performs inter-frame interpolation processing for acquiring textures from other frames. Accordingly, the generation of the virtual viewpoint video can be performed by combining not only the separation process and the inter-camera interpolation process but also the inter-frame interpolation process, so that the visual quality of the virtual viewpoint video can be improved.

  (2) In the virtual viewpoint video generation device according to (1), the determination unit is acquired by the first texture acquisition unit as a texture of an object including a portion hidden by the occlusion. Proposing a virtual viewpoint video generation device characterized by determining in order of priority of a texture obtained by the second texture obtaining unit, and a texture obtained by the third texture obtaining unit Yes.

  According to this invention, in the virtual viewpoint video generation device of (1), the determination unit uses the texture acquired by the first texture acquisition unit as the texture of the object including the portion hidden by occlusion, The textures acquired by the second texture acquisition unit and the textures acquired by the third texture acquisition unit are determined in order of priority.

  For this reason, the acquired texture can be used in the order of priority of the first texture acquisition unit, the second texture acquisition unit, and the third texture acquisition unit as the texture of the object including the portion hidden by occlusion. .

  (3) In the virtual viewpoint video generation device according to (1) or (2), the third texture acquisition unit may include the third texture acquisition unit among the plurality of frames constituting the viewpoint video in which the occlusion occurs. There has been proposed a virtual viewpoint video generation device characterized in that the texture of the object is acquired from a frame before or after a frame where occlusion occurs.

  According to the present invention, in the virtual viewpoint video generation device according to (1) or (2), the third texture acquisition unit generates this occlusion among a plurality of frames constituting the viewpoint video where the occlusion occurs. The texture of the object is obtained from the frame before or after the current frame.

  For this reason, when acquiring a texture in the inter-frame interpolation process, a frame constituting a viewpoint video in which occlusion occurs is used as another frame. Therefore, since the viewpoint is the same as compared with the case where a frame constituting another viewpoint video different from the viewpoint video in which occlusion occurs, the visual quality of the virtual viewpoint video can be further improved.

  (4) In the virtual viewpoint video generation device according to any one of (1) to (3), the determination unit determines whether to use the texture acquired by the first texture acquisition unit. Of the first determination procedure (e.g., corresponding to the process of step S9 in FIG. 2), the texture acquired by the second texture acquisition unit, and the texture acquired by the third texture acquisition unit A virtual viewpoint video generation apparatus characterized by performing a second determination procedure for determining which one to use (for example, corresponding to the processing of step S11 in FIG. 2) is proposed.

  According to this invention, in the virtual viewpoint video generation device according to any one of (1) to (3), the determination unit performs the first determination procedure and the second determination procedure. In the first determination procedure, it is determined whether to use the texture acquired by the first texture acquisition means. In the second determination procedure, it is determined which of the texture acquired by the second texture acquisition unit and the texture acquired by the third texture acquisition unit is to be used.

  Therefore, by performing the first determination procedure and the second determination procedure in this order, if it is determined to be used in the first determination procedure, the second determination procedure is not performed. be able to. Therefore, the processing of the virtual viewpoint video generation device may be reduced.

  (5) In the virtual viewpoint video generation device according to (4), the determination unit performs object detection on the texture acquired by the first texture acquisition unit in the first determination procedure. Thus, a virtual viewpoint video generation storage characterized by determining based on the result of object detection is proposed.

  Here, in Non-Patent Document 1, as described above, the inter-camera interpolation process is applied uniformly to the object existing behind the foremost object instead of the separation process. However, even if the object is in the back, if the occlusion is small, the loss of texture is small, so it is better to apply the separation process than the inter-camera interpolation process to maintain the consistency of the viewpoint between frames. And the visual quality of the virtual viewpoint video is improved.

  Therefore, according to the present invention, in the virtual viewpoint video generation device of (4), in the first determination procedure, object detection is performed on the texture acquired by the first texture acquisition unit by the determination unit, The determination is based on the result of object detection.

  For this reason, it is possible to determine whether or not the texture acquired by the separation process is appropriate as a texture of an object including a portion hidden by occlusion, using object detection. According to this, it is possible to more appropriately determine which one of the texture acquired by the separation process and the texture acquired by the inter-camera interpolation process or the inter-frame interpolation process should be used. Therefore, the visual quality of the virtual viewpoint video can be further improved.

  (6) The virtual viewpoint video generation device according to (5) proposes a virtual viewpoint video generation device characterized in that the determination means uses a HOG feature amount in the object detection.

  According to the present invention, in the virtual viewpoint video generation device of (5), the determination unit uses the HOG feature amount for object detection.

  For this reason, object detection can be performed using the HOG feature amount.

  (7) The present invention relates to the virtual viewpoint video generation device according to any one of (4) to (6), wherein the determination means includes the object including a portion hidden by the occlusion in the second determination procedure. There is no occlusion in the viewpoint video from which the texture has been acquired by the second texture acquisition means, and the angle difference between the viewpoint of the viewpoint video from which the texture has been acquired by the second texture acquisition means and the virtual viewpoint Is less than a predetermined threshold value, it is determined that the texture acquired by the second texture acquisition unit is used, and the texture including the portion hidden by the occlusion is detected by the second texture acquisition unit. The angle difference between the case where occlusion has occurred Wherein the larger than the threshold, at least in the case of either, we propose a virtual viewpoint video generation apparatus and judging the used texture acquired by the third texture acquisition means.

  According to this invention, in the virtual viewpoint video generation device according to any one of (4) to (6), the following first condition and second condition are satisfied in the second determination procedure by the determination procedure. If it is determined that the texture acquired by the second texture acquisition unit is to be used, and if at least one of the following first condition or second condition is not satisfied, the third texture It was decided to use the texture acquired by the acquisition means. The first condition is that no occlusion occurs in the viewpoint video in which the texture is acquired by the second texture acquisition unit for the object including the portion hidden by the occlusion. The second condition is that the angle difference between the viewpoint of the viewpoint video from which the texture is acquired by the second texture acquisition means and the virtual viewpoint is equal to or less than a predetermined threshold.

  Therefore, instead of the texture acquired by the inter-camera interpolation process between the case where the occlusion occurs even in the viewpoint video from which the texture is acquired by the second texture acquisition means and the case where the above-described angle difference is larger than the threshold value. In addition, the texture acquired by the inter-frame interpolation process is used. Therefore, the visual quality of the virtual viewpoint video can be further improved.

  (8) The present invention provides first texture acquisition means (for example, equivalent to the object separation unit 10 in FIG. 1), second texture acquisition means (for example, equivalent to the inter-camera interpolation unit 20 in FIG. 1), third Texture acquisition means (for example, equivalent to the inter-frame interpolation section 30 in FIG. 1), determination means (for example, equivalent to the determination section 40 in FIG. 1), and video generation means (for example, equivalent to the video generation section 50 in FIG. 1). ), And a virtual viewpoint video generation method (for example, equivalent to the virtual viewpoint video generation device 1 in FIG. 1) that generates a video at a virtual viewpoint from a plurality of viewpoint videos, When the occlusion occurs in any one of the plurality of viewpoint videos, the texture acquisition unit separates each of the plurality of objects forming the occlusion and outputs the plurality of objects. When the first step (for example, corresponding to the process of step S6 in FIG. 2) and the second texture acquisition unit generate occlusion in any of the plurality of viewpoint videos, A second step (for example, step S7 in FIG. 2) of obtaining the texture of the object including the portion hidden by the occlusion from the plurality of viewpoint videos excluding the viewpoint video in which the occlusion occurs. When the occlusion occurs in any of the plurality of viewpoint videos, the third texture acquisition unit hides the occlusion from the frame before or after the frame where the occlusion occurs. The third step to get the texture of the object (For example, corresponding to the process of step S8 in FIG. 2), the determination means includes the texture acquired in the first step, the texture acquired in the second step, and the third step. A fourth step (e.g., corresponding to the processing of steps S9 and S11 in FIG. 2) of determining the texture to be used as the texture of the object including the portion hidden by the occlusion, The video generation means includes a fifth step (for example, corresponding to the process of step S12 in FIG. 2) that generates a video at the virtual viewpoint using the texture determined in the fourth step. We have proposed a virtual viewpoint video generation method characterized by this.

  According to the present invention, when occlusion occurs in any one of the plurality of viewpoint videos by the first texture acquisition unit, the plurality of objects forming the occlusion are separated, and the textures of the plurality of objects are separated. Decided to get. Further, when occlusion occurs in any one of the plurality of viewpoint videos by the second texture acquisition means, it is hidden by this occlusion from the viewpoint videos other than the viewpoint video in which this occlusion is generated. I decided to get the texture of the object including the part. Further, when occlusion occurs in any of a plurality of viewpoint videos by the third texture acquisition means, an object including a portion hidden by this occlusion from the frame before or after the frame where the occlusion occurs is displayed. I decided to get a texture. Further, the occlusion from the texture acquired by the first texture acquisition means, the texture acquired by the second texture acquisition means, and the texture acquired by the third texture acquisition means by the determination means. It was decided to determine what to use as the texture of the object including the hidden part. In addition, the video generation unit generates the video at the virtual viewpoint using the texture determined by the determination unit. For this reason, the effect similar to the effect mentioned above can be produced.

  (9) The present invention provides first texture acquisition means (for example, equivalent to the object separation unit 10 in FIG. 1), second texture acquisition means (for example, equivalent to the inter-camera interpolation unit 20 in FIG. 1), third Texture acquisition means (for example, equivalent to the inter-frame interpolation section 30 in FIG. 1), determination means (for example, equivalent to the determination section 40 in FIG. 1), and video generation means (for example, equivalent to the video generation section 50 in FIG. 1). ) And causing a computer to execute a virtual viewpoint video generation method in a virtual viewpoint video generation device (for example, equivalent to the virtual viewpoint video generation device 1 in FIG. 1) that generates a video at a virtual viewpoint from a plurality of viewpoint videos. When the first texture acquisition unit generates occlusion in any of the plurality of viewpoint videos, the plurality of objects forming the occlusion A first step (for example, corresponding to the process of step S6 in FIG. 2), and the second texture acquisition means are configured to acquire the texture of each of the plurality of objects. When occlusion occurs in any of the viewpoint videos, a texture of an object including a portion hidden by the occlusion is acquired from the plurality of viewpoint videos except for the viewpoint video in which the occlusion occurs. When the occlusion occurs in any one of the plurality of viewpoint videos, the third texture acquisition unit, before the frame in which the occlusion has occurred, for example, corresponds to step S7 in FIG. Or, from the back frame, the occlusion that includes the part hidden by the occlusion The third step of acquiring the texture of the object (for example, corresponding to the process of step S8 in FIG. 2), and the determination means are acquired in the second step and the texture acquired in the first step. A fourth step (for example, step S9 in FIG. 2) of determining the texture to be used as the texture of the object including the part hidden by the occlusion from the texture obtained in the third step. , Corresponding to the process of S11), and the video generation means generates a video at the virtual viewpoint using the texture determined in the fourth step (for example, step S12 of FIG. 2). And a program for causing a computer to execute the program.

  According to the present invention, by executing a program using a computer, when occlusion occurs in any one of a plurality of viewpoint videos by the first texture acquisition unit, each of a plurality of objects forming the occlusion is separated. Thus, the texture of each of the plurality of objects is acquired. Further, when occlusion occurs in any one of the plurality of viewpoint videos by the second texture acquisition means, it is hidden by this occlusion from the viewpoint videos other than the viewpoint video in which this occlusion is generated. I decided to get the texture of the object including the part. Further, when occlusion occurs in any of a plurality of viewpoint videos by the third texture acquisition means, an object including a portion hidden by this occlusion from the frame before or after the frame where the occlusion occurs is displayed. I decided to get a texture. Further, the occlusion from the texture acquired by the first texture acquisition means, the texture acquired by the second texture acquisition means, and the texture acquired by the third texture acquisition means by the determination means. It was decided to determine what to use as the texture of the object including the hidden part. In addition, the video generation unit generates the video at the virtual viewpoint using the texture determined by the determination unit. For this reason, the effect similar to the effect mentioned above can be produced.

  According to the present invention, the visual quality of a virtual viewpoint video can be improved.

It is a block diagram of the virtual viewpoint video generation device concerning one embodiment of the present invention. It is a flowchart of the virtual viewpoint video generation processing performed by the virtual viewpoint video generation device. It is a flowchart of the virtual viewpoint video generation processing performed by the virtual viewpoint video generation device.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the constituent elements in the following embodiments can be appropriately replaced with existing constituent elements, and various variations including combinations with other existing constituent elements are possible. Accordingly, the description of the following embodiments does not limit the contents of the invention described in the claims.

[Configuration of Virtual Viewpoint Video Generating Device 1]
FIG. 1 is a block diagram of a virtual viewpoint video generation device 1 according to an embodiment of the present invention. The virtual viewpoint video generation device 1 receives a plurality of viewpoint videos that are videos from different viewpoints, and generates a virtual viewpoint video that is a video at a virtual viewpoint from these viewpoint videos. The virtual viewpoint video generation apparatus 1 includes an object separation unit 10, an inter-camera interpolation unit 20, an inter-frame interpolation unit 30, a determination unit 40, a video generation unit 50, and a control unit 60.

  The object separation unit 10 performs separation processing (see processing in step S6 in FIG. 2 described later). According to this, when occlusion occurs in any of a plurality of viewpoint videos, it is possible to separate each of the plurality of objects forming the occlusion and acquire the texture of each of the plurality of objects.

  The inter-camera interpolation unit 20 performs an inter-camera interpolation process (see the process in step S7 in FIG. 2 described later). According to this, when occlusion occurs in any of a plurality of viewpoint videos, the texture of an object including a portion hidden by occlusion from those viewpoint videos excluding the viewpoint video where the occlusion occurs. Can be obtained.

  The inter-frame interpolation unit 30 performs inter-frame interpolation processing (see processing in step S8 in FIG. 2 described later). According to this, when occlusion occurs in any of a plurality of viewpoint videos, it is possible to acquire the texture of an object including a portion hidden by occlusion from a frame before or after the frame where the occlusion occurs. .

  The determination unit 40 is a portion hidden by occlusion from among the texture acquired by the object separation unit 10, the texture acquired by the inter-camera interpolation unit 20, and the texture acquired by the inter-frame interpolation unit 30. What is used as the texture of an object including

  The video generation unit 50 generates a virtual viewpoint video using the texture determined by the determination unit 40.

  The control unit 60 performs various processes as described later with reference to FIGS.

[Operation of Virtual Viewpoint Video Generation Device 1]
The operation of the virtual viewpoint video generation device 1 having the above configuration will be described below with reference to FIGS.

  FIG. 2 is a flowchart of a virtual viewpoint video generation process performed by the virtual viewpoint video generation device 1.

  In step S <b> 1, for each of the plurality of viewpoint videos, the control unit 60 holds between the planar projection matrix H established between the viewpoint video and the field coordinates and the field coordinates at the viewpoint and virtual viewpoint of the viewpoint video. The central projection matrix P is estimated, and the process proceeds to step S2. The planar projection matrix H is estimated by designating four or more corresponding points of the viewpoint video on the field plane. The central projection matrix P is estimated by designating four or more corresponding points of the viewpoint video on the field plane and designating two or more corresponding points on the field space.

  In step S2, the control unit 60 divides each frame image into a foreground area and a background area in each of the plurality of viewpoint videos, and moves the process to step S3. Specifically, a background area in which the foreground area is not captured is acquired, a background difference is performed from the acquired background area and the current frame image, and a mask image in which the foreground area and the background area are divided is generated. Note that the foreground area is an area whose position and shape change with time. When the target space is a soccer field, for example, it is an object area such as a soccer player, a referee, or a ball. The background area is an area other than the foreground area.

  In step S3, the control unit 60 performs object tracking processing using a particle filter for each object by time-series processing between frames in the current frame, and moves the processing to step S4. The time-series object tracking processing by the particle filter will be described in detail below with reference to FIG.

  FIG. 3 is a flowchart of time-series object tracking processing by the particle filter performed by the control unit 60.

  In step S31, the control unit 60 sets the circumscribed area of the mask image generated in step S2 as a particle filter circumscribed frame for each object in a predetermined frame, and moves the process to step S32. Note that the circumscribing area of the mask image may be set by the user with respect to the object in which the occlusion has occurred by visually confirming the mask image generated in step S2, for example.

  In step S32, the control unit 60 defines a state quantity c (t) for each particle existing in the particle filter of each object at time t as shown in the following equation (1), and moves the process to step S33. .

  In equation (1), x (t) and y (t) indicate the two-dimensional coordinates of the particle at time t, Δx (t) indicates the lateral velocity of the particle at time t, and Δy (t) Indicates the vertical velocity of the particles at time t. Note that the state transition function is set as shown in the following equation (2) using Gaussian noise ω (t) on the assumption that the particles perform a uniform linear motion between frames.

  In step S33, the control unit 60 obtains the likelihood of each particle, and moves the process to step S34. Specifically, first, it is determined whether the position of each particle is the foreground area or the background area in the mask image. Then, the likelihood is set to zero for the particles determined to be the background region, and the likelihood is set based on the prior information given for each object for the particles determined to be the foreground region.

For example, when the object is a soccer player, based on the color information I _{ref of the} uniform, it is determined whether or not the pixel value I of the particle position satisfies the following formula (3) using the threshold th _ref. . And when satisfy | filling, likelihood is set to zero, and when not satisfy | filling, likelihood is set to "1".

  In step S34, the control unit 60 eliminates the particles having the likelihood of zero in step S33, rearranges the particles in the vicinity of the particles having the likelihood “1”, and moves the process to step S35.

  In step S35, the control unit 60 calculates the circumscribed area of the particle group rearranged in step S34 for each object, updates the particle filter circumscribed frame of each object, and moves the process to step S36.

  In step S36, the control unit 60 determines whether or not the processing in steps S32 to S35 has been performed up to the final frame. If it is determined that the process has not been performed, the process proceeds to step S37. If it is determined that the process has been performed, the process proceeds to step S4 in FIG.

  In step S37, the control unit 60 updates the process target frame, and moves the process to step S32.

  Returning to FIG. 2, in step S4, the control unit 60 determines whether or not occlusion has occurred. If it is determined that the error has occurred, the process proceeds to step S5. If it is determined that the error has not occurred, the process proceeds to step S12.

  In step S5, the control unit 60 detects an occlusion area based on the result of dividing the foreground area and the background area in step S2 and the result in the current frame by the object tracking process in step S3. The process is moved to S6. Specifically, for all the objects tracked by the object tracking process in step S3, a plurality of objects are included by determining in which connected foreground area the object is included by the position of the circumscribed frame of the particle filter. The connected foreground area is obtained, and the obtained foreground area is set as an occlusion area.

  In step S6, the object separation unit 10 performs a separation process on the occlusion area detected in step S5, and moves the process to step 7. Specifically, for the occlusion area detected in step S5, a plurality of objects forming the occlusion are separated, and textures of the plurality of objects are obtained. Object separation includes, for example, a method of separating a plurality of viewpoint videos by excluding a viewpoint video in which this occlusion occurs, that is, by projecting a texture in a viewpoint video at another viewpoint, or in step S3. A method of separating based on the result of the object tracking process can be applied.

  In step S7, the inter-camera interpolation unit 20 performs inter-camera interpolation processing on the occlusion area detected in step S5, and the process proceeds to step S8. Specifically, with respect to the occlusion area detected in step S5, a part of the plurality of viewpoint videos excluding the viewpoint video in which this occlusion occurs, that is, a portion hidden by occlusion from the viewpoint video at another viewpoint Get the texture of the object containing.

  Here, in the object tracking process in step S3, in order to track each object between the viewpoints of the plurality of viewpoint videos, the same ID (identification) is assigned between the viewpoints of the plurality of viewpoint videos for each object. . For this reason, in step S7, by using the ID described above, an object including a portion hidden by occlusion can be discriminated in the viewpoint video of another viewpoint, and a texture can be acquired from the viewpoint video of the other viewpoint. it can.

  In step S8, the inter-frame interpolation unit 30 performs inter-frame interpolation processing on the occlusion area detected in step S5, and the process proceeds to step S9. Specifically, for the occlusion area detected in step S5, the texture of the object including the portion hidden by the occlusion is acquired from the frame before or after the frame where the occlusion occurs.

  Here, in the object tracking process in step S3, in order to track each object between frames, the same ID is allocated between frames for each object. For this reason, also in step S8, by using the above-mentioned ID, an object including a portion hidden by occlusion can be discriminated in another frame, and a texture can be acquired from the other frame.

  In step S9, the determination unit 40 determines whether it is appropriate to use the texture acquired by the separation process in step S6 as the texture of the object including the portion hidden by occlusion, and performs the process in step S10. Move. For detection, object detection is applied for each texture. When only one object is detected, in other words, when the missing or excessive area of the object is equal to or less than the threshold value, it is determined to be appropriate. On the other hand, when an object is not detected, or when a plurality of objects are detected, in other words, when an object loss or excess area is larger than a threshold, it is determined that the object is not appropriate.

  In the separation process, an object to be separated may be separated while some or all of the other objects remain connected. The area of another object connected to the object to be separated is called an excess area as described above.

  In the above-described object detection, for example, HOG (Histogram of Oriented Gradient) feature values of the image of the object group are learned as positive examples according to the type of the target object, and the SVM (Support Vector Machine) is used. A discriminator may be created.

  In step S10, the determination unit 40 moves the process to step S12 if it is determined to be appropriate in step S9, and moves to step S11 if it is determined that it is not appropriate in step S9.

  In step S11, the determination unit 40 selects an object including a portion hidden by occlusion from the texture acquired by the inter-camera interpolation process in step S7 and the texture acquired by the inter-frame interpolation process in step S8. What is appropriate for use as a texture is determined, and the process proceeds to step S12. For the determination, the following first condition and second condition are used. The first condition is that no occlusion occurs in the viewpoint video in which the texture is acquired by the inter-camera interpolation unit 20 for an object including a portion hidden by occlusion. The second condition is that the angle difference between the viewpoint of the viewpoint video from which the texture is acquired by the inter-camera interpolation unit 20 and the virtual viewpoint is equal to or less than a predetermined threshold. If both the first condition and the second condition are satisfied, it is determined that the texture acquired by the inter-camera interpolation process in step S7 is appropriate. On the other hand, if at least one of the first condition and the second condition is not satisfied, it is determined that the texture acquired by the inter-frame interpolation process in step S8 is appropriate.

  Note that the angle difference between the viewpoint of the viewpoint video from which the texture is acquired by the inter-camera interpolation unit 20 and the virtual viewpoint is based on the orientation of the viewpoint of the viewpoint video from which the texture is acquired by the inter-camera interpolation unit 20 and the virtual viewpoint. It can be obtained from the similarity defined based on the inner product of the direction. When the degree of similarity is greater than or equal to the threshold, it is determined that the angle difference is less than or equal to a predetermined threshold, and when the degree of similarity is less than the threshold, the angle difference is greater than the predetermined threshold. Is determined. The inner product of the viewpoint direction of the viewpoint video from which the texture is acquired by the inter-camera interpolation unit 20 and the direction of the virtual viewpoint is the optical axis obtained from the virtual viewpoint vector and the central projection matrix P estimated in step S1. It is possible to calculate from these vectors.

  In step S12, the video generation unit 50 generates a virtual viewpoint video, and ends the process of FIG. Specifically, based on the texture position in the video and the planar projection matrix H estimated in step S1, each texture is arranged on a three-dimensional field to synthesize a virtual viewpoint video.

  In step S9, when it is determined that the texture acquired by the separation process in step S6 is appropriate, the texture acquired by the separation process in step S6 is used for generating a virtual viewpoint video. If it is determined in step S11 that the texture acquired by the inter-camera interpolation process in step S7 is appropriate, the texture acquired by the inter-camera interpolation process in step S7 is used to generate a virtual viewpoint video. Use.

  On the other hand, if it is determined in step S11 that the texture acquired by the interframe interpolation process in step S8 is appropriate, the texture acquired by the interframe interpolation process in step S8 is used to generate a virtual viewpoint video. Use. In this case, the texture of the object including the portion hidden by the occlusion is obtained for the texture of the nearest frame properly acquired by the foreground extraction process in step S2 or the separation process in step S6, and the object in step S3 is obtained. The obtained texture is arranged at the position estimated by the tracking process.

  According to the virtual viewpoint video generation device 1 described above, the following effects can be obtained.

  The virtual viewpoint video generation device 1 separates an object to acquire a texture, an inter-camera interpolation process to acquire a texture from another viewpoint video, an inter-frame interpolation process to acquire a texture from another frame, I do. For this reason, since the generation of the virtual viewpoint video can be performed by combining not only the separation process and the inter-camera interpolation process but also the inter-frame interpolation process, the visual quality of the virtual viewpoint video can be improved.

  In addition, in step S9 of FIG. 2, the virtual viewpoint video generation device 1 determines whether or not the texture acquired by the separation process is appropriate. If it is determined that the texture is appropriate, the virtual viewpoint video generation device 1 uses the texture to determine the virtual viewpoint. Generate video. On the other hand, if it is determined that it is not appropriate, it is determined in step S11 of FIG. 2 whether or not the texture acquired by the inter-camera interpolation processing is appropriate. The virtual viewpoint video is generated using the texture, and when it is determined that the virtual viewpoint video is not appropriate, the virtual viewpoint video is generated using the texture acquired by the inter-frame interpolation processing. For this reason, the acquired textures can be used in the order of priority of the texture acquired by the separation process, the texture acquired by the inter-camera interpolation process, and the texture acquired by the inter-frame interpolation process.

  In addition, in step S9 of FIG. 2, the virtual viewpoint video generation device 1 determines whether or not the texture acquired by the separation process is appropriate. If it is determined that the texture is appropriate, the virtual viewpoint video generation device 1 uses the texture to determine the virtual viewpoint. Generate video. On the other hand, if it is determined that it is not appropriate, it is determined in step S11 of FIG. 2 whether or not the texture acquired by the inter-camera interpolation processing is appropriate. The virtual viewpoint video is generated using the texture, and when it is determined that the virtual viewpoint video is not appropriate, the virtual viewpoint video is generated using the texture acquired by the inter-frame interpolation processing. For this reason, step S11 is not performed when it determines with it being appropriate in step S9. Therefore, there are cases where the processing of the virtual viewpoint video generation device 1 can be reduced.

  The virtual viewpoint video generation device 1 uses object detection using the HOG feature amount when determining whether or not the texture acquired by the separation process is appropriate in step S9 in FIG. For this reason, it is possible to more appropriately determine which one of the texture acquired by the separation process and the texture acquired by the inter-camera interpolation process or the inter-frame interpolation process should be used. Therefore, the visual quality of the virtual viewpoint video can be further improved.

  Further, the virtual viewpoint video generation device 1 uses the case where the occlusion occurs in the viewpoint video in which the texture is acquired by the inter-camera interpolation unit 20 for the object including the portion hidden by the occlusion, and the inter-camera interpolation unit 20 In the case where the angle difference between the viewpoint of the viewpoint video from which the texture is acquired and the virtual viewpoint is larger than the threshold value, the texture acquired by the inter-frame interpolation process is used instead of the texture acquired by the inter-camera interpolation process. Therefore, the visual quality of the virtual viewpoint video can be further improved.

  The processing of the virtual viewpoint video generation apparatus 1 of the present invention is recorded on a computer-readable non-transitory recording medium, and the program recorded on this recording medium is read by the virtual viewpoint video generation apparatus 1 and executed. Thus, the present invention can be realized.

  Here, for example, a nonvolatile memory such as an EPROM or a flash memory, a magnetic disk such as a hard disk, a CD-ROM, or the like can be applied to the above-described recording medium. Further, reading and execution of the program recorded on the recording medium is performed by a processor provided in the virtual viewpoint video generation apparatus 1.

  Further, the above-described program may be transmitted from the virtual viewpoint video generation device 1 storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  Further, the above-described program may be for realizing a part of the above-described function. Furthermore, what can implement | achieve the above-mentioned function in combination with the program already recorded on the virtual viewpoint image | video production | generation apparatus 1, and what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design that does not depart from the gist of the present invention.

  For example, in the above-described embodiment, in the inter-frame interpolation process in step S8, a texture is acquired from another frame for an object including a portion hidden by occlusion. Here, the other frame may be a frame in the same viewpoint video as the viewpoint video including the object in which occlusion occurs, or a frame in a different viewpoint video. When the frames in the same viewpoint video are used, since the viewpoints are the same as in the case of the frames in different viewpoint videos, the visual quality of the virtual viewpoint video can be further improved.

  In the above-described embodiment, each process is performed in the order of the separation process (step S6), the inter-camera interpolation process (step S7), and the inter-frame interpolation process (step S8). However, for example, the inter-camera interpolating process and the inter-frame interpolating process may be performed before the process of step S11 in FIG. According to this, when it is determined to be appropriate in step S9, not only the step S11 but also the inter-camera interpolation process and the inter-frame interpolation process are not performed. Therefore, the processing of the virtual viewpoint video generation device 1 may be further reduced.

DESCRIPTION OF SYMBOLS 1 ... Virtual viewpoint image generation apparatus 10 ... Object separation part 20 ... Interpolation part between cameras 30 ... Interpolation part between frames 40 ... Determination part 50 ... Video generation part 60 ... Control Part

Claims (9)

A virtual viewpoint video generation device that generates a virtual viewpoint video from a plurality of viewpoint videos,
When occlusion occurs in any of the plurality of viewpoint videos, a first texture acquisition unit that separates each of a plurality of objects forming the occlusion and acquires a texture of each of the plurality of objects;
When occlusion occurs in any of the plurality of viewpoint videos, a texture of an object including a portion hidden by the occlusion is obtained from the plurality of viewpoint videos except for the viewpoint video in which the occlusion occurs. Second texture acquisition means for
When occlusion occurs in any of the plurality of viewpoint videos, third texture acquisition that acquires the texture of an object including a portion hidden by the occlusion from the frame before or after the frame in which the occlusion occurs Means,
The texture acquired by the first texture acquisition means, the texture acquired by the second texture acquisition means, and the texture acquired by the third texture acquisition means are hidden by the occlusion. Determining means for determining what is used as the texture of the object including the portion that is
A virtual viewpoint video generation apparatus comprising: video generation means for generating a video at the virtual viewpoint using the texture determined by the determination means.   The determination unit is used as a texture of an object including a portion hidden by the occlusion, the texture acquired by the first texture acquisition unit, the texture acquired by the second texture acquisition unit, the first 3. The virtual viewpoint video generation device according to claim 1, wherein the determination is performed in order of priority of the textures acquired by the texture acquisition unit of 3.   The third texture acquisition means acquires the texture of the object from a frame before or after the frame where the occlusion occurs, among a plurality of frames constituting the viewpoint video where the occlusion occurs. The virtual viewpoint video generation device according to claim 1, wherein the virtual viewpoint video generation device is provided. The determination means includes
A first determination procedure for determining whether to use the texture acquired by the first texture acquisition means;
Performing a second determination procedure for determining which one of the texture acquired by the second texture acquisition means and the texture acquired by the third texture acquisition means is to be used. The virtual viewpoint video generation device according to any one of claims 1 to 3.   The determination unit performs object detection on the texture acquired by the first texture acquisition unit in the first determination procedure, and determines based on a result of object detection. 4. Virtual viewpoint video generation storehouse according to 4.   The virtual viewpoint video generation apparatus according to claim 5, wherein the determination unit uses a HOG feature amount in the object detection. In the second determination procedure, the determination unit includes:
No occlusion has occurred in the viewpoint video in which the texture is acquired by the second texture acquisition unit for the object including the portion hidden by the occlusion, and the texture is acquired by the second texture acquisition unit When the angle difference between the viewpoint of the viewpoint video and the virtual viewpoint is equal to or less than a predetermined threshold, it is determined that the texture acquired by the second texture acquisition unit is used,
At least one of the case where occlusion occurs in the viewpoint image in which the texture is acquired by the second texture acquisition unit for the object including the portion hidden by the occlusion and the case where the angle difference is larger than the threshold value. In this case, it is determined that the texture acquired by the third texture acquisition unit is used. 7. The virtual viewpoint video generation apparatus according to claim 4, wherein: In a virtual viewpoint video generation device that includes a first texture acquisition unit, a second texture acquisition unit, a third texture acquisition unit, a determination unit, and a video generation unit, and generates a video at a virtual viewpoint from a plurality of viewpoint videos A virtual viewpoint video generation method,
When the first texture acquisition unit separates each of the plurality of objects forming the occlusion when occlusion occurs in any of the plurality of viewpoint videos, the first texture acquisition unit acquires the texture of each of the plurality of objects. 1 step,
When the second texture acquisition unit causes occlusion in any of the plurality of viewpoint videos, the second texture acquisition unit hides the viewpoint video from the viewpoint videos in which the occlusion is generated out of the plurality of viewpoint videos. A second step of obtaining a texture of the object including the portion that is
When occlusion occurs in any of the plurality of viewpoint videos, the third texture acquisition unit includes an object including a portion hidden by the occlusion from a frame before or after the frame where the occlusion occurs. A third step of obtaining a texture;
The determination means is hidden by the occlusion from the texture acquired in the first step, the texture acquired in the second step, and the texture acquired in the third step. A fourth step of determining what to use as the texture of the object including the part that is present;
A virtual viewpoint video generation method comprising: a fifth step in which the video generation means generates a video at the virtual viewpoint using the texture determined in the fourth step. In a virtual viewpoint video generation device that includes a first texture acquisition unit, a second texture acquisition unit, a third texture acquisition unit, a determination unit, and a video generation unit, and generates a video at a virtual viewpoint from a plurality of viewpoint videos A program for causing a computer to execute a virtual viewpoint video generation method,
When the first texture acquisition unit separates each of the plurality of objects forming the occlusion when occlusion occurs in any of the plurality of viewpoint videos, the first texture acquisition unit acquires the texture of each of the plurality of objects. 1 step,
When the second texture acquisition unit causes occlusion in any of the plurality of viewpoint videos, the second texture acquisition unit hides the viewpoint video from the viewpoint videos in which the occlusion is generated out of the plurality of viewpoint videos. A second step of obtaining a texture of the object including the portion that is
When occlusion occurs in any of the plurality of viewpoint videos, the third texture acquisition unit includes an object including a portion hidden by the occlusion from a frame before or after the frame where the occlusion occurs. A third step of obtaining a texture;
The determination means is hidden by the occlusion from the texture acquired in the first step, the texture acquired in the second step, and the texture acquired in the third step. A fourth step of determining what to use as the texture of the object including the part that is present;
A program for causing the computer to execute a fifth step in which the video generation means generates a video at the virtual viewpoint using the texture determined in the fourth step.

Images