JP2020140519A - Image generation device and control method thereof - Google Patents

Abstract

To be able to generate a virtual viewpoint image with improved position accuracy of back ground and foreground.SOLUTION: An image generation device generating a virtual viewpoint image from a multi-view image shot by a plurality of cameras comprises: a setting unit setting a virtual model representing a virtual straight line or virtual plane including a position of a specified camera among the plurality of cameras and a back ground model representing three-dimensional shape of the back ground in a three-dimensional space; and a generation unit generating a virtual viewpoint image from a specified virtual viewpoint by using the multi-view image and the back ground model. The generation unit uses, for generation of the virtual viewpoint image, the shot image by the specified camera in preference to the shot image by other cameras as closer the specified virtual viewpoint is to the virtual model.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image generation device that generates a virtual viewpoint image from a multi-viewpoint image taken by a plurality of cameras and a control method thereof.

A technique is known in which a plurality of cameras are installed at different positions to shoot from a multi-viewpoint position, and a virtual viewpoint image is generated from an arbitrarily specified viewpoint position using the multi-viewpoint image obtained by the shooting. There is. According to this technology, images obtained from various viewpoint positions and angles can be viewed. Therefore, for highlight scenes such as soccer and basketball, it is possible to provide an image that can give a high sense of presence to the viewer as compared with a normal image.

Patent Document 1 discloses a technique for generating a virtual viewpoint image from a multi-viewpoint image obtained by synchronously photographing a stadium with a plurality of cameras. According to Patent Document 1, a foreground texture image, a background texture image, a three-dimensional model, etc. are generated from a multi-viewpoint image, and an appropriate foreground texture and background texture are mapped to the three-dimensional model from an arbitrarily specified viewpoint. A virtual viewpoint image is generated by superimposing and synthesizing.

Japanese Unexamined Patent Publication No. 2017-21128

However, in the virtual viewpoint image generated by the conventional method, there may be a problem due to the low accuracy of the positional relationship between the foreground and the background. For example, when the stadium is measured in advance and a 3D shape model of the background (hereinafter referred to as the background model) is generated by 3D modeling, the background model and the actual shape of the stadium due to measurement errors and low 3D modeling accuracy. And there may be an error. As a result, the foreground image cannot be superimposed on the exact position in the background image. In a virtual viewpoint image of a finish scene where multiple athletes finish in at almost the same time in a race or swimming, if there is a deviation in the positional relationship between the athlete and the finish line, an accurate judgment will be made when using the virtual viewpoint image for video judgment. Cannot be done.

The present invention has been made in view of the above problems, and an object of the present invention is to generate a virtual viewpoint image with improved position accuracy of the background and the foreground.

An image generator according to one aspect of the present invention for solving the above problems has the following configuration. That is,
An image generator that generates virtual viewpoint images from multi-viewpoint images taken by multiple cameras.
A setting means for setting a virtual model representing a virtual straight line or a virtual plane including the position of a specific camera among the plurality of cameras and a background model representing a three-dimensional shape of the background in a three-dimensional space.
A generation means for generating a virtual viewpoint image from a specified virtual viewpoint by using the multi-viewpoint image and the background model is provided.
The generation means uses the image captured by the specific camera in preference to the image captured by the other camera to generate the virtual viewpoint image as the distance between the designated virtual viewpoint and the virtual model becomes closer. ..

According to the present invention, it is possible to generate a virtual viewpoint image with improved position accuracy of the background and the foreground.

The figure which shows the configuration example of the virtual viewpoint image generation apparatus in 1st Embodiment. The figure which shows the virtual plane at the sprint finish position in 1st Embodiment. The figure explaining the line-of-sight of a virtual plane and a virtual viewpoint in the 1st Embodiment. The flowchart explaining the correction of the background model in 1st Embodiment. The figure explaining virtual plane generation and background model correction in 1st Embodiment. A flowchart illustrating a process of generating a virtual viewpoint image. The figure which shows the display example of a virtual viewpoint image. The figure explaining the image pickup system and the virtual viewpoint in 3rd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

<First Embodiment>
FIG. 1 is a block diagram illustrating the overall configuration of the virtual viewpoint image generation device 100 according to the first embodiment.

In FIG. 1, the imaging system 110 is a system that captures a multi-viewpoint image necessary for generating a virtual viewpoint image, and is composed of a plurality of cameras (shooting devices) 101a to 101z. The cameras 101a to 101z are installed so as to surround the competition field of the stadium such as a stadium, and an IP address is assigned to each of them. The image processing unit 103 controls shooting conditions such as the shutter speed, aperture value, and focal length of the cameras 101a to 101z, and shooting start / end, and performs synchronous shooting. A multi-viewpoint image composed of a plurality of captured images captured by the imaging system 110 is input to the image input unit 104 of the image processing unit 103 via the communication line 102. The image input unit 104 performs various correction processes such as brightness correction, gradation correction, and resolution correction on the captured image, and then outputs the processed captured image to the separation unit 105.

The separation unit 105 separates the foreground region, which is the main subject such as an athlete, and the background area other than the main subject from the captured image of the cameras 101a to 101z, and separates the foreground texture generation unit 106, the background texture generation unit 107, and the image. Output to the generation unit 108. A known technique can be applied to the foreground background separation method used by the separation unit 105. For example, before the start of the competition, only the stadium is photographed in advance, the photographed image is stored as a reference image, and when the difference value for each pixel between the camera photographed image and the reference image is equal to or greater than a predetermined threshold value, the foreground area There is a way to judge that. Alternatively, there is a method of determining the foreground region when the difference value for each pixel of a plurality of images taken by the same camera at different times is equal to or greater than a predetermined threshold value. The operation UI 109 is an interface for allowing the user to specify a virtual camera (virtual viewpoint) and a virtual plane (described later).

The foreground texture generation unit 106 and the background texture generation unit 107 classify and accumulate the foreground texture and the background texture of all the captured images by the cameras 101a to 101z for each camera ID. The image generation unit 108 generates a virtual viewpoint image based on the foreground model, the foreground texture, the background model, the background texture, the virtual viewpoint, the virtual plane, and the virtual line-of-sight evaluation value. The foreground model is generated and held by the image generation unit 108 based on the foreground image input from the separation unit 105. Further, the background model is generated in advance based on measurement data or the like as described later, and is held by the image generation unit 108.

As for the foreground, which is the main subject, an image from the entire circumference of the main subject is taken by the imaging system 110. Therefore, the three-dimensional shape model (foreground model) of the foreground is generated by the camera-based (image-based) method. On the other hand, since the number of cameras in the field of view is small in stadiums and structures, it is difficult to generate a three-dimensional background model (background model) based on the camera. Therefore, the shape of the stadium or the structure is measured in advance with a laser scanner or the like (not shown), and the three-dimensional shape model data is created by the three-dimensional modeling software or the like and saved in the image generation unit 108 as a background model. deep. In the background model, an error occurs between the background shape and the actual background shape due to measurement error and approximation error in modeling. Due to this error, misalignment occurs when the background model and the camera-based foreground model are superimposed and combined. In the present embodiment, the correction for eliminating the above error is performed by using the virtual plane generated by the virtual plane generation unit 112 described later.

Next, the image generation unit 108 maps the foreground texture and the background texture to the foreground model and the background model based on the information of the virtual camera (position and direction of the virtual viewpoint) specified by the operation UI 109, and generates a virtual viewpoint image. To do. Which camera's captured image is used as the background texture and the foreground texture is determined according to the output value of the line-of-sight determination unit 114 described later. The image generation unit 108 displays the generated virtual viewpoint image on the display 111.

Next, the virtual plane generated by the virtual plane generation unit 112 will be described. The virtual plane generation unit 112 puts a virtual model representing a virtual straight line or a virtual plane including the position of a specific camera among a plurality of cameras and a background model used for generating a virtual viewpoint image into the same three-dimensional virtual space. This is an example of a configuration that functions as a setting unit for setting. In the following, a configuration for setting a virtual plane parallel to the line-of-sight direction of the virtual viewpoint will be described as a virtual model. The virtual plane of the present embodiment is a plane including a characteristic area in the stadium, is superimposed and displayed on the virtual viewpoint image, and is used for correcting the position error between the background image and the foreground image. Further, the virtual plane can be used as an auxiliary display for determining the ranking of sprinting, for example, as shown in FIG.

FIG. 2 is a diagram showing an example in which the virtual viewpoint image generation device 100 according to the first embodiment is applied to generate a virtual viewpoint image at the time of a sprint finish. A plane orthogonal to the plane (plane) of the track 205, including the finish line 204 drawn on the track 205 for competition, is set as the virtual plane 201. The imaging system 110 having a plurality of cameras photographs the athletes 202 and 203 passing through the finish line 204 drawn on the track 205 to obtain a multi-viewpoint image. The image processing unit 103 generates a virtual viewpoint image from the acquired multi-view image and displays it on the display 111.

It is assumed that the vicinity of the finish line 204 including the athlete 202 and the athlete 203 is generated from the multi-viewpoint image taken by the imaging system 110. In this case, the separation unit 105 separates the regions of the athlete 202 and the athlete 203 as the foreground image and the other regions as the background image. The background model is a three-dimensional shape model generated by three-dimensional modeling by measuring the shape of the stadium including the track 205 in advance. The image generation unit 108 maps the background texture to the surface of the background model to generate the background image observed from the virtual viewpoint, and maps the foreground texture to the surface of the foreground model to generate the foreground image observed from the virtual viewpoint. To do. The image generation unit 108 generates a virtual viewpoint image by superimposing and synthesizing the generated background image and the foreground image.

The viewer (user) can easily determine the ranking of each player by observing the virtual viewpoint image from the virtual viewpoint in the vicinity of the finish line 204. However, if there is an error in the positional relationship between the background image including the finish line 204 and the foreground image, an accurate ranking cannot be determined. In the present embodiment, such an error in the positional relationship between the background image and the foreground image is eliminated. Hereinafter, a configuration for eliminating the error between the positions of the background image and the foreground image will be described.

FIG. 3 is a diagram showing an example in which the imaging system 110 according to the first embodiment is applied to athletics. FIG. 4 is a flowchart illustrating the background model adjustment process according to the first embodiment. As shown in FIG. 3, the cameras 101a to 101f are arranged at substantially equal intervals so as to surround the finish line 204, which is a photographing target area. Of these plurality of cameras, the camera 101a and the camera 101d are arranged above both extension lines of the finish line 204, and their line-of-sight directions are directed to the finish line 204 (S401). The camera 101a and the camera 101d are cameras whose line-of-sight directions include the finish line 204 and are included in the virtual plane 201 orthogonal to the ground, and are also referred to as specific cameras below. In the example of FIG. 4, the number of specific cameras is two, but the number is not limited to this, and the number of specific cameras may be one or three or more. In the present embodiment, it is assumed that the specific camera faces the finish line 204 and the line-of-sight direction of the specific camera is parallel to the virtual plane 201, but the present invention is not limited to this. For example, if the position of a specific camera is included in the virtual plane 201 and the shooting range of the camera includes the virtual plane, the virtual plane does not have to be located at the center of the shooting range.

FIG. 5 is a diagram showing an example of an image captured by the camera 101a or the camera 101d (hereinafter, will be described as an image captured by the camera 101d). In the case of a competition such as sprinting, it is easy to determine the finish ranking by using an image taken from an extension of the finish line as shown in FIG. 5A. Therefore, in the present embodiment, at least one of the plurality of cameras constituting the imaging system 110 is arranged on an extension line of the finish line. However, since the placement position differs depending on the type of sports competition, the optimum placement location is set according to the type of competition.

After arranging the cameras, the user uses the operation UI 109 to specify the virtual viewpoint at a position where the finish line 204 can be easily seen, and instructs the generation of the virtual viewpoint image. The image processing unit 103 generates a virtual viewpoint image from a designated virtual viewpoint using the multi-view image captured by the imaging system 110, and displays it on the display 111 (S402). The virtual viewpoint specified in S402 may be preset based on the background model, or may be automatically set based on the background model.

The virtual plane generation unit 112 causes the user to specify two points on the finish line 204 by the operation UI 109 in the virtual viewpoint image displayed on the display 111 (S403). The virtual plane generator 112 treats the two points specified on the display screen as points specified for the surface including the track 205 in the background model, and determines the line segment AB having the specified two points as both ends. To do. Then, the virtual plane generation unit 112 calculates a plane including the line segment AB and orthogonal to the plane (for example, the ground) of the track 205 as the plane equation ax + by + cz + d = 0 based on the coordinate XYZ axes shown in FIG. S404). The virtual plane generation unit 112 stores the calculated plane equation as a virtual model in the virtual plane storage unit 113. The virtual plane stored in the virtual plane storage unit 113 is sent to the image generation unit 108.

The image generation unit 108 is an example of a drawing unit that draws a virtual model observed from a virtual viewpoint matching the specific camera on an image captured by the specific camera (camera 101d). For example, the image generation unit 108 sets the virtual plane represented by the plane equation as a part of the constituent of the background model, sets the position of the virtual viewpoint to the position of the specific camera 101d, and sets the virtual plane from the virtual viewpoint. Generates 201 images. Then, the image generation unit 108 superimposes the image of the virtual plane 201 on the captured image of the camera 101d and displays it on the display 111 (S405). FIG. 5B is a display image of the display 111 in which the virtual plane 201 is superimposed on the captured image of the camera 101d. In FIG. 5B, due to an error in the background model, the finish line 204 and the virtual plane 201 projected on the captured image of the camera 101d are misaligned.

The user confirms the positions of the image captured by the camera 101d displayed on the display 111 and the virtual plane 201, and corrects (adjusts) the background model until they match (S406, S407). For example, the user moves / rotates the background model to match the background model with the captured image. In this way, the image generation unit 108 is an example of an adjustment unit that adjusts the three-dimensional model so that the captured image of the specific camera and the drawn virtual model match. FIG. 5C shows a state in which the background model has been modified (adjusted) and the captured image of the camera 101d and the virtual plane 201 are aligned (the virtual plane 201 is observed as a straight line, and the straight line is the finish line 204. The state that matches with) is shown. By adjusting the background model as described above, the position and direction of the image captured by the camera 101d and the background model match with high accuracy.

After that, the image generation unit 108 is an example of a generation unit that generates a virtual viewpoint image corresponding to the designated virtual viewpoint by using the multi-viewpoint image taken by the plurality of cameras and the background model adjusted as described above. Is. As described above, by matching the image taken from a specific camera and the background model using the image of the virtual model, the accuracy of the positional relationship between the background image and the foreground image of the virtual viewpoint image in the vicinity of the virtual model can be improved. improves. Therefore, for example, by setting a virtual plane including the finish line 204 as a virtual model, the positional relationship between the finish line 204 and the athlete can be reproduced with higher accuracy in the virtual viewpoint image for observing the vicinity of the finish line 204. ..

Since the relationship between the background model and the specific camera 101d is adjusted with high accuracy, the image (texture) taken by the specific camera is prioritized over the image taken by the other cameras in the generation of the virtual viewpoint image. It is preferable to use it. For example, if the closer the virtual viewpoint is to the virtual plane, the image taken by a specific camera is preferentially used to generate the virtual viewpoint image, the positions of the foreground image and the background image in the virtual viewpoint image for observing the region of interest. The relationship can be reproduced with higher accuracy. A specific example of such control will be described below. In the present embodiment, the image generation unit 108 evaluates the proximity of the position and direction between the virtual viewpoint and the virtual model designated for generating the virtual viewpoint image. Then, when the evaluated proximity is equal to or higher than a predetermined level, the image generation unit 108 generates a virtual viewpoint image by using the image captured by a specific camera in preference to the image captured by another camera.

FIG. 6A is a flowchart illustrating a process of generating a virtual viewpoint image according to the first embodiment. First, the line-of-sight determination unit 114 accepts the user designation of the virtual viewpoint input via the operation UI 109 (S601). Next, the line-of-sight determination unit 114 evaluates the proximity of the designated virtual viewpoint and the virtual plane, and acquires the evaluation value p (S602). For example, the line-of-sight determination unit 114 calculates an evaluation value based on the distance between the virtual viewpoint and the virtual plane and the angle between the line-of-sight direction of the virtual viewpoint and the virtual plane (normal line). An embodiment of calculation of the evaluation value p by the line-of-sight determination unit 114 will be described below.

Assuming that XYZ is a three-dimensional coordinate axis and abc is a plane direction vector, the plane equation is ax + by + cz + d = 0. The line-of-sight determination unit 114 calculates an evaluation value based on the distance between the virtual viewpoint and the virtual plane and the angle formed by the virtual viewpoint line-of-sight direction and the virtual plane.

FIG. 3 shows the position and the line-of-sight direction (optical axis direction) of the virtual plane 201 and the virtual camera (virtual viewpoint 301) designated in the vicinity of the virtual plane 201. As shown in FIG. 3, assuming that the position of the virtual viewpoint 301 is (x ₁ , y ₁ , z ₁ ) and the direction vector of the virtual viewpoint 301 in the line-of-sight direction is (l, m, n), the virtual viewpoint 301 is virtual. The distance h to the plane 201 is expressed as [Equation 1].

Further, the angle θ formed by the line-of-sight direction of the virtual viewpoint 301 and the virtual plane 201 is expressed as in [Equation 2].

The line-of-sight determination unit 114 obtains an evaluation value as a function p = f (h, θ) of the distance h from the virtual camera to the plane and the angle θ formed by the line-of-sight direction of the virtual camera and the plane. As an embodiment of the evaluation value p, the angle θ formed by the distance h is multiplied by the sensitivity coefficients α and β, respectively, and the sum of them p = αh + βθ is used as the evaluation value. The evaluation value p becomes smaller as the line of sight of the virtual viewpoint is along the virtual plane, and when p = 0, the line of sight of the virtual viewpoint is completely included in the virtual plane.

The line-of-sight determination unit 114 sends the evaluation value p calculated as described above to the image generation unit 108. When the evaluation value p based on the relationship between the position and direction of the virtual plane and the virtual viewpoint is smaller than ε (p <ε), the image generation unit 108 gives priority to the image captured by a specific camera over the image captured by another camera. To generate a virtual viewpoint image (S603, S604). That is, the image generation unit 108 uses the background texture and the foreground texture captured by the camera 101d as a reference source for texture mapping of the virtual viewpoint image in preference to other cameras. In this way, when the virtual viewpoint is close to the virtual plane 201, the texture captured by the camera 101d is mapped to the background model adjusted as described above. On the other hand, when p ≧ ε, the image generation unit 108 generates a virtual viewpoint image using the multi-viewpoint image without giving priority to the image taken from the specific camera (S603, S605).

7 (a) and 7 (b) are views showing a virtual viewpoint image at the finish of the athlete on the display 111. FIG. 7A is a virtual viewpoint image when the virtual camera is at a position (p ≧ ε) away from the virtual plane. By superimposing the virtual plane 201 on the display 111 as shown in FIG. 7A, the positional relationship between the player and the finish line becomes easier to understand. FIG. 7B is a virtual viewpoint image obtained by arranging the virtual viewpoint on the virtual plane 201, that is, a virtual viewpoint image at an evaluation value p = 0.

As described above, when the closeness evaluated with respect to the virtual viewpoint 301 and the virtual plane 201 is equal to or higher than a predetermined level (in this example, p <ε), the image generation unit 108 uses the background model and the foreground model. The texture to be pasted is acquired from the captured image of the camera 101d as much as possible. The texture of the portion of the background model that is not included in the captured image of the camera 101d is acquired from the captured image of another camera. For example, in the virtual viewpoint image of FIG. 7B, the captured image of the camera 101d is preferentially used as the background texture and the foreground texture. Since the captured image of the camera 101d and the background model are aligned with high accuracy by the process shown in FIG. 4, the positional relationship between the background image and the foreground image is accurately reproduced in the virtual viewpoint image.

As described above, according to the first embodiment, as shown in FIG. 7B, the positional error between the background image and the foreground image is corrected, and the positional deviation between the background image and the foreground image in the virtual viewpoint image is eliminated. To. Therefore, for example, by superimposing and displaying the virtual plane on the finish line, it can be used as an aid for ranking determination at the time of sprinting finish. Further, according to the first embodiment, the closer the virtual viewpoint line of sight is to the virtual plane set in the field of view of the virtual viewpoint image, the higher the placement accuracy of the background image and the foreground image is.

If there are a plurality of cameras whose line-of-sight direction is included in the virtual plane 201, the image taken by which camera is further considered in consideration of the relationship between the line-of-sight direction / position of the virtual viewpoint and the line-of-sight direction / position of the camera. May be decided whether to use. That is, the proximity is evaluated based on the distance between the virtual viewpoint and the camera, and the inner product of the virtual viewpoint and the line-of-sight direction of the camera, and based on this evaluation result, which camera's captured image is preferentially used is determined. You may. For example, as described with reference to FIG. 3, when the lines of sight of the camera 101a and the camera 101d are included in the virtual plane 201, the camera to be used preferentially is determined based on the above-mentioned evaluation result.

<Second embodiment>
Hereinafter, the virtual viewpoint image generator according to the second embodiment will be described. In the virtual viewpoint image generation device 100 of the second embodiment, when the proximity between the virtual viewpoint and the virtual plane is equal to or higher than a predetermined level (p <ε), the virtual viewpoint image is switched to the image captured by a specific camera. The configuration of the virtual viewpoint image generation device 100 is the same as that of the first embodiment (FIGS. 1 and 2). However, the determination result of the line-of-sight determination unit 114 is also input to the separation unit 105.

The generation of the virtual plane and the storage of the virtual plane information in the second embodiment are the same as those in the first embodiment. The line-of-sight determination unit 114 calculates the evaluation value p in the same manner as in the first embodiment. It is assumed that the threshold value ε used for comparison with the evaluation value p for determining whether or not the line of sight of the virtual camera is included in the virtual plane is set in advance. FIG. 6B is a flowchart illustrating a process of generating a virtual viewpoint image according to the second embodiment. Hereinafter, the operation of the image generation unit 108 according to the second embodiment will be described with reference to the flowchart of FIG. 6B.

The process of accepting the designation of the virtual viewpoint and acquiring the evaluation value p for evaluating the proximity of the virtual viewpoint and the virtual plane is the same as that of the first embodiment (S601 to S602 of FIG. 6A). When the proximity of the virtual viewpoint 301 and the virtual plane 201 specified by the operation of the operation UI 109 is equal to or higher than a predetermined level (when p <ε), the image generation unit 108 replaces the image captured by the specific camera with the virtual viewpoint image. Is displayed on the display 111 (S613, S614). On the other hand, when the proximity of the virtual viewpoint 301 and the virtual plane 201 is not equal to or higher than a predetermined level (p ≧ ε), the image generation unit 108 generates a virtual viewpoint image from a virtual viewpoint designated by using the multi-view image (). S613, S615).

An example of the image display processing by S614 will be described more specifically. When the proximity of the virtual viewpoint 301 and the virtual plane 201 is equal to or higher than a predetermined level (when p <ε), the line-of-sight determination unit 114 outputs the evaluation value p to 0 to the image generation unit 108 and the separation unit 105. When p = 0 is received from the line-of-sight determination unit 114, the separation unit 105 transmits the entire image captured by the specific camera (camera 101d) to the foreground texture generation unit 106 as a foreground image. Further, when p = 0 is received from the line-of-sight determination unit 114, the image generation unit 108 acquires the foreground image, which is the entire image taken by the specific camera, from the foreground texture generation unit 106 and displays it on the display 111 as it is. By this operation, the display on the display 111 is switched from the virtual viewpoint image generated from the multi-viewpoint image to the captured image taken by a specific camera.

7 (a) to 7 (c) are diagrams showing an example of a state in which a virtual viewpoint image at the finish of a player is displayed on the display 111. As described in the first embodiment, FIG. 7A is a virtual viewpoint image when the virtual camera is located at a position (p ≧ ε) away from the virtual plane. By superimposing the virtual plane 201 on the display 111 as shown in FIG. 7A, the positional relationship between the player and the finish line becomes easier to understand. FIG. 7B is a diagram showing a state in which the virtual viewpoint image at the time of finishing is displayed on the display 111, which is obtained by arranging the virtual viewpoint on the extension line of the finish line (p = 0). FIG. 7B is an image generated by the image processing block of FIG. 1 of the first embodiment, and as described above, a virtual plane is superimposed on the three-dimensional background model to assist the ranking determination.

FIG. 7C is a diagram showing a state in which the image generation unit 108 of the present embodiment displays an image captured by a camera arranged in a virtual plane (for example, the camera 101d of FIG. 4) on the display 111. FIG. 7C is an image generated in S614 of the second embodiment, which is the image itself taken by the camera 101d arranged on the extension line of the finish line. That is, the captured image itself is displayed instead of the virtual viewpoint image generated by separating the finish line and the player. The camera number display 701 may be added to the screen of the display 111. The camera number display 701 makes it possible to clearly indicate to the referee which camera the image is when the virtual viewpoint image generation device 100 is used for camera determination (video determination).

The evaluation value p for determining whether or not to switch from the virtual viewpoint image to the image captured by the specific camera is not limited to the above. For example, it may be a condition that the difference between the position and / or the line-of-sight direction of a specific camera and the position and / or the line-of-sight direction of the virtual viewpoint is within a predetermined range. Alternatively, it may be a condition that the line of sight of the virtual viewpoint points to a specific range in the vicinity of the finish line. Further, the operations of the first embodiment and the second embodiment may be combined. For example, two threshold values ε1 and ε2 (ε1> ε2) are set, and when ε1> p ≧ ε2, a virtual viewpoint image is generated by preferentially using the image captured by the specific camera (S604 of the first embodiment). ), P <2 may be used to switch to the captured image of the specific camera. Alternatively, a plurality of evaluation values may be calculated under different conditions, and it may be switched whether to execute the process of S604 or the process of S614 based on the evaluation values.

<Third Embodiment>
In the first embodiment and the second embodiment, the case where one virtual plane is set in the three-dimensional space has been described. However, the number of virtual planes is not limited to one. In the third embodiment, a case where a plurality of virtual planes (two virtual planes) intersecting at one line of intersection is set will be described.

FIG. 8 is a diagram showing an example of a configuration in which the virtual viewpoint image generation device 100 is applied to curling of winter sports. FIG. 8A is a diagram showing a state in which the imaging system 110 (multi-view camera) is arranged in the upper part of the curling competition area.

An imaging system 110 composed of five cameras 101a to 101e is arranged above the seat 800, which is a curling competition area, as a plurality of cameras for capturing a multi-viewpoint image. Four of the five cameras 101a-101d are arranged so as to surround the periphery of the house 803 in order to generate a multi-viewpoint image in the vicinity of the house 803. Further, one camera 101e is arranged vertically above the center of the house 803. That is, the camera 101e is used as a specific camera whose line-of-sight direction is included in the line of intersection of the virtual plane 804 and the virtual plane 805.

FIG. 8B is an image when the virtual viewpoint is designated in the vertically upper part of the center of the house 803. The image from this viewpoint position helps the viewer to grasp the competition situation and understand the victory or defeat. In the third embodiment, two virtual planes are set. That is, the plane including the center line 801 and orthogonal to the plane of the sheet 800 is referred to as the first virtual plane 804, and the plane including the tee line 802 and orthogonal to the plane of the sheet 800 is referred to as the second virtual plane 805.

The virtual plane generation unit 112 generates a virtual plane 804 from two designated positions on the center line 801 and a virtual plane 805 from two designated positions on the tee line 802 as a virtual model, and acquires the plane equations thereof. The position designation on the center line 801 and the tee line 802 is performed by the user via the operation UI 109 connected to the virtual plane generation unit 112, as in the first embodiment. The virtual plane generation unit 112 provides the image generation unit 108 with the plane equations of the virtual planes 804 and 805.

The line-of-sight determination unit 114 calculates the evaluation value p regarding the distance between the virtual viewpoint and the virtual plane and the angle formed by the direction of the virtual viewpoint and the normal of the virtual plane for each of the virtual plane 804 and the virtual plane 805. An evaluation value p ₈₀₄ and an evaluation value p ₈₀₅ are obtained. Then, these evaluation values are integrated to obtain a new evaluation value. For example, the linear sum p = γp ₈₀₄ + ξp ₈₀₅ of each evaluation value is set as a new evaluation value. In the third embodiment, when the virtual viewpoint position and the line-of-sight direction are on the line of intersection 806 of the virtual plane 804 and the virtual plane 805, it is most effective for helping the viewer to understand the competition situation and the victory or defeat. At this time, since the virtual viewpoint is on the planes of the virtual plane 804 and the virtual plane 805, the evaluation value p is 0.

The evaluation value p = γp ₈₀₄ + ξp ₈₀₅ is sent from the line-of-sight determination unit 114 to the image generation unit 108. Hereinafter, the image generation unit 108 can switch the generation process of the virtual viewpoint image based on the evaluation value p, as in the first embodiment or the second embodiment. The relationship between the two virtual planes was evaluated as the evaluation value p, but the evaluation value is not limited to this. For example, the relationship between the line of intersection of two virtual planes and the virtual viewpoint (for example, the inner product of the direction vector of the line of intersection and the direction vector of the branch line, the distance between the line of intersection and the virtual viewpoint) may be evaluated.

As described above, although the virtual plane is used as the virtual model in each of the above embodiments, a virtual straight line may be used. For example, by designating the center of the house, a virtual straight line perpendicular to the seat surface and passing through the center of the house is set. In this case, the virtual straight line coincides with the line of intersection. The evaluation value can be calculated based on, for example, the distance between the virtual straight line and the virtual viewpoint, and the difference between the direction of the virtual straight line and the line-of-sight direction of the virtual viewpoint.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100: Virtual viewpoint image generator, 110: Imaging system, 101a-101z camera, 102: Communication line, 103: Image processing unit, 104: Image input unit, 105: Separation unit, 106: Foreground texture generation unit, 107: Background Texture generation unit, 108: Image generation unit, 109: Operation UI, 111: Display, 112: Virtual plane generation unit, 114: Line-of-sight determination unit

Claims (13)

An image generator that generates virtual viewpoint images from multi-viewpoint images taken by multiple cameras.
A setting means for setting a virtual model representing a virtual straight line or a virtual plane including the position of a specific camera among the plurality of cameras and a background model representing a three-dimensional shape of the background in a three-dimensional space.
A generation means for generating a virtual viewpoint image from a specified virtual viewpoint by using the multi-viewpoint image and the background model is provided.
The generation means uses the image captured by the specific camera in preference to the image captured by another camera to generate the virtual viewpoint image as the distance between the designated virtual viewpoint and the virtual model becomes closer. An image generator characterized by this. An image generator that generates virtual viewpoint images from multi-viewpoint images taken by multiple cameras.
A virtual model representing a virtual straight line or a virtual plane including the position of a specific camera among the plurality of cameras and a background model representing a three-dimensional shape of the background used to generate the virtual viewpoint image are set in a three-dimensional space. Setting means to be
A drawing means for drawing the virtual model observed from a virtual viewpoint matching the specific camera on an image taken by the specific camera.
An adjustment means for adjusting the background model so that the captured image and the drawn virtual model match.
An image generation device comprising: a multi-viewpoint image taken by the plurality of cameras and a generation means for generating a virtual viewpoint image using the background model adjusted by the adjustment means. The image generation device according to claim 2, wherein the generation means uses the image captured by the specific camera in order to generate the virtual viewpoint image in preference to the image captured by another camera. Further provided with an evaluation means for evaluating the closeness of the position and direction between the virtual viewpoint designated for generating the virtual viewpoint image and the virtual model.
The generation means generates a virtual viewpoint image by using the image captured by the specific camera in preference to the image captured by another camera when the proximity evaluated by the evaluation means is equal to or higher than a predetermined level. The image generator according to any one of claims 1 to 3, wherein the image generator is characterized by the above. An evaluation means for evaluating the closeness of the position and direction between the virtual viewpoint designated for generating the virtual viewpoint image and the virtual model, and
A switching means for switching the image used for display from the virtual viewpoint image generated by the generation means to the image captured by the specific camera when the proximity evaluated by the evaluation means is equal to or higher than a predetermined level. The image generator according to any one of claims 1 to 3, further comprising. The image according to claim 5, wherein the switching means adds a display indicating that the image used for display is an image actually captured by the camera when the image used for display is switched to the image captured by the specific camera. Generator. The method according to any one of claims 1 to 3, wherein the virtual model includes a plurality of virtual planes intersecting at one line of intersection, and the line-of-sight direction of the specific camera is included in the one line of intersection. Image generator. The virtual model includes a plurality of virtual planes intersecting at one line of intersection, and the line-of-sight direction of the specific camera is included in the one line of intersection.
The evaluation means integrates the evaluation results of the proximity to the virtual viewpoint designated for generating the virtual viewpoint image for each of the plurality of virtual planes, and the virtual model and the designated virtual viewpoint are combined. The image generator according to any one of claims 4 to 6, wherein the closeness is evaluated. The setting means is
A multi-viewpoint image from the plurality of cameras and a virtual viewpoint image generated based on the background model are displayed on the screen.
Any of claims 1 to 8 characterized in that the virtual model is set by determining a virtual plane or a virtual straight line based on a position designated by the user with respect to the displayed virtual viewpoint image on the screen. The image generator according to item 1. The virtual model is a virtual plane
The evaluation means according to any one of claims 4, 5, 6 and 8, characterized in that the position and direction of the virtual plane and the position and direction of the designated virtual viewpoint are evaluated. The image generator described. It is a control method of an image generator that generates a virtual viewpoint image from a multi-view image taken by a plurality of cameras.
A setting step of setting a virtual model representing a virtual straight line or a virtual plane including the position of a specific camera among the plurality of cameras and a background model representing a three-dimensional shape of the background in a three-dimensional space.
A generation step of generating a virtual viewpoint image from a designated virtual viewpoint by using the multi-viewpoint image and the background model is provided.
In the generation step, the closer the distance between the designated virtual viewpoint and the virtual model is, the more the image captured by the specific camera is used for generating the virtual viewpoint image in preference to the image captured by the other camera. A control method for an image generator, characterized in that. It is a control method of an image generator that generates a virtual viewpoint image from a multi-view image taken by a plurality of cameras.
A virtual model representing a virtual straight line or a virtual plane including the position of a specific camera among the plurality of cameras and a background model representing a three-dimensional shape of the background used to generate the virtual viewpoint image are set in a three-dimensional space. Setting process to be done and
A drawing step of drawing the virtual model observed from a virtual viewpoint matching the specific camera on an image taken by the specific camera.
An adjustment step of adjusting the background model so that the captured image and the drawn virtual model match.
A control method of an image generation device, comprising: a multi-viewpoint image taken by the plurality of cameras and a generation step of generating a virtual viewpoint image using the background model adjusted by the adjustment step. A program for causing a computer to execute each step of the control method according to any one of claims 11 or 12.

Images