JP7506022B2 - Drawing device and program - Google Patents

Description

The present invention relates to a rendering device and program for rendering free viewpoint video.

Free viewpoint video systems that generate video from any viewpoint from video captured by multiple cameras are known. Examples of conventional technology for realizing free viewpoint video are disclosed in Patent Documents 1 and 2, which disclose the following method. In Patent Document 1, video captured by a physical camera is delivered to a terminal, and the free viewpoint video is generated on the terminal, thereby reducing the load on the server. In Patent Document 2, the load of generating the free viewpoint video is reduced by changing the generation frequency between the dynamic area of interest and the rest.

JP 2019-121867 A JP 2018-067106 A

However, the conventional techniques described above have issues with efficiently providing free viewpoint video.

Patent Document 1 reduces the load on the server, but there is a problem in that the load on the terminal increases. In particular, the load is too high for terminals with relatively poor processing performance, such as smartphones and smart glasses, to process. Patent Document 2 has the problem that it does not mention reducing the processing load for generating texture on the surface of the subject. In free viewpoint video, texture is generated from images from a nearby physical camera according to the virtual viewpoint, but the higher the resolution of the image, the larger the processing required to read the image and expand it into memory. In particular, when there are many viewers, the load increases in proportion to the number of viewpoints, and there is a problem in that free viewpoint video cannot be generated in real time.

In view of the above-mentioned problems, the present invention aims to provide an efficient drawing device and program.

To achieve the above object, the present invention is a rendering device comprising: a construction unit that constructs a three-dimensional model from multi-viewpoint images; an integration unit that accepts virtual viewpoints designated by each of a plurality of users and integrates the plurality of virtual viewpoints to obtain a representative viewpoint; a rendering unit that obtains a rendering result at the representative viewpoint by rendering the texture of the multi-viewpoint images on the three-dimensional model from a virtual camera position set at the representative viewpoint; and a conversion unit that converts the rendering result at the representative viewpoint into a rendering result at a virtual viewpoint designated by each of the plurality of users. The present invention is also characterized as a program that causes a computer to function as the rendering device.

According to the present invention, rendering is performed only for a representative viewpoint that combines multiple virtual viewpoints, and the rendering results for each virtual viewpoint are obtained by converting this, enabling efficient rendering.

FIG. 1 is a configuration diagram of a drawing system according to an embodiment. FIG. 1 is a diagram showing an example of camera arrangement in an imaging facility. 1 is a diagram showing a schematic example of a free viewpoint video display at a virtual viewpoint desired by a user, provided in a rendering system; FIG. 1 is a functional block diagram of a drawing system according to an embodiment. 1 is a flowchart of an operation of a drawing system according to an embodiment. FIG. 13 is a diagram illustrating a schematic example for explaining a process of an integration unit according to an embodiment. FIG. 13 is a diagram showing a schematic example of a conversion process in a conversion unit. FIG. 13 is a diagram showing an example of distinguishing the gaze points of a target. FIG. 1 is a diagram illustrating a hardware configuration of a typical computer.

Figure 1 is a configuration diagram of a drawing system according to one embodiment. The drawing system 100 comprises M (M≧2) terminals 10-1, 10-2, ..., 10-M each used by a user who watches a free viewpoint video, a server 20 as a drawing device, and an imaging device 30 consisting of N (N≧2) cameras C1, C2, ..., CN that capture images of common content (e.g., sports) for which the free viewpoint video is to be generated. These components of the drawing system 100 can communicate with each other via a network NW.

Figure 2 shows an example of camera arrangement in imaging equipment 30, where the number of cameras is N=8. Each camera C1-C8 is arranged to circumferentially surround a target OB (e.g., a player in a sports game) that is rendered by generating a 3D model using free viewpoint video, and images of this target OB are captured at each camera position, and captured images are obtained at each time t (=1, 2, ...) in real time, thereby acquiring images at each camera position. Each camera C1-C8 and the target OB are positioned at approximately the same height above the ground PL on the field in the real world where the imaging is being performed.

Note that the arrangement of the imaging equipment 30 in FIG. 2 is merely a schematic example, and the imaging equipment 30 may be configured with any number of N cameras arranged in the real world to capture the target OB, whose three-dimensional model is to be generated in the free viewpoint video, at different camera positions and camera attitudes (directions). For example, instead of capturing the target OB by surrounding it in a circular shape on a plane as in FIG. 2, the target OB may be captured by surrounding it on a hemispherical surface configured on the plane. While capturing the target OB by surrounding it, all or some of the cameras in the imaging equipment 30 may move manually or automatically within the field in the real world according to time t, or all or some of the cameras may be fixedly installed on the field and the position and attitude may not change. The target OB may also be composed of one or more objects in the real world (e.g., multiple players in a sports game).

In the drawing system 100 of the configuration shown in FIG. 1, the communication function transmits and receives from each of a number M of terminals 10 (any one of terminals 10-1, 10-2, ..., 10-M is referred to as terminal 10, and the same applies below) coordinate information of the virtual viewpoint that each user wishes to view as information transmitted by the terminal 10 and received by the server 20, and conversely, the server 20 transmits and receives the drawing result from that virtual viewpoint as information transmitted by the terminal 10 and received by the terminal 10.

The overall operation of the drawing system 100 is to capture images of the target OB at each time t that is time-synchronized by multiple N cameras C1, C2, ..., CN of the imaging equipment 30 according to a predetermined processing rate such as 30 fps (frames per second), perform drawing processing according to a specified virtual viewpoint in the server 20, and display the drawing results on the terminal 10, thereby providing a free viewpoint video display according to the specified virtual viewpoint to the user using the terminal 10. Details of this operation will be described later with reference to Figures 4 and 5.

Figure 3 is a diagram showing a schematic example of a free viewpoint video display at a user-desired virtual viewpoint provided by the drawing system 100, and shows three examples of drawing results G1 to G3 in which a three-dimensional model MD of the object OB (schematically showing a person such as an athlete) in Figure 2 is generated at a common time t and drawn at a virtual viewpoint desired by the user. The first drawing result G1 is drawn by a first user specifying a virtual viewpoint from which the three-dimensional model MD is viewed from a far-distance position facing the right hand side of the front, the second drawing result G2 is drawn by a second user specifying a virtual viewpoint from which the three-dimensional model MD is viewed from a near-distance position facing the front, and the third drawing result G3 is drawn by a third user specifying a virtual viewpoint from which the three-dimensional model MD is viewed from a medium-distance position facing the left hand side of the front.

As shown in the example of Figure 3, in the drawing system 100 of this embodiment, a three-dimensional model MD is generated for a common real-world object OB at a common time t, and then the drawing result of the three-dimensional model MD can be provided at the virtual viewpoint desired by each user.

Figure 4 is a functional block diagram of a drawing system 100 according to an embodiment. As shown in the figure, in the drawing system 100, the terminal 10 includes a designation unit 11 and a display unit 12, and the server 20 (drawing device 20) includes a construction unit 21, an integration unit 22, a generation unit 23, and a conversion unit 24. As described above, the terminal 10 in Figure 4 is any one of the M terminals shown in Figure 1, and shows a common functional block configuration in the M terminals 10-1, 10-2, ..., 10-M. Figure 5 is a flowchart of the operation of the drawing system 100 according to an embodiment. Below, the details of the processing contents of each unit in Figure 4 will be described while explaining each step in Figure 5.

The flow in FIG. 5 comprises steps S1 to S6, and the overall outline thereof is as follows. That is, steps S1 to S5 as processing step group SG(t) represent each process for enabling viewing of free viewpoint video from a virtual viewpoint on each user's terminal 10 by repeatedly executing the rendering system 100 at each real-time time t=1, 2, ... as described above, and step S6 represents a time update (and management of the processing timing in the rendering system 100) to indicate that processing is thus performed at each real-time time t=1, 2, ....

In step S1, the following two processes are performed at the current time t, and then the process proceeds to step S2.

(1) Imaging is performed using the imaging equipment 30 to obtain a multi-view image MV(t) of the target OB.

Each of the N cameras C1, C2, ..., CN of the imaging equipment 30 captures an image of the target OB, thereby obtaining a multi-view image MV(t) = {Pic1(t), Pic2(t), ..., PicN(t)} from N viewpoints (where PicK(t) (K = 1, 2, ..., N) is an image captured by camera CK (K = 1, 2, ..., N) at time t). This multi-view image MV(t) is sent to the server 20, which outputs it to the construction unit 21 and the generation unit 23.

(2) For each of the M terminals 10, specify the coordinates pi(t) of the virtual viewpoint and the gaze direction di(t).

The designation unit 11 of each of the M terminals 10 accepts designation of the virtual viewpoint coordinates pi(t) and gaze direction di(t) (i is an identifier of the terminal 10 and the user who uses this terminal 10, where i = 1, 2, ..., M) through user input, and outputs these virtual viewpoint coordinates pi(t) and gaze direction di(t) (i = 1, 2, ..., M) to the integration unit 22 by transmitting them to the server 20.

The virtual viewpoint coordinates pi(t) (i=1,2,...,M) of each of the M users are specified as three-dimensional coordinate positions pi(t)=(xi(t),yi(t),zi(t)) in the three-dimensional virtual space VSP in which the three-dimensional model of the target OB is rendered, and the specification of the coordinates pi(t) can be accepted by accepting user input in the specification unit 11 using any interface for specifying a spatial position. For example, the user may operate an input interface such as a keyboard, a touch panel, or a mouse to give an instruction to continuously move the virtual viewpoint in the three-dimensional virtual space VSP, and may specify a movement amount Δpi(t)=(Δxi, Δyi, Δzi) from the position pi(t-1) at the previous time t-1 at the time t, thereby accepting the specification of the coordinates pi(t) at the current time t as follows:
pi(t)=pi(t-1)+Δpi(t)

In addition, as a result of moving the virtual viewpoint coordinate pi(t) in the three-dimensional virtual space VSP by the operation by the user input as described above, it may become the position of the virtual viewpoint that the user does not want, so in such a case, an instruction to reset the virtual viewpoint coordinate pi(t) to a predetermined position Ref(t) (for example, a position where the three-dimensional model of the target OB is viewed from the front at the time t) may be accepted as follows. Note that, by applying this reset at the initial time t=0, the virtual viewpoint coordinate pi(0) may be set to the initial position Ref(0).
pi(t)=Ref(t)

The gaze direction of the virtual viewpoint di(t)=(dxi(t),dyi(t),dzi(t)) (i=1,2,…,M) can also be accepted as an instruction for continuous movement or a reset instruction, similar to the specification of the virtual viewpoint coordinate pi(t) (i=1,2,…,M) above.

Alternatively, the gaze direction di(t) of the virtual viewpoint may be automatically specified according to the manually specified virtual viewpoint coordinates pi(t) instead of being manually specified by the user as described above. For example, the user may specify in advance the target OB (one of multiple) that the user wishes to view, and a predetermined position (e.g., the center of gravity position) in the three-dimensional model MD(t) of the target OB that the user wishes to view may be set as pos(t), and the gaze direction di(t) may be set to be parallel to a direction vector that starts at the virtual viewpoint coordinates pi(t) and ends at this predetermined position pos(t). According to this setting, the virtual viewpoint is set so that it always looks toward the three-dimensional model MD(t) from the virtual viewpoint coordinates pi(t) specified by the user.

In step S2, the following two processes are performed at the current time t, and then the process proceeds to step S3.

(1) The construction unit 21 generates a three-dimensional model MD(t) of the object OB at the current time t from the multi-viewpoint image MV(t).

The construction unit 21 performs image processing on the multi-view images MV(t)={Pic1(t), Pic2(t), ..., PicN(t)} from N viewpoints obtained from the imaging equipment 30 to construct a three-dimensional model MD(t) that represents the three-dimensional shape of the captured object OB at the current time t, and outputs the model to the generation unit 23 and the conversion unit 24. Any existing method may be used to construct the three-dimensional model MD(t) from the multi-view images MV(t), and for example, the three-dimensional model MD(t) may be constructed by the visual hull intersection method. As is known, in the visual hull intersection method, for example, a background subtraction method is applied to each image of the multi-view images to extract a foreground silhouette area (area in the image plane) occupied by the object OB, and a volumetric area occupied by the object OB in three-dimensional space can be obtained as an overlapping area at each viewpoint of a cone (visual hull) obtained by performing a three-dimensional back projection from the camera position of each viewpoint onto this foreground silhouette area. The three-dimensional backprojection is performed, for example, in voxel space, and the volumetric area that the target OB occupies in three-dimensional space is obtained as a voxel area. Then, a three-dimensional model MD(t) of the target OB as a polygon model can be constructed using any existing method, such as the marching cubes method.

(2) The integration unit 22 integrates the M virtual viewpoints (coordinates pi(t) and gaze directions di(t)) to obtain a representative viewpoint.

The integration unit 22 applies clustering to the M virtual viewpoint coordinates pi(t) obtained from each of the terminals 10 to obtain a representative viewpoint as a representative position in each of the fewer than M clusters, and outputs this representative viewpoint to the generation unit 23. Here, the clustering is performed while taking into consideration the role of the virtual viewpoint (virtual camera position) that serves as the reference position for drawing, so that a representative viewpoint can be obtained that is integrated to become the optimal virtual viewpoint when generating the texture of the three-dimensional model MD(t) in the downstream generation unit 23, and specifically, this can be done as follows.

Figure 6 is a diagram showing a schematic example for explaining the processing of the integration unit 22 according to one embodiment. In Figure 6, the configuration of the imaging equipment 30 is the same as that of the eight cameras C1 to C8 shown in Figure 2, which capture images of the target OB by surrounding it in a circular shape. In Figure 2, the target OB and the eight cameras C1 to C8 are arranged in the real world, but the three-dimensional world coordinate system of this real world can be used as is to define a virtual space VSP as shown in Figure 6 (a virtual space VSP for depicting the target OB as its three-dimensional model MD(t)).

In addition, in the above-mentioned designation unit 11, by providing the virtual viewpoint (coordinates pi(t) and gaze direction di(t)) in the virtual space VSP defined in a coordinate system common to the three-dimensional coordinate system of the real world, the arrangement of the virtual viewpoint, i.e., the virtual camera, can be given in correspondence with the arrangement of the physical cameras C1 to C8. (For example, it is possible to designate one virtual camera so that it is in the same arrangement as the physical camera C1.)

The processing of the integration unit 22 will be described below with reference to the example in FIG. 6. Note that in this description, since time t refers only to the current time, notation dependent on time t will be omitted, and the coordinates pi(t) of the virtual viewpoint will be written as pi, and the gaze direction di(t) will be written as di.

First, in order to perform the clustering described below in units of directions for virtual viewpoints having the same gaze point when the gaze point of the target OB is set as the origin, multiple arbitrary viewpoints are normalized. That is, for M viewpoint coordinates pi (i = 1, 2, ..., M), each normalized coordinate Pi is given by the following equation.
Pi=pi/|pi|

The center of gravity may be set as the gaze point of the target OB, for example. The normalized coordinate Pi is calculated as a unit direction vector from the target OB toward the viewpoint coordinate pi by using the viewpoint coordinate pi with the center of gravity set as the origin.

In the example of Figure 6, M=6, and six virtual viewpoint coordinates p1-p6 and their normalized coordinates P1-P6, of which the positions P1-P3 are shown as white circles (○) on the surface of a unit sphere SP centered on the gaze point of the target OB. (Note that normalized coordinates P4-P6 are located behind the normalized coordinates P1-P3 shown in the figure, and are therefore not shown in order to avoid cluttering the drawing.)

Note that the "gazing point of the target OB" used by the integration unit 22 is determined according to the gaze direction di as seen from the viewpoint coordinate pi of each user, but is a broader concept than the line of sight direction (the optical axis direction of the virtual camera) determined by pi and di. In other words, the "gazing point of the target OB" is what distinguishes which of one or more objects constituting the 3D content rendered in the rendering system 100 are to be viewed and which are not (i.e., what specifies which of the multiple 3D models needs to be rendered for each user), and is determined according to the 3D content to be rendered by each user specifying the viewpoint coordinate pi and gaze direction di in the specification unit 11. (For example, even if the viewpoint coordinate pi of one user and the viewpoint coordinate pj of another user are the same (pi=pj) and the gaze directions di, dj are slightly different, if the same target OB is in the field of view, the "gazing point of the target OB" will be common to both users.) The example in FIG. 6 shows an example in which the rendered 3D content is composed of only a single target OB (for example, one sports player), and all M=6 users specify viewpoint coordinates pi and gaze directions di such that the single target OB is included in the viewing target, thereby determining the "gazing point of the target OB" as a single point common to all users. Below, we will explain the case where the "gazing point of the target OB" is a single point common to all users, as in the example in FIG. 6, and will discuss later the cases where there are multiple points or where they are not common.

Next, the normalized coordinates Pi (i=1,2,...,M) are clustered, and the normalized center of gravity Gj (j=1,2,...,Mc) on the unit sphere SP of each cluster CLj (Mc<M) which is less than the initial number M of virtual viewpoints is calculated using the following formula. (Num_j is the number of elements in cluster CLj, and the sum Σ _i is calculated as the sum of Num_j elements Pi∈CLj belonging to the cluster CLj.) For clustering, existing methods such as the k-means method, the x-means method which automatically determines the number of clusters, and the g-means method can be used.
gj=Σ _i Pi/Num_j
Gj=gj/|gj|

In another embodiment, since degradation of image quality becomes less noticeable in proportion to the distance from the gaze point to the virtual viewpoint pi (the original coordinate pi, not the normalized coordinate Pi), the center of gravity Gj (the center of gravity of the normalized coordinate Pi) normalized on the unit sphere SP is calculated using the inverse of the distance as weight wi, as shown in the following formula.
gj = (Σ _i wi * Pi)/(Σ _i wi)
Gj=gj/|gj|

In other words, when drawing from a distant virtual viewpoint, the drawing size becomes smaller, making the degradation of texture quality less noticeable, and the opposite is true from a closer viewpoint, so the weight wi, which is the inverse of the distance, can be used to bring it closer to the closer viewpoint, so that the texture from the closer viewpoint is used preferentially. (This use is performed in the generation unit 23 at the subsequent stage.) Conversely, if motion blur is noticeable up close, the weight wi can be set to the distance rather than the inverse of the distance, and it can be used to bring it closer to a distant viewpoint, and because there is a lack of texture when it is too far away, the weight can be kept at a constant value without being increased any further at distances farther than a certain value.

In the example of Figure 6, two clusters CL1 = {P1, P2, P3} and CL2 = {P4, P5, P6} are formed as a result of clustering six normalized coordinates P1 to P6, and the center of gravity G1 is calculated from the first cluster CL1 = {P1, P2, P3}. (The center of gravity G2 is calculated in the same way from the second cluster CL2 = {P4, P5, P6}, but as mentioned above, it is not shown in the figure.)

In addition, since the normalized coordinates Pi correspond to the coordinates pi of the original virtual viewpoint, the clustering result at the normalized coordinates Pi is the clustering result of the original virtual viewpoint coordinates pi as it is (in other words, the normalized coordinates Pi are used as an evaluation index for clustering the virtual viewpoint coordinates pi). In the example of FIG. 6, it is as follows.
CL1={P1,P2,P3}={p1,p2,p3}
CL2={P4,P5,P6}={p4,p5,p6}

Next, the distance Lj (j=1, 2, ..., Mc) closest to the fixation point among the coordinates pi of the clustered virtual viewpoints is calculated for each cluster CLj using the corresponding normalized coordinates Pi according to the following formula: (As described above, since the fixation point is set to the origin, the distance between the coordinate pi and the fixation point is the absolute value |pi|.) By setting the distance as shortest, it is possible to obtain the effect of maintaining high resolution of the pattern of the 3D model (rendered by the generation unit 23 at the subsequent stage).
Lj = min |pi|
pi ∈ Lj

In the example of Figure 6, the distance of p3, which is the shortest in cluster CL1={p1, p2, p3}, is calculated as L1.

Alternatively, instead of the shortest distance, the average or median of the distance to the gaze point for each cluster can be used as the distance Lj. In this case, the effect of suppressing differences due to differences in the depth of the viewpoint can be obtained. As described above, the representative distance Lj can be determined as the shortest distance, average value, median value, etc.

Finally, the representative distance Lj of each cluster is multiplied by the center of gravity coordinate Gj of the corresponding cluster as shown in the following formula, and the coordinate Vj of the representative viewpoint generated in the generation unit is output. Since the gaze point is defined as the origin as described above, the following multiplication (vector Gj multiplied by scalar Lj) makes it possible to calculate the coordinate Vj of the representative viewpoint as a position moved by the distance Lj in the direction from the gaze point toward the center of gravity Gj (|Gj|=1) normalized on the unit sphere SP.
Vj = Lj*Gj

In the example of Figure 6, a virtual viewpoint V1 is calculated as the representative viewpoint of cluster CL1 = {P1, P2, P3} = {p1, p2, p3} at a position on the extension line connecting the center of gravity G1 and the central point (point of gaze) at the same distance as L1 (the distance of virtual viewpoint p3 as the shortest distance, as mentioned above).

Step S2 in FIG. 5 has been described above. In step S3, the generation unit 23 sets the representative viewpoint Vj of each cluster CLj obtained by the integration unit 22 as the position of the virtual camera, and sets the gaze point in the same manner as in the integration unit 22 to set the direction of the virtual camera, and draws the three-dimensional model MD(t) of the target OB constructed by the construction unit 21 by pasting (pasting to surface elements such as polygons of the three-dimensional model MD(t)) the texture of the image of the physical camera determined to be in the vicinity of the representative viewpoint Vj among the textures of the multi-view image MV(t) obtained by the imaging equipment 30, and outputs the drawing result Gj for each representative viewpoint Vj obtained to the conversion unit 24, and then proceeds to step S4. The drawing result Gj is, in other words, a three-dimensional model MD(t) drawn on the image plane with the representative viewpoint Vj as the virtual camera position.

Any existing method may be used for the texture drawing in the generation unit 23. For example, when there are multiple physical cameras that are determined to be nearby, a texture obtained by blending the textures of the multiple physical cameras (weighted sum based on distance, etc.) according to the distance between the position of the physical camera and the representative viewpoint Vj, etc., may be applied, or only the texture of the physical camera closest to the representative viewpoint Vj may be applied for each surface element such as a polygon. For example, an existing method such as Patent Document 2 may be used. Note that when the number N of physical cameras is large, when selecting a physical camera near the representative viewpoint Vj, the coordinates of the physical cameras may be clustered to calculate representative points, and the distances between the representative points and the virtual viewpoints may be compared hierarchically. The effect of reducing the processing load is obtained by further calculating the distance of each physical camera in the cluster including the representative point with the shortest distance.

In addition, if there are multiple physical cameras with the same distance from the representative viewpoint Vj at the current time t (determined to be the same distance by a threshold), it is possible to use a nearby physical camera at the representative viewpoint Vj at past time t-1. (In this case, if the representative viewpoint Vj is in a different position at the current time t and past time t-1, it is sufficient to use the representative viewpoint from past time t-1 that is closest to the representative viewpoint Vj at the current time t.)

In the example of Figure 6, if the existing method uses one of the nearest physical cameras to generate a texture image, and if a method that does not apply the embodiments of the present invention is applied as a comparison to the embodiments of the present invention, the following processing will be performed. That is, in this comparison, physical camera C1 is assigned to virtual viewpoint p1, physical camera C2 to virtual viewpoint p2, and physical camera 3 to virtual viewpoint p3, and each of the three images will be read and expanded, which increases the amount of processing.

On the other hand, in the embodiment of the present invention, for the three virtual viewpoints p1, p2, and p3, the generation unit 23 uses only the physical camera C2 closest to the virtual viewpoint V1 as the representative viewpoint of these three, which has the effect of reducing the amount of images to be read by one-third compared to the comparative example. In other words, when multiple virtual viewpoints have similar normalized coordinates, the processing load can be reduced by integrating the optimal virtual viewpoint as a result of clustering in the integration unit 22 and generating a texture, so that the generation unit 23 does not need to refer to images captured by all cameras (or multiple cameras) of the imaging equipment 30, and the processing time required for image reading during texture generation can be shortened by limiting the images to those captured by cameras near the integrated representative viewpoint Vj.

In addition, by reflecting a distance close to the specified virtual viewpoint coordinates as the aforementioned representative distance Lj, the effect of improving quality can be obtained. Furthermore, by successively changing the number of clusters, the effect of being able to balance between reducing the processing load and improving quality can be obtained. Alternatively, the effect of improving quality can be obtained by proportionally adjusting the number of physical cameras used for proximity determination to acquire textures according to the variation in virtual viewpoints within a cluster.

In other words, the variance that quantifies the variation in the coordinates pi of the virtual viewpoints belonging to cluster CLj (or the variance of normalized coordinates Pj) is set as var_j, and when a three-dimensional model MD(t) is drawn for cluster CLj with the representative viewpoint Vj as the viewpoint of the virtual camera, the number of nearby physical cameras referenced for texture acquisition may be set to a number proportional to the variance var_j (or a larger number the larger the variance var_j), and an optimized number may be set for each cluster CLj.

In step S4, by referring to the surface element information of the three-dimensional model MD(t), the conversion unit 24 converts (transforms) the rendering results Gj obtained for each cluster CLj by the generation unit 23 for each surface element, thereby obtaining rendering results at the virtual viewpoints pi (i = 1, 2, ..., M) specified by the user on each of the M terminals 10, and transmits the rendering results to the display units 12 of the corresponding terminals 10 before proceeding to step S5.

The conversion unit 24 applies a conversion process TR such as planar projective transformation (conversion process TR corresponding to changing the virtual camera position from Vj to pi) to each surface element constituting the drawing result Gj of the cluster to which the virtual viewpoint pi belongs (cluster CLj) by assuming that the representative coordinate Vj, which is the virtual camera position used to draw the drawing result Gj, has been changed to the position of the virtual viewpoint pi, thereby obtaining the drawing result at the virtual viewpoint pi.

Figure 7 is a diagram showing a schematic example of the above conversion process in the conversion unit 24. The rendering result Gj at the representative viewpoint Vj is composed of multiple surface elements poly_k (k = 1, 2, ...) such as polygons by referring to the three-dimensional model MD (t). The rendering result TGi = TR (Gj) at the virtual viewpoint pi can be obtained as a set of surface elements TR (poly_k) to which the conversion TR is applied. Since there are generally many surface elements poly_k (and surface elements TR (poly_k) converted from it), only one of them is shown in Figure 7, and these surface elements are enlarged and shown separately at the bottom as a schematic diagram. The two-dimensional surface element poly_k rendered at the representative viewpoint Vj is a perspective projection of the surface element 3Dpoly_k (not shown) in the three-dimensional space of the three-dimensional model MD (t) at the representative viewpoint Vj, so the position and shape of the surface element TR (poly_k) can be obtained by perspective projecting the three-dimensional surface element 3Dpoly_k onto the virtual viewpoint pi. This correspondence between "poly_k ⇒ 3D_poly_k ⇒ TR(poly_k)" is the transformation TR, and this transformation TR is recorded as the correspondence for each polygon. When transforming a texture and assigning it to TR(poly_k), you can simply transform the texture of poly_k using a planar projection transformation or similar and assign it.

When rendering the three-dimensional model MD(t) at the representative viewpoint Vj, not only the front side where no occlusion occurs, but also a part of the back side where occlusion occurs that is determined to be a position near the front side (determined by a threshold value determination based on the distance between polygons, etc.) may be rendered and stored as the rendering result for each polygon at the representative viewpoint Vj. The distinction between the front side and the back side based on the presence or absence of occlusion may be determined for each polygon using any existing method in three-dimensional computer graphics (for example, by projecting to the virtual camera position and determining whether it is at the forefront). For a polygon 3Dpoly_k (a polygon of the three-dimensional model MD(t)) that is occluded when rendered from the representative viewpoint Vj, when it appears on the front side without occlusion from the virtual viewpoint pi, the rendering result of the polygon TR(poly_k) can be obtained by converting the result of rendering including the occlusion area for the representative viewpoint Vj.

As a modified example of the processing of the generation unit 23 and the conversion unit 24 in steps S3 and S4, a billboard method (a method in which the three-dimensional model MD(t) is simplified and generated as a single rectangle (billboard)) may be used. In this case, the billboard that faces the representative viewpoint Vj may not face the virtual viewpoint pi, or may face it.

In step S5, the rendering result obtained in step S4 is displayed to the user on the display unit 12 (display as hardware) of the terminal 10, thereby enabling the user to view the free viewpoint video from the virtual viewpoint desired by the user at that time t, and the process proceeds to step S6. As described above, step S6 is a step that represents a time update, and by updating the current time t to the next current time t+1 and returning to step S1, the above steps S1 to S5 (step group GS(t)) are repeated in the same manner as step group SG(t+1) for the next current time t+1.

As described above, according to one embodiment of the present invention, by integrating multiple virtual viewpoints for the same direction to generate an optimal virtual viewpoint, the number of images required to load and expand in order to generate the texture for a 3D model can be reduced, and the generated 3D model and texture can be used to draw from the original virtual viewpoint to obtain drawing information.

In other words, according to one embodiment of the present invention, it is possible to reduce the load on the system by reading and extracting the minimum amount of images required. Here, by reducing the load, it is also possible to increase the number of terminals that can be processed simultaneously.

Below, we explain various supplementary, additional, and alternative examples.

(1) According to an embodiment of the present invention, as an application example, it becomes possible to view a target OB (e.g., a player in a sports game) in a remote location as a free viewpoint video with a sense of realism. This makes it possible to view content such as a sports game without necessarily having to actually travel to the remote location, or to use it as a display interface for remote communication to give advice to a target OB in a remote location (e.g., advice on improving sports skills). By saving the energy resources required for user movement, carbon dioxide emissions can be reduced, which makes it possible to contribute to Goal 13 of the United Nations-led Sustainable Development Goals (SDGs), which is to "take urgent action to combat climate change and its impacts."

(2) When the number of terminals 10 is large, it is possible to simplify the system configuration by sharing the integration unit 22 as a pre-stage process when a configuration in which Ns drawing systems 100 (Ns is the number of clusters resulting from the clustering obtained by the integration unit 22) are arranged in parallel can be used.

The parallel arrangement of Ns units (Ns≧2) is realized according to the number of clusters resulting from the clustering in the integration unit 22. For example, in the example of FIG. 6, Ns=2, and the first server 20-1 (equipped with the construction unit 21-1, generation unit 23-1, and conversion unit 24-1 similar to those in FIG. 4) may perform the rendering process for cluster CL1, and the second server 20-2 (equipped with the construction unit 21-2, generation unit 23-2, and conversion unit 24-2 similar to those in FIG. 4) may perform the rendering process for cluster CL2. In this way, the processing of the integration unit 22 (and the imaging equipment 30) is performed in common within the rendering system 100, and the subsequent processing is assigned to the servers for each cluster resulting from the clustering and performed in parallel, thereby limiting the number of images read from the imaging equipment 30 in each server.

(3) The terminal 10 may be configured to perform augmented reality display such as an optical see-through or video see-through head-mounted display, and the rendering result from the virtual viewpoint obtained from the conversion unit 24 may be superimposed on the scenery in the real world and displayed on the display unit 12. The virtual viewpoint specified by the specification unit 11 may be linked at each time t to the viewpoint in the real world of the user obtained by a sensor or the like provided in the head-mounted display.

(4) The conversion unit 4 generates the equivalent of the rendering result of the three-dimensional model MD(t) at the virtual viewpoint pi by converting the rendering result Gj at the representative viewpoint Vj corresponding to the virtual viewpoint pi in the relationship of "virtual viewpoint pi∈cluster CLj" obtained from the generation unit 23. At this time, the background at the virtual viewpoint pi may also be rendered to obtain a rendering result in which the three-dimensional model is superimposed on the background. A three-dimensional model (preferably a model with a low rendering load, such as a simple plane) is provided for the background in advance, and it is rendered at the virtual viewpoint pi.

(5) In the integration unit 22, for each "gazing point of the target OB", clustering was performed based on the direction of the virtual camera determined by the normalized coordinates of the position of the virtual camera set with the gazing point as the origin. This gazing point may be provided for each gazing scene that is distinguished according to which of one or more target OBs set in advance as a drawing target is within the field of view of the virtual camera and which is outside the field of view. FIG. 8 is a diagram showing an example of distinguishing the gazing points of the target OB, in the case where there are two targets, target OB={OB1, OB2}, that there can be 2 ² =4 different gazing points of the target OB. As in the example of FIG. 8, when the target OB is composed of n pieces, there are 2 ⁿ different distinctions of the gazing points.

In FIG. 8, for example, for virtual camera C11, only object OB1 exists within its angle of view range R11, so a predetermined point of object OB1 (such as center of gravity) is set as the point of gaze, for virtual camera C12, only object OB2 exists within its angle of view range R12, so a predetermined point of object OB2 is set as the point of gaze, for virtual camera C13, both objects OB1 and OB2 exist within its angle of view range R13, so a predetermined point of both objects OB1 and OB2 (such as center of gravity of both objects) is set as the point of gaze, and for virtual camera C14, no object OB = {OB1, OB2} exists within its angle of view range R14, so rendering processing for object OB is treated as unnecessary.

In the example of FIG. 8, the virtual viewpoints are grouped in advance into four types: <1> virtual camera C11, where only target OB1 is in the field of view; <2> virtual camera C12, where only target OB2 is in the field of view; <3> virtual camera C13, where both targets OB1 and OB2 are in the field of view; and <4> virtual camera C14, where target OB={OB1, OB2} is not in the field of view. Then, for each of the four groups, clustering is performed by the integration unit 22 as described with reference to FIG. 6, and the processing after the generation unit 23 is also performed. For virtual viewpoints grouped into virtual camera C14, as in the example of <4>, clustering processing and rendering processing for target OB={OB1, OB2} are not required, and when background rendering is performed, only background rendering may be performed.

As is known in the field of 3D computer graphics, the field of view range (R11 to R14 in the example of FIG. 8, etc.) of a virtual camera is determined by giving the position pi and line of sight di (corresponding to the external parameters of the camera) in the 3D virtual space and using information on the internal parameters of the camera that have been set in advance. When constructing the 3D model MD(t) in the construction unit 21, information on how many different objects are included in the modeled object OB may also be given in advance. The determination of whether one or more objects are within the field of view range of each virtual camera may be made by determining whether at least a part of the object is within the field of view range, or by setting at least one representative point on the object and determining whether at least one of the representative points is within the field of view range.

(6) FIG. 9 is a diagram showing an example of the hardware configuration of a general computer device 70. Each of the terminal 10, server 20, and imaging equipment 30 constituting the drawing system 100 can be realized as one or more computer devices 70 having such a configuration. When each of the terminal 10, server 20, and imaging equipment 30 is realized by two or more computer devices 70, information required for processing may be sent and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 78, and a bus BS for transmitting and receiving data between them.

Each functional unit of the drawing system 100 can be realized by a CPU 71 and/or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, the display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, the communication interface 75 also operates in conjunction with the GPU 72. The display unit 12 may be realized by the display 76, and each camera of the imaging equipment 30 may be realized as a camera 78.

100...drawing system, 30...imaging equipment, 20...server/drawing device, 10...terminal
11: Designation unit, 12: Display unit, 21: Construction unit, 22: Integration unit, 23: Generation unit, 24: Conversion unit

Claims (14)

A construction unit that constructs a three-dimensional model from the multi-viewpoint images;
an integration unit that receives designation of a virtual viewpoint from each of a plurality of users and integrates the plurality of virtual viewpoints to obtain a representative viewpoint;
a generation unit that obtains a rendering result at a representative viewpoint by rendering a texture of the multi-viewpoint image on the three-dimensional model from a virtual camera position that is set at the representative viewpoint;
a conversion unit that converts a rendering result from the representative viewpoint into a rendering result from a virtual viewpoint designated by each of the plurality of users. The rendering device according to claim 1, characterized in that the integration unit performs clustering to obtain the representative viewpoint that is composed of a number less than the number of the multiple virtual viewpoints. The rendering device according to claim 2, characterized in that, in the integration unit, when clustering the multiple virtual viewpoints, the direction from the gaze point to each virtual viewpoint is used as a criterion for the clustering. The drawing device according to claim 3, characterized in that the integration unit determines the direction by normalizing a vector from the gaze point toward each virtual viewpoint, and obtains a clustering result of the multiple virtual viewpoints by clustering the normalized vectors. The drawing device according to claim 4, characterized in that the integration unit obtains a representative vector of the normalized vector for each cluster, and sets an end point that is a predetermined distance away from the gaze point in the direction of the representative vector as the representative viewpoint for that cluster. The drawing device according to claim 5, characterized in that the integration unit determines the representative vector by calculating a weighted sum of the normalized vectors for each cluster, the weighted sum being determined by the magnitude of the vector before normalization. The rendering device according to claim 5 or 6, characterized in that the integration unit uses the minimum, median or average of the distances between the gaze point and each virtual viewpoint in the cluster as the specified distance. The rendering device according to any one of claims 3 to 7, characterized in that the integration unit sets the gaze point for each scene that is distinguished by which of one or more objects constituting the three-dimensional model is present and which is not present within a field of view range determined by the position of the virtual viewpoint and the line of sight direction received from each of the multiple users. The rendering device according to claim 8, characterized in that the integration unit groups the virtual viewpoints for each of the distinguished scenes, and performs the clustering for the virtual viewpoints belonging to each group. The rendering device according to any one of claims 3 to 9, characterized in that the integration unit sets the gaze point as a predetermined point of the three-dimensional model within a range of an angle of view determined by the positions of virtual viewpoints and line-of-sight directions received from each of a plurality of users. The rendering device according to any one of claims 1 to 10, characterized in that the generation unit performs rendering using textures of only a portion of the multiple viewpoint images constituting the multi-view image, and determines, as the portion, a viewpoint image captured by a physical camera whose position is determined to be close to the representative viewpoint, among multiple physical cameras capturing the multi-view image. The rendering device according to any one of claims 1 to 11, characterized in that the generation unit renders the three-dimensional model for the front side where no occlusion occurs in the three-dimensional model and the rear side near the front side where occlusion occurs, based on a virtual camera position set to the representative viewpoint. The integration unit integrates a plurality of virtual viewpoints by performing clustering to obtain a representative viewpoint,
The rendering device according to any one of claims 1 to 12, characterized in that the generation unit performs rendering at a representative viewpoint using textures of only a portion of the multiple viewpoint images that constitute the multi-viewpoint image, and determines the number of images of only a portion to be used in accordance with the variation of virtual viewpoints belonging to a cluster corresponding to the representative viewpoint. A program that causes a computer to function as a drawing device according to any one of claims 1 to 13.

Images