JP7842645B2 - Viewpoint changing device and its program - Google Patents

Description

This invention relates to a viewpoint transformation device and its program.

Stereoscopic video display devices are expected to diversify into various forms, not only the vertical-screen, stationary displays commonly found in homes, but also mobile devices (e.g., smartphones and tablets), tabletop displays, and HMDs (Head-Mounted Displays). If the same content (stereoscopic video) can be delivered in a format suitable for different types of stereoscopic video display devices, viewing opportunities in daily life will increase, and experiences that leverage the characteristics of each display device can be expected. Therefore, conventional technologies have been proposed to transform the viewpoint of stereoscopic video so that the video representation is appropriate for each type of stereoscopic video display device (Patent Document 1 and Non-Patent Documents 1-6).

Patent No. 4272966

Vamsi Kiran Adhikarla, Fabio Marton, Tibor Balogh, and Enrico Gobbetti. 2015. Real-time adaptive content retargeting for live multi-view capture and light field display. Vis. Comput. 31, 6-8 (2015), 1023-1032 Manuel Lang, Alexander Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, and Markus Gross. 2010. Nonlinear disparity mapping for stereoscopic 3D. ACM SIGGRAPH 2010 Belen Masia, Gordon Wetzstein, Carlos Aliaga, Ramesh Raskar, and Diego Gutierrez. 2013. Display adaptive 3D content remapping. Comput. Graph. 37, 8 (2013), 983-996. Pere-Pau Vazquez, Miquel Feixas, Mateu Sbert, and Wolfgang Heidrich. 2001. Viewpoint selection using viewpoint entropy. labelManagement vi... Vision, Model. Vis. Work. (2001), 273-280. Mateu Sbert, Dimitri Plemenos, Miquel Feixas, and Francisco Gonzalez,Viewpoint Quality: Measures and Applications, In Proceedings of the First Eurographics conference on Computational Aesthetics in Graphics 2005. Miquel Feixas, Mateu Sbert, and Francisco Gonzalez. 2009. A unified information-theoretic framework for viewpoint selection and mesh saliency. ACM Trans. Appl. Percept. 6, 1 (2009)

The conventional technology described above has a problem in that it does not take into account the composition based on the margin ratio when converting stereoscopic images. Referring to Figure 6, the problems of the conventional technology will be explained in detail. Figure 6 illustrates an example of converting the viewpoint of a stereoscopic image displayed on a stationary display 90 for display on a tabletop display 91.

Arrow a represents the horizontal direction of the virtual space reproduced by the tabletop display 91. The dashed line b connects the center of the tabletop display 91 to the viewer 9's position and represents the shooting direction of the virtual camera. This virtual camera virtually photographs the subject Obj within the virtual space and is usually positioned at the viewpoint (viewer 9's position). Hereafter, the virtual camera may be abbreviated as "camera." The angle θ represents the angle between the horizontal direction of the virtual space and the camera's shooting direction, i.e., the angle between arrow a and dashed line b.

As shown in Figure 6(b), consider the case where the viewpoint of the stereoscopic image is transformed so that the retinal image of the subject Obj is equal between the stationary display 90 and the tabletop display 91 (θ = 0). In this case, a discrepancy in the direction of gravity or horizontal direction may occur between the virtual world and the real world. If such a discrepancy becomes extremely large, it may cause discomfort to the viewer 9.

As shown in Figure 6(c), consider the case where the viewpoint of the stereoscopic image is transformed between the stationary display 90 and the tabletop display 91 to match the direction of gravity in the virtual world and the real world (for example, θ = π/4). In this case, the discrepancy in the angle θ when viewing the subject Obj between the stationary display 90 and the tabletop display 91 becomes large, and the resulting image may not be what the creator of the stereoscopic image intended. Therefore, as shown in Figure 6(d), it is necessary to adjust the angle θ so that the balance between the degree of agreement of the direction of gravity and the similarity of the angle θ is appropriate (for example, θ = π/8).

Here, when changing the viewpoint of a stereoscopic image between different types of stereoscopic image display devices, simply rotating the camera around the subject Ojb may not be sufficient to maintain the composition due to the margins. As shown in Figure 7(a), in the stationary display 90, the margin ratio from the outline of the subject Obj to the top and bottom screen frames is approximately 0.775. This margin ratio is the ratio of the length _LU from the top screen frame to the top edge of the subject to the length _LD from the bottom screen frame to the bottom edge of the subject.

In contrast, as shown in Figure 7(b), when the subject Obj is not rotated (θ=0) when displayed on the tabletop display 91, the margin ratio is approximately 0.357. Also, as shown in Figure 7(c), when the subject Obj is rotated by an angle θ=π/4, the margin ratio is approximately 0.351. Furthermore, as shown in Figure 7(d), when the subject Obj is rotated by an angle θ=π/8, the margin ratio is approximately 0.367. Thus, after the viewpoint transformation of the stereoscopic image, the margin ratio on the tabletop display 91 fluctuates around 0.35, and the composition with margins as when displayed on the stationary display 90 is not maintained.
In Figure 7, the viewer 9 is looking down at the tabletop display 91 at an angle, so the tabletop display 91 is trapezoidal in shape.

The present invention aims to provide a viewpoint transformation device and program that can maintain the composition due to margins when transforming the viewpoint of a stereoscopic image.

To solve the aforementioned problems, the viewpoint conversion device according to the present invention is a viewpoint conversion device that converts the viewpoint of a stereoscopic image so that the stereoscopic image displayed by a first stereoscopic image display device is displayed on a second stereoscopic image display device having a display screen at a different angle from the display screen of the first stereoscopic image display device, and comprises a vertex coordinate calculation unit, an uppermost and lowermost point selection unit, a margin calculation unit, and a conversion parameter calculation unit.

In this configuration, the vertex coordinate calculation unit pre-sets a normalization screen that is perpendicular to the optical axis of the virtual camera at a unit distance from the virtual camera located at the viewpoint, and calculates the coordinates of each vertex of the object when the object of the 3D image is projected onto the normalization screen.
The uppermost and lowermost point selection unit selects the uppermost and lowermost points from the coordinates of each vertex calculated by the vertex coordinate calculation unit.

The margin calculation unit calculates the upper margin from the top frame to the top point of the normalized screen, and the lower margin from the bottom frame to the bottom point of the normalized screen.
The conversion parameter calculation unit calculates the amount of movement required to move the subject along the display screen of the second stereoscopic image display device as a conversion parameter, such that the margin ratio between the upper margin and the lower margin becomes a preset value.

By using these conversion parameters, when displaying the stereoscopic image shown by the first stereoscopic image display device on the second stereoscopic image display device, the relationship between the top and bottom points and the margins is adjusted, thereby maintaining the composition created by the margins.

Furthermore, this invention can also be implemented using a program that causes a computer to function as the aforementioned viewpoint transformation device.

According to this invention, when changing the viewpoint of a stereoscopic image, the composition created by the margins can be maintained.

This diagram illustrates a method for moving the center of gravity of a subject along the display screen in an embodiment. In the embodiment, (a) is a diagram illustrating the abstraction of the shape of the subject, (b) is an explanatory diagram illustrating the projection onto the normalization screen, and (c) is a diagram illustrating the selection of the uppermost and lowermost points. This is a block diagram showing the configuration of a viewpoint changing device in an embodiment. A flowchart illustrating the operation of the viewpoint changing device in this embodiment is shown. This figure shows an example in which the viewpoint of a stereoscopic image is transformed. Figures (a) to (d) show examples of transforming the viewpoint of stereoscopic images using conventional technology. Figures (a) to (d) illustrate the margins in the prior art.

The embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below are intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to these embodiments. Furthermore, the same reference numerals are used for identical means, and their descriptions may be omitted.

(Embodiment)
[Perspective Shifting Techniques]
The following describes a specific method of viewpoint transformation as a prerequisite for explaining the viewpoint transformation device 1 according to the embodiment.
To maintain the composition created by the negative space, when changing the viewpoint of the stereoscopic image, in addition to rotating the camera around the center of the subject Obj as the axis of rotation, the camera is also translated. As shown in Figure 1, this camera operation is equivalent to fixing the camera at viewpoint c, rotating the subject Obj, and then translating the subject Obj. In other words, after rotating the subject Obj, the viewpoint of the stereoscopic image is changed by moving the center of gravity _vg of the subject Obj parallel to the display screen (actual screen) 91a of the tabletop display 91. Note that in Figure 1, the display state of the subject Obj before the movement is shown in darker colors, and the display state of the subject Obj before the movement is shown in lighter colors.

Furthermore, the angle θ between arrow a, which represents the horizontal direction in the virtual space, and the dashed line b, which represents the camera's shooting direction, represents the rotation angle of the subject Obj (i.e., the camera's rotation angle). Also, to prevent the subject Obj's position from changing in the depth direction when changing the viewpoint, the creator will manually set the subject Obj's movement distance.

In this embodiment, the subject object (Obj) is assumed to be an automobile, but the type of subject object (Obj) is not particularly limited. Furthermore, since the three-dimensional shape of the subject object (Obj) must be known, the target scene is assumed to be a 3D computer graphics (CG) consisting of a mesh structure or point cloud format.

After the viewpoint transformation, the centroid v _g of the subject Obj is expressed by the following equation (1), using the unknown transformation parameter α.

Note that s _c represents the center coordinates of the display screen 91a, and d represents the distance between the display screen 91a and the subject Obj. Also, n represents the unit normal vector to the display screen 91a, and t represents the unit vector parallel to the display screen 91a and pointing upward.

When the object Obj has a complex shape, accurately determining the margins in response to changes in the centroid _vg of the object Obj would require an enormous amount of computation. Therefore, the shape of the object Obj is abstracted before determining the margins. Specifically, as shown in Figure 2(a), the shape of the object Obj is abstracted using a bounding box, and the coordinates of the vertices _vi are determined from the box-shaped abstracted object Obj. Here, the coordinates of each vertex _vi of the object Obj after the viewpoint transformation are expressed by the following equation (2).

Note that ∫ _∫ represents a vector that, before the viewpoint transformation, points from the centroid v _g of the subject Obj to each vertex of the abstracted subject Obj (i.e., each vertex of the bounding box). Here, the bounding box refers to the rectangular boundary surrounding the subject Obj in the virtual space.
Furthermore, if the shape of the subject Obj is abstracted using a bounding box, the centroid v _g of the subject Obj can be the average of the vertices vi _￣ of the bounding box.

Substituting equation (1) into equation (2), the coordinates of each vertex _vi are expressed by the following equation (3).

If we denote the constant vector in equation (3) as h _i in equation (4), then the coordinates of each vertex _vi are expressed by the following equation (5).

As shown in Figure 2(b), we consider projecting each vertex _vi onto a normalization screen 91b. This normalization screen 91b is a plane perpendicular to the optical axis of the camera at a unit distance (e.g., 1 meter) from the camera located at viewpoint c. In this case, the homogeneous coordinates _v'i ￣ of the projected point when vertex _vi is projected onto the normalization screen 91b are expressed by the following equations (6) and (7) using a transformation matrix.

P represents the projection matrix onto the normalized screen 91b, and M represents the transformation matrix from the world coordinate system to the camera coordinate system. Furthermore, H _i represents the matrix that translates in the direction of the constant vector h _i = (hi _{i, x} , h _{i, y} , h _{i, z} ), and A represents the matrix that scales the vector by a factor of α.

In Figure 2(b), _s1 to _s4 represent the four vertices of the display screen 91a. Vertices _s'1 to _s'4 are points obtained by projecting vertices _s1 to _s4 of the display screen 91a onto the normalized screen 91b.

The homogeneous coordinates v _i ' ￣ = (v _{' i, x} ￣, v _{' i, y} ￣, v _{' i, z} ￣) from equation (6) are transformed into a three-dimensional vector to obtain the vertex v ' _i represented by equation (8). Note that v _{' i, w} ￣ represents the w component of the homogeneous coordinates v _i ' ￣.

Next, as shown in Figure 2(c), the uppermost point v' top and the lowermost point v' _bottom are selected from the coordinates of the eight vertices v' _i projected onto the normalized screen 91b, as shown in equations (9) to (11 ₎ below. Note that k represents the upward unit vector on the normalized screen 91b.

In other words, the uppermost point v' _top is the point among the vertices v' _i closest to the upper frame s' ₁ - s _{' 4.} Similarly, the lowermost point v' _bottom is the point among the vertices v' _i closest to the lower frame s' ₂ - s _{' 3.} Equation (11) assumes a left-handed coordinate system, where the upward direction of the normalized screen 91b is the y-axis of the left-handed coordinate system. In this case, the camera position (viewpoint c) is assumed to be the horizontal center of the display screen 91a, and the camera does not roll.

The upper margin _m'top from the upper frame _s'1 - _s'4 to the top point _v'top' of the normalized screen 91b is expressed by the following equation (12). Also, the lower margin _m'bottom from the lower frame _s'2 - _s'3 to the bottom point _v'bottom of the normalized screen 91b is expressed by the following equation (13).

Here, the top and bottom margin ratio r _v is defined by the following equation (14). Then, the relationship expressed by the following equation (15) is obtained.

By determining the unknown transformation parameter α from equation (15) and substituting this transformation parameter α into equation (1), the centroid v _g of the subject Obj that satisfies the predetermined margin ratio r _v can be obtained.

[Configuration of the viewpoint changing device]
Referring to Figure 3, the configuration of the viewpoint changing device 1 will be explained.
The viewpoint conversion device 1 converts the viewpoint of the stereoscopic image displayed by the first stereoscopic image display device in order to display the stereoscopic image on a second stereoscopic image display device that has a display screen at a different angle than the first stereoscopic image display device.

In this embodiment, the first stereoscopic image display device is a stationary display 90 having a vertical display screen. The second stereoscopic image display device is a tabletop display 91 having a horizontal display screen.

As shown in Figure 3, the viewpoint transformation device 1 comprises a calculation unit 10 and a drawing unit (viewpoint transformation unit) 20. The calculation unit 10 also includes a subject abstraction unit 11, a vertex coordinate calculation unit 12, an uppermost and lowermost point selection unit 13, a margin calculation unit 14, and a transformation parameter calculation unit 15.

Here, the calculation unit 10 receives the position and orientation information of the actual screen and camera { _s1 , ..., _s4 }, c, and the position and orientation information of the subject Obj before viewpoint transformation ( _vg , _vii￣ ). The angle θ can be determined from the position and orientation information of the actual screen and camera { _s1 , ..., _s4 }, c, and the position and orientation information of the subject Obj before viewpoint transformation ( _vg , _vii￣ ). Furthermore, the calculation unit 10 receives parameters (distance d, margin ratio _rv ) manually set by the creator.

The subject abstraction unit 11 abstracts the shape of the subject Obj using a bounding box. As shown in Figure 2(a), the subject abstraction unit 11 abstracts the shape of the subject Obj into a box shape. For example, the bounding box only needs to be the size that encloses the subject Obj.

The vertex coordinate calculation unit 12 pre-sets the normalization screen 91b and calculates the coordinates of each vertex _v'i when the subject Obj of the 3D image is projected onto the normalization screen 91b. In this embodiment, the vertex coordinate calculation unit 12 calculates the coordinates of each vertex _v'i when the subject Obj, abstracted by a bounding box Box, is projected onto the normalization screen 91b. Specifically, the vertex coordinate calculation unit 12 calculates the coordinates of each vertex _v'i using the above-mentioned equations (2) to (8).

The uppermost and lowermost point selection unit 13 selects the uppermost point v' _top and the lowermost point v' _bottom from among the vertices v' _i calculated by the vertex coordinate calculation unit 12. Specifically, the uppermost and lowermost point selection unit 13 selects the uppermost point v' _top and the lowermost point v' _bottom using the aforementioned equations (9) to (11).

The margin calculation unit 14 calculates the upper margin _m'top from the upper frame _s'1 - _s'4 of the normalized screen 91b to the top point _v'top , and the lower margin m'bottom from the lower frame _s'2 - _s'3 of the normalized screen 91b to the bottom _point . Specifically, the margin calculation unit 14 calculates the upper margin _m'top and the lower margin _m'bottom using the above-mentioned formulas (12) and (13).

The conversion parameter calculation unit 15 calculates the amount of movement required to move the subject obj along the display screen 91a as a conversion parameter α, such that the margin ratio between the upper margin m' _top and the lower margin m' _bottom becomes a preset value r _v . Specifically, the conversion parameter calculation unit 15 calculates the conversion parameter α by solving equations (14) and (15). After that, the calculation unit 10 outputs the calculated conversion parameter α to the drawing unit 20.

The drawing unit 20 transforms the viewpoint of the stereoscopic image based on the transformation parameter α. Specifically, the drawing unit 20 can find the centroid v _g of the subject Obj that satisfies the margin ratio r _v set by the creator by substituting the transformation parameter α input from the calculation unit 10 into equation (1). Then, the drawing unit 20 renders the stereoscopic image by capturing the subject Obj located at the centroid v _g with the camera at viewpoint c.

[Operation of the viewpoint changing device]
Referring to Figure 4, the operation of the viewpoint changing device 1 will be explained.
As shown in Figure 4, in step S1, various parameters are input to the viewpoint changing device 1.
In step S2, the subject abstraction unit 11 abstracts the shape of the subject Obj using a bounding box Box.

In step S3, the vertex coordinate calculation unit 12 pre-sets the normalization screen 91b and calculates the coordinates of each vertex _v'i when the object Obj of the 3D image is projected onto the normalization screen 91b.
In step S4, the uppermost and lowermost point selection unit 13 selects the uppermost point v' _top and the lowermost point v' _bottom from among the vertices v' _i calculated in step S3.

In step S5, the margin calculation unit 14 calculates the upper margin m' _top and the lower margin m' _bottom .
In step S6, the conversion parameter calculation unit 15 calculates the conversion parameter α such that the margin ratio between the upper margin m' _top and the lower margin m' _bottom becomes a predetermined value r _v .
In step S7, the drawing unit 20 transforms the viewpoint of the stereoscopic image based on the transformation parameter α.

[Effects and Actions]
Figure 5 illustrates stereoscopic images when the angle θ is changed while the margin ratio r _v is set to 1, that is, when the upper margin m' _top and lower margin m' _bottom are equal. In the example in Figure 5, the distance d is changed to -0.1, 0, 0.1, and the angle θ is changed to 0, π/8, π/4, 3π/8, π/2. As shown in Figure 5, the viewpoint changing device 1 adjusts the relationship between the top and bottom points and the margins even when the angle θ is changed, so the composition with margins can be maintained. This makes it easier to create stereoscopic images that reflect the creator's intentions.

(Variant)
Although embodiments have been described in detail above, the present invention is not limited to the embodiments described above, and includes design changes and the like that do not depart from the spirit of the present invention.

In the embodiment described above, the composition was explained using the vertical margin ratio, but the viewpoint changing device can similarly maintain the composition using the horizontal margin ratio.

In the embodiments described above, the display screens of the first and second stereoscopic image display devices were assumed to be rectangular in shape. However, the shape of the display screen is not particularly limited. For example, even if the display screen has any shape, such as a circle or an ellipse, the viewpoint transformation device can maintain the composition based on the margin ratio in the same way, as long as information about the edges of the display screen is available.

In the above-described embodiment, the stereoscopic image was explained as a still image, but the stereoscopic image may also be a moving image. In this case, the size of the bounding box should be the range of motion of the subject.

In the embodiment described above, the shape of the subject was abstracted using a bounding box, but it is not necessary to abstract the shape of the subject. In this case, the vertex coordinate calculation unit calculates the coordinates for each vertex of the subject, and the uppermost and lowermost point selection unit selects the uppermost and lowermost points from the coordinates of each vertex. In other words, the number of vertices in equations (9) and (10) should be changed.

In the above-described embodiment, the viewpoint conversion device was described as rendering a stereoscopic image, but the invention is not limited to this. For example, the viewpoint conversion device located on the distribution side may calculate conversion parameters and transmit these parameters to a rendering device located on the viewer side. In this case, the rendering device on the viewer side would render the stereoscopic image.

In the embodiments described above, the viewpoint transformation device was assumed to be an independent piece of hardware; however, the present invention is not limited to this. For example, the present invention can also be implemented using a program that causes hardware resources such as a computer's CPU, memory, and hard disk to function as the viewpoint transformation device. This program may be distributed via a communication line or written to a recording medium such as a CD-ROM or flash memory.

1 Viewpoint Conversion Device 10: Calculation Unit 11, Subject Abstraction Unit 12, Vertex Coordinate Calculation Unit 13, Uppermost/Lowestmost Point Selection Unit 14, Margin Calculation Unit 15, Conversion Parameter Calculation Unit 20, Drawing Unit (Viewpoint Conversion Unit)

Claims (6)

A viewpoint conversion device for converting the viewpoint of a stereoscopic image, in order to display the stereoscopic image displayed by a first stereoscopic image display device on a second stereoscopic image display device having a display screen at a different angle from the display screen of the first stereoscopic image display device,
A vertex coordinate calculation unit calculates the coordinates of each vertex of the subject when the subject of the stereoscopic image is projected onto the normalization screen, which is set in advance at a unit distance from the virtual camera located at the viewpoint and perpendicular to the optical axis of the virtual camera.
The vertex coordinate calculation unit selects the highest and lowest points among the vertices calculated by the vertex coordinate calculation unit,
A margin calculation unit that calculates the upper margin from the upper frame of the normalization screen to the uppermost point and the lower margin from the lower frame of the normalization screen to the lowermost point,
A conversion parameter calculation unit calculates the amount of movement to move the subject along the display screen of the second stereoscopic image display device as a conversion parameter, such that the margin ratio between the upper margin and the lower margin becomes a predetermined value.
A viewpoint changing device characterized by being equipped with the following features. The system further comprises a subject abstraction unit that abstracts the shape of the subject using a bounding box,
The viewpoint transformation device according to claim 1, characterized in that the vertex coordinate calculation unit calculates the coordinates of each vertex when the subject abstracted by the bounding box is projected onto the normalization screen. The viewpoint transformation device according to claim 1, characterized in that the first stereoscopic image display device is a stationary display having a vertical display screen. The viewpoint transformation device according to claim 1, characterized in that the second stereoscopic image display device is a tabletop type display having a horizontal display screen. The viewpoint conversion device according to claim 1, further comprising a viewpoint conversion unit that converts the viewpoint of the stereoscopic image based on the conversion parameters. A program for a computer to function as a viewpoint transformation device according to any one of claims 1 to 5.