JP2022034091A - Three-dimensional data reproduction device - Google Patents

Abstract

To provide a 3D data generation device and 3D data reproduction device, which are compatible with various three-dimensional scenes and various tracking functions.SOLUTION: A 3D data reproduction device (1) provided herein is configured to determine at least either of the scene placement and user's viewpoint in a virtual space on the basis of scene information and tracking information.SELECTED DRAWING: Figure 1

Description

One aspect of the present invention relates to a three-dimensional data reproduction device constituting a system for providing a service for a user to control a viewpoint and appreciate a three-dimensional scene.

The service (6DoF service) in which the user controls the position and direction of the viewpoint to view the 3D scene (3D scene, scene) is a 3D data representing the 3D scene generated by the 3D data generator. It is realized by inputting to the original data reproduction device and reproducing it.

The three-dimensional data reproduction device refers to the tracking information for detecting the direction or position of the user, and generates and presents an image (viewpoint image) corresponding to the three-dimensional scene of the user's viewpoint. For example, a head mounted display (HMD) is used to present an image.

A typical 6DoF service is realized by a single service provider preparing a 3D scene for an HMD with a specific tracking function and providing it to the user. For example, Patent Document 1 discloses a method of generating and distributing a viewpoint image according to a user's state from a three-dimensional scene (VR content) accumulated based on a user's position, speed, and acceleration.

Japanese Unexamined Patent Publication No. 2018-166279

In advanced 6DoF services, it is assumed that various service providers or users provide 3D scene components, and another service provider or user configures and provides 3D scenes. In this case, a large number of diverse 3D scenes will be provided.

In addition, the development of tracking technology has progressed, and HMDs and tracking systems with various tracking functions exist. For example, a system capable of tracking the direction of 3 degrees of freedom indicating the direction of the user and a system capable of tracking a total of 6 degrees of freedom of the direction indicating the direction of the user and the position of the user are used.

When a specific service provider provides a 6DoF service to a user who uses a specific tracking system, the target tracking system can be determined by deciding in advance how to associate the appropriate 3D scene with the user's viewpoint. 6DoF service can be provided to users. However, when providing various 3D scenes to users who use various tracking systems, it is difficult to design in consideration of all combinations in advance, and the conventional technology provides a preferable viewpoint image to the user. There was a problem that it could not be done.

In order to solve the above problems, the predictive image generation device according to one aspect of the present invention is a scene arrangement unit that determines the scene arrangement based on the scene information and the tracking information, or a user viewpoint based on the scene information and the tracking information. The tracking information includes tracking system information.

According to one aspect of the present invention, when playing back various three-dimensional scenes with various tracking systems, it is possible to synthesize and present a viewpoint image suitable for the user.

It is a functional block diagram of the 3D data reproduction apparatus which concerns on Embodiment 1. FIG. It is a functional block diagram of the 3D data distribution system including the 3D data reproduction apparatus which concerns on Embodiment 1. FIG. It is a figure which uses a 3D scene, a virtual space (drawing target space), a user viewpoint, a tracking system, and a viewpoint image for explanation. It is a flow diagram which shows the procedure of 3D data reproduction processing. It is a figure which illustrated the structure of the tracking information. It is a figure which illustrates the structure of 3D scene information. It is a figure which shows the example of the 3D scene arrangement in the arrangement target area using a scene range.

Hereinafter, embodiments of the present invention will be described in detail. However, the configuration described in the present embodiment is not intended to limit the scope of the present invention to the present invention unless otherwise specified, and is merely an explanatory example.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic diagram showing a configuration of a three-dimensional data reproduction device 1 according to the present embodiment (first embodiment, first embodiment). The 3D data reproduction device 1 is composed of a tracking information acquisition unit 11, a 3D scene information acquisition unit 12, a 3D scene arrangement unit 13, a viewpoint setting unit 14, a media data acquisition unit 15, and a viewpoint image drawing unit 16. Generally, the 3D data reproduction device 1 generates a viewpoint image by arranging a 3D scene (3D scene) acquired from the outside in a virtual space (drawing target space) and drawing according to the user's viewpoint. And output.

Prior to the explanation of the 3D data reproduction device 1, refer to FIG. 3 for the 3D scene and scene coordinates, the user viewpoint and virtual space (drawing target space), the tracking system and tracking coordinates, and the viewpoint image referred to in the following explanation. I will explain.

FIG. 3A is a diagram illustrating the relationship between the 3D scene and the scene coordinates. A 3D scene is composed of multiple components. FIG. 3 (a) shows the arrangement of the 3D scene S1 as viewed from the front (direction orthogonal to the x-axis and y-axis) and the arrangement of the 3D scene S1 as viewed from above (direction orthogonal to the x-axis and z-axis). It is shown in the figure. The 3D scene S1 contains two 3D objects (O1 and O2) as components. Further, although not shown, it includes a component BG corresponding to a background image. The positions and sizes of the 3D objects O1 and O2 are represented using scene coordinates.

FIG. 3 (b) is a diagram illustrating the relationship between the virtual space (drawing target space), the user's viewpoint, and the 3D scene.

The virtual space is used to draw the viewpoint image by arranging the 3D scene and the user's viewpoint in a common space. In that sense, the virtual space is also called the drawing target space. Figure 3 (b) illustrates the layout of the virtual space as seen from the front (directions orthogonal to the x-axis and y-axis) and the virtual space as viewed from above (directions orthogonal to the x-axis and z-axis). ing. The 3D scene S1 and the user viewpoint VP are arranged in the virtual space. The user viewpoint VP includes information (illustrated by a triangle) corresponding to the line-of-sight direction and the visual field in addition to the position in the virtual space. Coordinates for expressing the position and size in virtual space are called presentation coordinates. Scene coordinates and presentation coordinates do not always match, but they may.

FIG. 3 (c) is a diagram illustrating the relationship between the tracking system, the tracking coordinates, and the user position.

The tracking coordinates are the coordinates used by the tracking system and can express the detected position and direction. FIG. 3 (c) shows an overview of the tracking range TA in an outside-in tracking system using two stationary infrared sensors (T1 and T2) when viewed from above in the vertical direction. The tracking system can detect the user position DP within the tracking range TA. The user position DP is expressed in tracking coordinates.

FIG. 3 (d) is a diagram illustrating a viewpoint image from a specific user's viewpoint.

The viewpoint image corresponds to an image when the scene S1 installed in the virtual space is observed from the user's viewpoint installed in the virtual space as described in FIG. 3 (b). In the example of the arrangement shown in Fig. 3 (b), both the objects O1 and O2 included in the 3D scene S1 are in the field of view from the user's viewpoint VP, and the object O2 is on the viewpoint side of the object O1. , The image shown in Fig. 3 (d) is the viewpoint image.

(Viewpoint video playback processing flow)
FIG. 4 is a flow chart showing a procedure of three-dimensional data reproduction processing.

The three-dimensional data reproduction device 1 reproduces three-dimensional data by generating and outputting a viewpoint image at each time point based on 3D scene information and tracking information input for each time. The three-dimensional data reproduction process at time t is executed by the following procedure.

(T1) The tracking information acquisition unit 11 acquires the tracking information at time t and outputs it to the 3D scene arrangement unit 13 and the viewpoint setting unit 14.

(T2) The 3D scene information acquisition unit acquires the 3D scene information at time t and outputs it to the 3D scene arrangement unit 13 and the viewpoint setting unit 14.

(T3) The 3D scene arrangement unit 13 arranges a scene in the drawing target space based on the input tracking information and 3D scene information, and outputs the arrangement result as the scene arrangement information to the media data acquisition unit 15 and the viewpoint image drawing unit 16. do.

(T4) The viewpoint setting unit 14 determines the user viewpoint in the drawing target space based on the input tracking information and the 3D scene information, and outputs the data to the media data acquisition unit 15 and the viewpoint video drawing unit 16.

(T5) The media data acquisition unit 15 determines the media data necessary for drawing the viewpoint image based on the input scene arrangement information and the user viewpoint, acquires the determined media data from the outside, and acquires the determined media data from the outside to obtain the viewpoint image drawing unit 16. Output to.

(T6) The viewpoint image drawing unit 16 draws and outputs a viewpoint image based on the input scene arrangement information, user viewpoint, and media data, and ends the three-dimensional data reproduction process at time t.

The three-dimensional data reproduction device 1 reproduces three-dimensional data by generating a viewpoint image at each time according to the above procedure. The procedure does not necessarily have to be in the exact order described in FIG. 4, and the order may be changed or executed in parallel as long as the input / output dependencies allow. For example, tracking information acquisition (T1) and scene information acquisition (T2) can be executed in parallel without any dependency.

(Tracking information acquisition unit 11)
The tracking information acquisition unit 11 acquires the tracking information from the tracking system and outputs it to the 3D scene arrangement unit 13 and the viewpoint setting unit 14. The tracking information includes positional information and tracking system information. The tracking system estimates at least one of the user's position or direction in the real world at predetermined time intervals and uses it as position direction information. The tracking system includes a system that detects the direction with a gyro sensor (3DoF system), a system that detects the position and direction of the user using a fixedly installed external sensor (outside-in 6DoF system), and equipment worn by the user. There is a system (inside-out 6DoF system) that detects the position and direction by self-position estimation using the image of the mounted camera. The position direction information is represented by, for example, a three-dimensional position vector represented by three-dimensional tracking coordinates and a three-dimensional vector representing rotation in the tracking coordinate system.
A three-dimensional vector representing rotation can be expressed by, for example, three parameters of rotation, roll, pitch, and yaw, which are commonly used parameters of rotation by combining rotations around each axis.

The tracking system information generally includes information indicating the function and performance of the tracking system. Since the tracking system information is changed less frequently than the positional information, the amount of tracking information data can be reduced by including it in the tracking information less frequently.

The tracking system information may include a tracking system type. The tracking system type is information that identifies which of the default tracking systems the tracking system used by the user is. For example, if the default tracking system type is 3DoF system, outside-in 6DoF system, or inside-out 6DoF system, the tracking system type included in the tracking system information can be used to identify which tracking system the user is using. .. The tracking system type can be used to determine the preferred scene layout and viewpoint position for the user.

The tracking system information may additionally include information (tracking function information) indicating whether or not a specific tracking function is provided and the degree of the function. Hereinafter, specific examples of tracking function information will be shown.

(1) Presence / absence of directional tracking function: Information indicating whether directional tracking is enabled or disabled. The presence or absence of the direction tracking function may be expressed by a set of the presence or absence of the tracking function for each degree of freedom indicating the direction. For example, when the direction is represented by three parameters of roll, pitch, and yaw, the set of the presence / absence of the rotation tracking function in the direction corresponding to each parameter may be the presence / absence of the direction tracking function. Although the amount of data increases compared to the case where the directional tracking function is expressed by a single flag, it is possible to support a wider variety of tracking systems.

(2) Presence / absence of position tracking function: Information indicating whether position tracking is enabled or disabled. The presence or absence of the direction tracking function may be expressed by the set of the presence or absence of the tracking function for each degree of freedom representing the position. For example, the set of the presence / absence of the tracking function in the direction of each coordinate axis whose position is the tracking coordinate may be the presence / absence of the position tracking function. Although the amount of data increases compared to the case where the directional tracking function is expressed by a single flag, it is possible to support a wider variety of tracking systems. In addition, the validity / invalidity of the function may be indicated for each type of position tracking. For example, whether or not there is a position tracking function based on information that individually indicates the presence or absence of tracking using GPS, the presence or absence of position tracking with respect to a reference point fixed in space, and the presence or absence of relative position tracking with the user position at a specific time as the reference point. Corresponding information may be configured.

(3) Directional tracking range: Information indicating the range in which directional tracking is effective. For example, the direction is defined by combining three types of rotations, roll, pitch, and yaw, and the range of rotation angles that each rotation component can take is specified. As a result, it is possible to notify information such that only horizontal rotation can be tracked and only vertical rotation can be tracked.

(4) Position tracking range: Information indicating the range in which position tracking is effective. For example, the position is specified by the range that can be taken by each of the x-axis, y-axis, and z-axis coordinate components defined in the tracking coordinate system. A range may be expressed by specifying a geometric shape that represents a traceable space. In that case, more flexible range specification becomes possible. The space may be divided into small areas by a grid or an ocree, and the range may be expressed by showing information within or outside the range for each small area. In that case, the amount of data increases, but it can be determined at high speed whether it is within or outside the range.

(5) Tracking interval: Information representing a time interval in which tracking is executed and direction or position information is acquired. Different tracking intervals may be set for the direction and position. The amount of 3D data to be received can be reduced by using it to determine the target to be acquired in advance when acquiring media data.

(6) Tracking delay: Information representing the delay until tracking is executed and direction or position information is acquired. Different tracking delays may be set for the direction and position. The amount of 3D data to be received can be reduced by using it to determine the target to be acquired in advance when acquiring media data.

The tracking system information may additionally include a position (tracking reference position) that is a reference of the position information included in the tracking information. For example, the ground position detected by the tracking system can be used as a reference position. You can use the ground position to align the ground position in real space with the ground position in the 3D scene. Further, for example, the tracking system information may include the user position (starting position) at the time of starting the tracking system or starting the application as a reference position. Using the startup position, you can more flexibly determine the position in the 3D scene than when using only the current position. Both the ground position and the start-up position may be included in the tracking information as a reference position. Further, the tracking system information may include coordinates with the vertical component as the ground position and the horizontal component as the starting position as the reference position.

FIG. 5 is a diagram illustrating the configuration of tracking information.

The tracking information shown in FIG. 5A periodically includes tracking system information. As a result, the 3D data reproduction device can acquire the tracking system information even when the tracking system is connected to the tracking system from the middle (from the time t> 0). Further, the tracking information includes the position direction information at each time. As shown in FIG. 5A, the number of tracking system information in the tracking information is preferably smaller than the number of position direction information. In other words, the reception frequency of the tracking system information is preferably lower than the reception frequency of the position direction information. Since the tracking system information changes less frequently than the positional information, the tracking information can be expressed with a small amount of data by the above configuration.

FIG. 5 (b) shows a configuration example of tracking system information. Tracking system information,
It contains the tracking system type (tracking_system_type) and the value is set to "3dof". Further, the tracking system information includes tracking function information (tracking_system_info). The tracking function information includes the presence / absence of the direction tracking function (orientation_tracking), the presence / absence of the position tracking function (position_tracking), the direction tracking range (orientation_tracking_range), the position tracking range (position_tracking_range), the tracking interval (tracking_frequency), and the tracking delay (tracking_latency). .. Also, the tracking system information includes the tracking reference position (tracking_ref_pos).

(3D scene information acquisition unit 12)
The 3D scene information acquisition unit 12 acquires 3D scene information and outputs it to the 3D scene arrangement unit 13 and the viewpoint setting unit 14. 3D scene information is also called scene information. To acquire the 3D scene information, the data recorded on the recording medium may be actively acquired, or the transmitted data may be passively acquired. The 3D scene information includes the components included in the 3D scene and the arrangement information of each component in the space. In addition, the 3D scene information includes scene placement auxiliary information that is referred to when the scene is placed in the virtual space. The scene arrangement auxiliary information includes at least one or more of a scene range, a scene observation range, a scene principal direction, a scene scale, and a scene reference ground information. Since the scene arrangement can be determined without analyzing each component of the 3D scene in detail by the scene arrangement auxiliary information, the processing required for the scene arrangement is reduced. Therefore, even when the processing amount of the 3D data reproduction device is limited, it is possible to determine the scene arrangement for various 3D scenes and generate the viewpoint image.

FIG. 6 is a diagram illustrating the structure of 3D scene information.

The 3D scene information (scene_info) shown in FIG. 6 corresponds to the 3D scene information describing the 3D scene information described in FIG. 3 (a). The 3D scene to be described shown in FIG. 3 (a) includes 3D object O1 and 3D object O2. Also, although not shown, the 3D scene is the scene background BGV.
, Scene audio BGA, audio object A1, audio object A2. Here, the voice object A1 corresponds to the 3D object O1, and the voice object A2 corresponds to the sound emitted by the 3D object O2. The 3D scene information (scene_info) shown in FIG. 6 includes a set of scene components. The scene component set includes a scene background (scene_background), a scene audio (scene_audio), two 3D objects (3d_object), and two audio objects (audio_object) as components. For each component, the location (url) of the media data required to play the component is included. The location of media data is specified, for example, by a character string representing a URL (Uniform Resource Locator). It also contains the object position (pos) and object orientation (dir) in the scene for each component corresponding to the 3D object and the audio object. The object position is described, for example, by a three-dimensional position vector in the scene coordinate system. [X0, y0, z0] in FIG. 6 represents a three-dimensional vector having x0, y0, and z0 as elements. The object direction is described. The object direction in the scene is described by rotation from the scene reference direction to the object direction. The object direction is described by, for example, a three-dimensional vector corresponding to the roll, pitch, and yaw components of rotation. [Roll0, pitch0, yaw0] in Fig. 6 represents a three-dimensional vector whose elements are roll0, pitch0, and yaw0. In addition, as the object direction, information indicating that the object direction is not specified may be described. In Fig. 6, the character string "na" is described in the value of the object direction (dir) to show that it does not have the object direction. The object position and the object direction may be described by another representation containing equivalent information. For example, the object position and the object direction may be matched and expressed by a matrix representing the transformation.

The 3D scene information includes a scene principal direction (scene_orientation) that represents the direction of the entire scene. The scene principal direction is described by a three-dimensional vector in the scene coordinate system. For example, a vector representing the direction opposite to the direction in which the 3D scene is viewed from the front is set as the scene direction. By using the scene principal direction, it is possible to perform a suitable arrangement according to the tracking performance when determining the scene arrangement. Details will be described in the explanation of the 3D scene arrangement unit 13. In other words, the scene principal direction can be said to be information indicating a recommended direction for observing the scene. From this point of view, it may be described as the value of the scene principal direction that the scene principal direction does not exist, as in the direction of the 3D object described above. Further, the principal direction of the scene does not necessarily have to be one direction, and may be configured from a plurality of directions. However, in that case, it is preferable to describe the directions in order from the most important direction so that the importance of the direction can be easily determined.

The 3D scene information includes the scene range (scene_range). The scene range is information indicating the range occupied by the entire scene described in the 3D scene information in the scene coordinate system. By using the scene range, the occupancy range of the entire scene can be known without individually calculating the layout and occupancy range of all the components of the 3D scene, so the amount of processing for scene layout calculation and viewpoint determination is reduced. .. The scene range can be expressed by, for example, a rectangular parallelepiped that includes the entire scene. In the example of FIG. 6, the rectangular parallelepiped is described by the positions of two vertices facing each other (box_corner). The scene range is not limited to a rectangular parallelepiped, and may be expressed by a convex hull that includes the entire scene. However, from the viewpoint of simplifying the scene arrangement calculation and reducing the amount of transmission data, the representation by the rectangular parallelepiped size is preferable. Further, the scene occupied space information does not necessarily have to describe a three-dimensional area, and a two-dimensional area may be described. For example, if the scene has ground, an expression that specifies a two-dimensional area horizontal to the ground can be used. In this case, although the application is limited, it is possible to reduce the derivation process of the scene range in the three-dimensional data generator and reduce the amount of data in the scene range. By including information (scene_range_type) indicating which expression method the scene range is described in in the scene range information, it is possible to select a preferable scene range expression method according to the nature of the 3D scene.

The 3D scene information includes the scene observation range (scene_observe_range). The scene observation range is information indicating the range in which the target 3D scene can be observed in the scene coordinate system. Unlike real 3D scenes, virtual 3D scenes represented by media data cannot always be observed from all positions and directions. For example, the background of a 3D scene expressed in a 360-degree video is within a certain distance centered on the shooting point because the deviation from the observation result by the eyes of the actual 3D scene increases when the distance from the shooting point of the 360-degree video is large. Specify the range of to the scene range. The scene observation range can be described by the same expression as the above-mentioned scene range. By using the scene observation range, the observable range of the entire 3D scene can be known without examining the observation range of the individual components that make up the 3D scene, so the amount of processing for scene placement and viewpoint determination can be reduced.

The scene information includes the scene scale (scene_scale). The scene scale is information indicating the ratio for adjusting the coordinate unit of the entire scene. For example, as a scene scale, the ratio between the unit of coordinates and the unit of meters used to describe the object position in 3D scene information can be used. If the application that uses the 3D scene information is known in advance, the 3D scene information may be configured using the unit of coordinates so that the application can be easily processed. However, in a situation where the application is not determined when creating 3D scene information, the 3D scene information can be used after being converted into a unit of desired coordinates using a scene scale. In the example of FIG. 6, the scale ratio value 0.01 is described in the scene scale. In this case, 0.01d, which is the value obtained by multiplying the distance d in the scene coordinate system by the scale ratio, is the distance in meters.

The 3D scene information includes scene reference ground information (scene_ground_info). The scene reference ground information is information representing the ground of a 3D scene. The scene reference ground information contains at least the information necessary to derive the height of the ground in a 3D scene. For example, when the y-axis of the scene coordinate system corresponds to the vertical direction, the height of the reference ground can be derived by describing the y-coordinate value corresponding to the ground. By deciding the scene arrangement and the user's viewpoint position with reference to the reference ground position, it is possible to present an image with less discomfort in height to the user.

(3D scene arrangement part 13)
The 3D scene arrangement unit 13 determines and outputs the scene arrangement based on the 3D scene information and the tracking system information. The scene arrangement is information indicating the position, direction, and scale of the scene described by the 3D scene information in the drawing target space. The scene arrangement is represented by, for example, a 4 × 4 size matrix. The following is a specific example of the scene arrangement derivation process.

<Scene layout derivation process 1: Scene layout at actual size>
The scene arrangement derivation process when there is a request to arrange the scene in the drawing target space in the actual size is as follows. The 3D scene arrangement unit 13 is arranged in the drawing target space after adjusting the scene to the actual size by referring to the scene arrangement auxiliary information included in the 3D scene information. Specifically, the coordinates of the 3D objects and audio objects that make up the scene are adjusted with reference to the scene scale, and each object is placed. If the scene scale is α, the object position p in the 3D scene information, and the scene reference position p0 are given, the object position p'in the drawing target space is derived by the expression p'= α (p --p0). When the scene is arranged in the drawing target space in the actual size by the above procedure, it is possible to realize an application in which the user freely moves and searches in the virtual space corresponding to the scene.

Further, the 3D scene arrangement unit 13 is arranged in the drawing target space after adjusting so that the ground position of the scene matches the ground position of the drawing target space with reference to the scene arrangement auxiliary information included in the 3D scene information. Specifically, the 3D scene is arranged so that the height of the ground of the scene matches the height of the ground assumed in the drawing target space with reference to the scene reference ground information. By arranging the scenes according to the above procedure, it is possible to reduce the discomfort caused by the difference in height that the user feels in the application.

<Scene placement derivation process 2: Scene placement in the placement target area>
The scene arrangement derivation process when there is a request to arrange a scene in the arrangement target area in the drawing target space is as follows. The 3D scene arrangement unit 13 refers to the scene arrangement auxiliary information included in the 3D scene information, adjusts the scene so that it fits in the arrangement target area, and then arranges the scene in the drawing target space. Specifically, the coordinates representing the position of each object included in the scene are corrected by referring to the scene range.

For example, when both the placement target area and the scene range are specified by a rectangular parallelepiped, the 3D scene is rotated and scaled so that the rectangular parallelepiped representing the scene range is included in the rectangular parallelepiped representing the placement target area and placed in the drawing target space. It is preferred to rotate and scale under conditions that maximize the size of the scaled scene range. By arranging the 3D scene by the above procedure, it is possible to realize an application for arranging and viewing a virtual scene in a designated area (for example, on a table) in the real world.

The arrangement target area does not necessarily have to be specified in a three-dimensional space, and may be specified as a two-dimensional area on a plane.

FIG. 7 is a diagram showing an example of 3D scene placement in the placement target area using the scene range.

FIG. 7 (a) shows the scene range TA in the scene coordinate system. Figure 7 (b) shows the placement target range TB in the drawing target space. FIG. 7 (c) shows the scene range TA'which is moved, rotated, and scaled so as to be included in the arrangement target range TB in the drawing target space.

When expressing the scene range in a two-dimensional area, the natural arrangement of the scene can be realized by referring to the vertical direction of the scene included in the scene arrangement auxiliary information. As a specific procedure, a rectangular two-dimensional occupied area on the horizontal plane of the scene is derived with reference to the scene occupied space and the vertical direction of the scene. Then, move, rotate, and scale the scene so that the two-dimensional occupied area is included in the placement target area. By the above procedure, a natural arrangement can be realized when arranging a scene in which the vertical direction is defined. For example, it is possible to prevent an unnatural tilt from occurring when a building transmitted as a virtual scene is placed on a table in the real world and viewed.

Further, the 3D scene arrangement unit 13 can arrange the scene in consideration of the user's viewpoint position by referring to the tracking information. Specifically, the user's viewpoint position at the time of arranging the 3D scene is referred to, and the scene is arranged so that the front of the scene arranged from the user's viewpoint position becomes the user's line of sight. As a procedure, first, when the user viewpoint position is pu and the center of the placement target area is pt, the scene is rotated so that the direction of the vector from pt to pu and the direction of the scene principal direction match. Then move and scale the scene so that it is included in the placement target area. By arranging the scene according to the above procedure, in the application in which the user specifies a specific area and arranges the virtual scene, the scene is arranged so that the front of the scene faces the user side. Therefore, the user can appreciate the scene without adjusting the orientation of the scene after the arrangement.

(Viewpoint setting unit 14)
The viewpoint setting unit 14 determines and outputs the user viewpoint position in the scene to be drawn based on the 3D scene information and the tracking information. By applying at least one of the viewpoint setting processes shown below to determine the user's viewpoint position, a suitable viewpoint can be set even when viewing various 3D scene information using tracking systems with different performances. can.

<Viewpoint setting process 1A: Height adjustment according to tracking system type>
When the type of the tracking system is the outside-in method, the coordinates detected by the tracking system are fixed with respect to the real world space. This is because the detected coordinates depend on the position of the external sensor fixed in the real world. On the other hand, when the tracking system type is the inside-out method, the coordinates detected by the tracking system are not fixed with respect to the real world space. This is because the detected coordinates depend on the camera position at a specific time (for example, the position of the camera-mounted HMD at startup). Therefore, when the tracking system type is the outside-in method, the height obtained by tracking can be used as it is for the height of the user's viewpoint position from the ground (or the reference plane). On the other hand, in the case of the inside-out method, the tracking system estimates the height from the ground and adds it to the tracking information as height correction information and sends it. The viewpoint setting unit 14 can set a preferable user viewpoint position in the drawing target space by correcting the position information included in the received tracking information with reference to the height correction information. The same function can be realized even when the tracking system corrects the height information and sends the corrected position information as tracking information. However, since the height may not be required depending on the application, the former configuration is preferable because it can support various applications.

Height information can also be included in the tracking information at regular intervals. In that case, the viewpoint setting unit 14 corrects the position information with reference to the latest height information and sets the viewpoint position. If the height information changes discontinuously here, the user's viewpoint position may change suddenly, which may cause discomfort to the user. Therefore, it is necessary to correct the change in the user's viewpoint position over time so that it is within a certain range. For example, from the time when the latest height information is received, the height information used for calculation is set to smoothly change from the previous height information to the latest height information according to the time over a specified time. It is possible to suppress a sudden decrease in the viewpoint position. When estimating the height from the ground using the stereo method in an inside-out tracking system, it is often the case that the height of the ground cannot be estimated accurately immediately after startup. In such a case, by updating the height information as described above, the content can be presented without making the user wait until the height estimation is completed.

<Viewpoint setting process 1B: Height adjustment by tracking function information>
When the tracking in the height direction is disabled in the tracking function information, it is necessary to estimate and set the height of the user's viewpoint position from the ground. The viewpoint setting unit 14 estimates the height of the user's viewpoint position from the ground by the following procedure. When the 3D scene information includes the recommended viewpoint position, the user viewpoint position is set so that the height corresponds to the recommended viewpoint position. Otherwise, set the user's viewpoint position to the default height. As the default height, for example, the eye height derived based on the height registered in advance by the user can be used. It should be noted that there are cases where it is not appropriate to use the tracking information in the height direction even in a tracking system that can use the tracking in the height direction. For example, when a user wearing an HMD equipped with an inside-out 6DoF tracking function sits on a chair and moves in the virtual space using a slipper-type controller, the height from the ground in the real space is the user's viewpoint in the virtual space. It should not be reflected directly in the height of the position. In such a case, a preferable viewpoint position can be set by notifying the tracking function information that tracking in the height direction is invalid and then setting the user viewpoint position by the above procedure. In the tracking system, the necessity of tracking in the height direction is estimated and controlled from the posture of the user and the type of the device being used, so that the procedure for the user to directly set the tracking function information can be omitted.

It should be noted that the tracking function information may be notified of information that enables only relative position tracking in the height direction and invalidates tracking of the absolute position in the height direction. The height of the first user viewpoint position is set based on the 3D scene information or the default value as in the above case. After that, the height of the user's viewpoint position is derived based on the relative height change calculated by referring to the tracking information received for each time. According to such a configuration, the user can walk in the virtual space to appreciate the scene while sitting in the real world, and in addition, the position of the head can be changed to change the position of the viewpoint.

From the above description of the viewpoint setting process 1A and the viewpoint setting process 1B, the viewpoint setting unit 14 determines the height of the viewpoint in the drawing target space from the reference plane (ground) based on the tracking type or the tracking function information. Can be expressed.

<Viewpoint setting process 2: Adaptation of viewpoint position at start by 3DoF and 6DoF>
When the front orientation of the scene is notified based on the scene orientation information, if the tracking type is 3DoF tracking, the line-of-sight direction of the user's viewpoint is set so that the front orientation of the user in the real world matches the front orientation of the scene. By making a decision, it is possible to provide the user with a preferable scene observation environment as compared with the case where the line-of-sight direction is set in another direction. On the other hand, when the tracking type is 6DoF tracking, there is a situation where it is not always appropriate to set the line-of-sight direction so that the front direction of the user in the real world matches the front direction of the scene. In general, 6DoF tracking does not allow tracking to an infinite range. Therefore, in order to provide an opportunity to enjoy the observation of the scene to the maximum, it is preferable to set the user viewpoint based on the scene observation range, the tracking range, and the tracking position. Specifically, in the drawing target space, the user viewpoint at the start time is determined so that the overlap range between the range corresponding to the scene observation range and the tracking range is maximized. At the subsequent time, the user viewpoint at each time is determined by moving the user viewpoint at the start time based on the relative change of the viewpoint position included in the tracking information. In the drawing target space, the user cannot move to a scene outside the tracking range. Therefore, as the overlap between the scene observation range and the tracking range increases, the scene can be viewed from various positions. Therefore, by determining the user's viewpoint at the start time so as to increase the overlapping range of the scene observation range and the tracking range, the user can enjoy the scene viewing more.

In the above explanation, the tracking type of 3DoF tracking corresponds to invalid position tracking, and the tracking type of 6DoF tracking corresponds to effective position tracking. Whether the position tracking is enabled or disabled may be determined by referring to the presence / absence of the position tracking function included in the tracking function information.

In the above description, an example of using the scene observation range has been described, but a similar effect can be realized by using the scene range. Specifically, a range obtained by adding a predetermined margin corresponding to the viewable distance to the scene range can also be used as the scene observation range. In this case, the degree of freedom in specifying the scene observation range is reduced, but the amount of scene information data can be reduced.

In determining the start time user viewpoint position when using 6DoF tracking, the user viewpoint position may be determined by adding an additional condition to the above-mentioned condition for maximizing the overlap range. Specifically, it is possible to use an additional condition of referring to the object position described in the 3D scene information and the occupied range of each object and setting the user viewpoint position to a position not included in the occupied range of any object. It is possible to avoid the unfavorable condition that the user is inside the object at the start time. Here, the occupied range of the object is information representing the range of the space occupied by the object, and can be expressed by, for example, a rectangular parallelepiped containing the object.

Further, when the scene observation range is smaller than the tracking range, an additional condition for setting the position and direction of the user's viewpoint so that the scene occupied space is arranged at a position close to the center of the tracking range can be used. This is preferable because the range in which the scene can be observed from the outside increases. In addition, in the case of outside-in 6DoF tracking, the central part of the tracking range is safer and the chances of out-of-range alerts can be reduced, thus increasing user satisfaction.

According to the viewpoint setting process 2, the viewpoint setting unit 14 identifies whether position tracking is valid or not based on the tracking type or tracking function information, and the user viewpoint is different depending on whether the position tracking is enabled or disabled. To decide. This makes it possible to set a suitable viewpoint position when using various tracking systems.

Further, according to the viewpoint setting process 2, when the position tracking is effective, the viewpoint setting unit 14 sets the viewpoint position using the scene range or the scene observation range included in the scene information. This makes it possible to set a suitable viewpoint position for various 3D scenes when position tracking is effective.

<Viewpoint setting process 3: Fitting of movable range>
When the scene observation range is wider than the tracking range in the drawing target space, the user cannot search the entire scene by the usual method of determining the user's viewpoint position. In such a case, by adjusting the user's viewpoint position according to the position within the scene observation range and the ratio between the scene observation range and the tracking range, it is possible to search the entire scene even in a narrow tracking range. Specifically, the user's viewpoint position is determined by the following procedure. First, the ratio β of the scene observation range and the tracking range in the drawing target space is derived by referring to the scene observation range included in the 3D scene information and the tracking range included in the tracking information. The ratio β means that the scene observation range is β times the tracking range. Subsequently, the displacement obtained by multiplying the displacement of the viewpoint position included in the tracking information by β is applied as the displacement of the user viewpoint position in the drawing target space. By setting the user viewpoint position by the above procedure, when the user moves from one end of the tracking range to the other, the user moves from one end to the other of the scene occupied space in the drawing target space. Will be able to explore the entire scene. When moving the viewpoint position by using the displacement multiplied by the above ratio β, it is preferable to present the user with information indicating that fact. For example, it is presented as "βx moving at high speed". As a result, the user can perceive that there is a gap between the movement amount in the real world and the movement amount estimated from the presented image, so that the sickness caused by the gap between the motion and the visual information can be suppressed. In addition, the movement of the body and the movement of the head are detected respectively, the viewpoint movement adjustment using the above-mentioned ratio β is applied to the movement of the body, and the normal movement of the head (equivalent to the ratio 1) is applied. You may move the viewpoint. As a result, the user can search the entire scene and observe the scene with natural parallax when the head is moved.

According to the viewpoint setting process 3, the viewpoint setting unit 14 determines the user viewpoint using the tracking range included in the tracking information and the scene range or the scene observation range included in the scene information. This makes it possible to set a suitable viewpoint position when using various scenes and various tracking systems.

<Viewpoint setting process 4: Processing when tracking freedom is restricted>
When a specific degree of freedom of tracking is restricted in the tracking function information, a more preferable experience can be provided to the user by determining the position and direction of the user's viewpoint in consideration of the restriction. Specifically, when the degree of freedom of directional tracking is restricted, the directional component of the restricted degree of freedom is derived so as to match the corresponding component of the scene principal direction included in the 3D scene information. For example, if horizontal rotation is enabled and vertical rotation is disabled, the combination of the horizontal rotation component detected by tracking the user's line-of-sight direction and the vertical rotation component in the main direction of the scene Derived. This allows the user to observe the scene from a preferred direction even in a situation where the user is observing the scene in a lying position using only the horizontal rotation of the neck.

(Media data acquisition unit 15)
The media data acquisition unit 15 identifies the media data necessary for generating the viewpoint image by referring to the 3D scene information, the scene arrangement, and the user viewpoint position. Then, the specified media data is output. The media data is specified by the following procedure. First, the components of the scene necessary for generating the image of the user's viewpoint are specified based on the position of the user's viewpoint and the direction of the line of sight in the rendering target space. Specifically, the 3D object and the scene background included in the user's field of view in the rendering target space are selected. When the 3D object O1 is included in the user's view and the 3D object O2 is not included in the 3D scene illustrated in FIG. 3 (a), the scene background BGV and the 3D object O1 are selected as the objects for which media data is required. Subsequently, for each selected component, media data is acquired and output by referring to the URL corresponding to each component included in the 3D scene information.

<Rendering of background omitted>
According to the above description, the scene background is selected as necessary for generating the viewpoint image regardless of the position of the user's viewpoint. However, when arranging and observing a scene in the real world, the background of the scene is not always necessary. Therefore, it is preferable not to select the scene background when the user's viewpoint position is set outside the scene in the drawing target space by referring to the scene occupied space included in the 3D scene information. As a result, the amount of media data transmitted required for scene reproduction can be suppressed.

<Selection of audio media>
In the above, the method of selecting the media data necessary for generating the image of the scene has been described, but when the sound of the scene is reproduced depending on the viewpoint, it is necessary to select the scene component related to the sound and acquire the media data. When the voice object O1 is included in the user's view and the voice object O2 is not included in the scene illustrated in FIG. 3 (a), the scene voice BGA, the voice object A1, and the voice object A2 are used as objects that require media data. select. Here, the audio object O2 that is not included in the visual field is selected because, unlike the visual information, the sound information can perceive sound in all directions regardless of the visual field. It should be noted that a selection method may be used in which the reach range of the sound is set in the voice object and the voice object is selected only when the user's viewpoint is within the reach range of the sound. In that case, the amount of media data transmitted for sound reproduction can be reduced. Further, an amount representing the loudness of the sound may be set in the voice object, and the reachable range of the sound may be calculated and derived according to the loudness. In that case, the calculation is increased as compared with the case of setting the direct reach range, but it becomes possible to select an appropriate audio object even in scenes having different sound transmission rates.

<Prefetch considering tracking range>
The acquisition of media data in the media data acquisition unit 15 does not necessarily have to be executed at the same frequency as the generation of the viewpoint image in the viewpoint image generation unit 16. This is because video data cannot always be selected with fine time granularity. Therefore, it is preferable that the media data acquisition unit 15 selects not only the media data necessary for generating the viewpoint image at the current time but also the media data that is likely to be required at a near time. For example, media data related to scene components necessary for synthesizing a viewpoint image at a position near the user's viewpoint position at the current time is also acquired.

If the tracking range can be referred from the tracking system information, more preferable media data can be obtained by referring to the tracking range. Specifically, the media data necessary for synthesizing the viewpoint image of the viewpoint position within the tracking range in the vicinity of the user's viewpoint position is acquired. Since the situation of synthesizing the viewpoint video outside the tracking range does not occur, the transmission amount of the media data can be reduced without deteriorating the quality of the viewpoint video experienced by the user.

Further, when a specific degree of freedom of tracking is restricted by referring to the tracking type or tracking function information, more preferable media data can be acquired by using the restriction. Specifically, the media data necessary for the viewpoint image composition of the viewpoint when the displacement or rotation is in the vicinity of the user viewpoint position and the degree of freedom is not limited is acquired. In this case as well, the amount of media data transmission can be reduced without degrading the quality of the viewpoint image that the user perceives.

(Viewpoint video drawing unit 16)
The viewpoint image generation unit 16 generates and outputs a viewpoint image by referring to a scene to be drawn, a user's viewpoint position, and media data. The viewpoint image drawing process at a specific time t is executed by the following procedure.

(S1) When media data corresponding to the scene background is input, the image corresponding to the observation result from the user's viewpoint of the background in the drawing target space is synthesized and recorded in the buffer by referring to the media data and the user's viewpoint. do.

(S2) Execute the processing of S2-1 for each media data corresponding to the 3D object. (S2-1) Refer to the target media data and the user's viewpoint, and from the user's viewpoint of the corresponding 3D object in the drawing target space. The image corresponding to the observation result of is synthesized and recorded in the buffer.

(S3) The buffers generated in S1 and S2-1 are integrated into one buffer and output as a viewpoint image. Buffer composition is also performed by referring to the depth.

In the above processing, the processing amount of the viewpoint image drawing process is limited to the media data selected by the media data acquisition unit 15 and input to the viewpoint image generation unit 16 instead of all the media data included in the scene. Is reduced. If the scene background observed from the user's viewpoint and the image corresponding to the observation result of the 3D object are combined in the drawing target space, they may be combined by another procedure. For example, instead of drawing a 3D object in multiple buffers and then combining them into one buffer, you can apply the procedure of overwriting the 3D object while referring to the depth of the 3D object in one buffer.

As described above, the three-dimensional data reproduction device according to the present embodiment is a three-dimensional data transmission device that reproduces three-dimensional data, and is a scene arrangement unit that determines a scene arrangement based on scene information and tracking information, or a scene arrangement unit. , A configuration including a viewpoint setting unit for setting a user viewpoint based on scene information and tracking information. According to the above configuration, it is possible to realize a three-dimensional data reproduction device that synthesizes and outputs a suitable viewpoint image even when various 3D scenes are reproduced by using various tracking systems.

[System including the three-dimensional data reproduction device of the first embodiment]
FIG. 2 is a block diagram showing a configuration of a three-dimensional data distribution system realized by including the three-dimensional data reproduction device of the first embodiment.

The three-dimensional data distribution system includes a three-dimensional data reproduction device 1, a three-dimensional data generation device 2, a three-dimensional data recording device 3, a three-dimensional data display device 5, and a tracking system 6.

The three-dimensional data generation device 2 generates and outputs three-dimensional data by inputting a three-dimensional scene to be distributed. The three-dimensional data generated by the three-dimensional data generation device 2 includes the scene information and the media data described in the description of the three-dimensional data reproduction device 1 of the first embodiment.

The three-dimensional data recording device 3 stores three-dimensional data and outputs scene information and media data at a specified time in response to an external request.

Transmission channel 4 represents a channel on which three-dimensional data is transmitted. For example, an internet network line can be used as the transmission channel 4.

The three-dimensional data display device 5 displays the input viewpoint image and presents it to the user. As the display device, for example, a two-dimensional flat display, a three-dimensional display, or a head-mounted display can be used.

The tracking system 6 generates and outputs tracking information. The tracking information is the same as the tracking information described in the description of the first embodiment.

The three-dimensional data distribution system shown in FIG. 2 is a viewpoint using the three-dimensional data generation device 2 that generates three-dimensional data including scene information, the tracking system 6 that generates tracking information, and the three-dimensional data and tracking information. It includes a three-dimensional data playback device 1 that generates and outputs video. According to the above configuration, by referring to both the scene information and the tracking information, even when various 3D scenes and various tracking systems are used, a suitable viewpoint image can be generated and presented to the user.

[Modification 1 of Embodiment 1]
The three-dimensional data reproduction device 1 described in the first embodiment has a 3D scene arrangement unit 13 and a viewpoint setting unit 14.
Although both are included, it is also possible to include only the viewpoint setting unit 14. Specifically, the 3D scene arrangement unit 13 is deleted from the 3D data reproduction device 1. As the scene arrangement output from the 3D scene arrangement unit 13, the scene arrangement included in the 3D scene information is directly used. In this case, the 3D data playback device cannot support various 3D scenes, but it is possible to output a viewpoint image suitable for users who use various tracking systems as in the case of the 3D data playback device 1. It can be realized with a smaller circuit scale.

[Example of implementation by software]
The control block (particularly the 3D scene arrangement unit, the viewpoint setting unit, the media data acquisition unit, and the viewpoint image drawing unit) of the 3D data reproduction device 1 is provided by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. It may be realized or it may be realized by software.

In the latter case, the three-dimensional data reproduction device includes a computer that executes instructions of a program that is software that realizes each function. This computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium in which the program is stored. Then, in the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium", for example, a ROM (Read Only Memory) or the like, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the above program may be further provided. Further, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the claims. That is, an embodiment obtained by combining technical means appropriately modified within the scope of the claims is also included in the technical scope of the present invention.

1 3D data playback device
11 Tracking information acquisition department
12 3D scene information acquisition department
13 3D scene placement section
14 Viewpoint setting section
15 Media data acquisition department
16 Viewpoint video drawing section
2 3D data generator
3 3D data recording device
5 3D data display device
6 Tracking system

Claims (10)

It is provided with a scene arrangement unit that determines the scene arrangement based on the scene information and the tracking information, or a viewpoint setting unit that sets the user viewpoint based on the scene information and the tracking information, and the tracking information includes tracking system information. A three-dimensional data playback device. The three-dimensional data reproduction device according to claim 1, wherein the tracking system information includes a tracking system type. The tracking system information is characterized by including the presence / absence of a position tracking function, the presence / absence of a direction tracking function, a direction tracking range, a position tracking range, a tracking interval, and tracking function information including at least one or more of tracking delays. The three-dimensional data reproduction device according to claim 2. The three-dimensional data reproduction device according to claim 3, wherein the presence / absence of the direction tracking function is represented by a set of presence / absence of the tracking function for each degree of freedom indicating a direction. The three-dimensional data reproduction device according to claim 3, wherein the presence / absence of the position tracking function is represented by a set of presence / absence of the tracking function for each degree of freedom representing the position. The three-dimensional data reproduction device according to claim 3, wherein the viewpoint setting unit determines the height of the viewpoint in the drawing target space from the reference plane based on the tracking type or the tracking function information. The viewpoint setting unit identifies whether or not position tracking is valid based on the tracking type or tracking function information, and determines the user viewpoint by different methods depending on whether position tracking is enabled or disabled. Item 3. The three-dimensional data reproduction device according to item 3. The three-dimensional data reproduction device according to claim 7, wherein the viewpoint setting unit sets a viewpoint position using a scene range or a scene observation range included in the scene information when position tracking is effective. The three-dimensional aspect according to claim 3, wherein the viewpoint setting unit determines a user viewpoint by using a tracking range included in the tracking information and a scene range or a scene observation range included in the scene information. Data playback device. From claim 1, the scene information includes scene arrangement assisting information including at least one or more of a scene range, a scene observation range, a scene principal direction, a scene scale, and a scene reference ground information. The three-dimensional data reproduction device according to any one of claims 9.

Images