WO2022249536A1 - Information processing device and information processing method - Google Patents

Abstract

An information processing device according to one embodiment of the present technology comprises a rendering unit, an estimation unit, and a generation unit. The rendering unit executes a rendering process on three-dimensional spatial data constituting a virtual space, on the basis of visual field information about the visual field of a user, thereby generating two-dimensional video data corresponding to the visual field of the user. The estimation unit estimates a recognition position at which the user recognizes a recognition target object recognized by the user, in an outside-the-visual-field area not included in the visual field of the user in the virtual space. The generation unit generates a saliency map expressing saliency in the outside-the-visual-field area, on the basis of the estimated recognition position of the recognition target object in the outside-the-visual-field area.

Description

Information processing device and information processing method

The present technology relates to an information processing device and an information processing method applicable to VR (Virtual Reality) video distribution and the like.

In recent years, omnidirectional video that is captured by an omnidirectional camera or the like and that allows users to look around in all directions has come to be distributed as VR video. Furthermore, recently, a viewer (user) can look around in all directions (freely select the line-of-sight direction) and can move freely in three-dimensional space (freely select the viewpoint position). ) Technology for distributing 6DoF (Degree of Freedom) video (also referred to as 6DoF content) is being developed.
Such 6DoF content dynamically reproduces a three-dimensional space with one or more three-dimensional objects according to the viewer's viewpoint position, line-of-sight direction, and viewing angle (viewing range) at each time. be.
In such video distribution, it is required to dynamically adjust (render) video data to be presented to the viewer according to the viewing range of the viewer. For example, as an example of such technology, the technology disclosed in Patent Document 1 can be given.

Non-Patent Document 1 describes saliency map estimation processing for an omnidirectional image.
In this estimation processing, planar images in various camera directions are extracted from the omnidirectional image, and a saliency map for each planar image is estimated by a saliency map estimation model for planar images. The saliency maps for each planar image are integrated and horizontally biased toward the horizontal in the center of the image to estimate the saliency map of the omnidirectional image.

Japanese Patent Publication No. 2007-520925

Takao Yamanaka, "Technology for estimating conspicuous places in images: Saliency map estimation using deep learning", [online], September 15, 2020, New Technology Briefing 2020, Internet <URL: https ://shingi.jst.go.jp/var/rev0/0001/1222/02_sophia_yamanaka.pdf＞

The distribution of virtual images (virtual images) such as VR images is expected to spread, and there is a demand for technology that enables the distribution of high-quality virtual images.

In view of the circumstances as described above, the purpose of the present technology is to provide an information processing device and an information processing method capable of realizing high-quality virtual video distribution.

To achieve the above object, an information processing apparatus according to an aspect of the present technology includes a rendering unit, an estimation unit, and a generation unit.
The rendering unit generates two-dimensional video data according to the user's field of view by executing rendering processing on three-dimensional space data forming a virtual space based on field-of-view information about the user's field of view.
The estimation unit estimates a recognition position recognized by the user of a recognition target object recognized by the user in an area outside the user's visual field in the virtual space.
The generating unit generates a saliency map representing saliency in the out-of-field region based on the estimated recognition position of the recognition target object in the out-of-field region.

In this information processing device, the recognition position of the recognition target object in the out-of-field area is estimated. A saliency map in the out-of-view region is generated based on the estimated recognition position. This makes it possible to generate a high-precision saliency map in the out-of-field region, and to use the saliency map to deliver high-quality virtual video.

The estimation unit may set, as the recognition target object, an object that has been rendered before the current time.

The two-dimensional video data may be composed of a plurality of frame images that are continuous in time series. In this case, for the recognition target object that is not included in the frame image at the current time, the estimation unit performs the virtual space corresponding to the position of the recognition target object in the most recent past frame image that includes the recognition target object. The recognition position may be estimated based on the position within.

The estimation unit may estimate a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image as the recognition position.

The estimation unit estimates, as the recognition position, a position shifted in the moving direction of the recognition target object from a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image. good too.

For the recognition target object that is not included in the frame image at the current time, the estimation unit estimates a position in the virtual space where the sound is generated when determining that the user has recognized a sound emitted by the recognition target object. , may be estimated as the recognition position.

The three-dimensional space data may include three-dimensional space description data defining the configuration of the virtual space and three-dimensional object data defining a three-dimensional object in the virtual space. In this case, the three-dimensional space description data may include role information representing a role of the object to be recognized, and fixed position information representing a fixed position related to the role. Further, the estimating unit determines that, for the recognition target object set with predetermined role information that is not included in the frame image at the current time, the recognition target object set with the same role information was rendered by the current time. , the home position associated with the role may be inferred as the recognition position.

The estimating unit determines, if the recognition target object for which the same role information is set has been rendered at the fixed position related to the role by the current time, may be estimated as the recognition position.

The three-dimensional space data may include three-dimensional space description data defining the configuration of the virtual space and three-dimensional object data defining a three-dimensional object in the virtual space. In this case, the three-dimensional space description data may include role information representing a role of the object to be recognized, and fixed position information representing a fixed position related to the role. For the recognition target object set with predetermined role information that is not included in the frame image at the current time, the estimation unit estimates the sound emitted by the recognition target object set with the same role information by the current time. The fixed position associated with the role may be estimated as the recognized position when it is determined that the user has recognized it.

If the estimation unit determines that the user has recognized a sound emitted while the recognition target object, for which the same role information is set, is in the fixed position related to the role by the current time, may be estimated as the recognition position.

The estimation unit may estimate the recognition position based on the position of the recognition target object in the two-dimensional video data in which the recognition target object is rendered.

The generation unit may generate the saliency map in which saliency based on bottom-up attention in the out-of-field area is zero.

The generation unit may generate the saliency map representing saliency based on top-down attention in the out-of-field area based on the recognition position of the recognition target object in the out-of-field area.

The generation unit may generate the saliency map in the out-of-field region and a saliency map representing saliency of the two-dimensional video data.

The information processing device may further include a prediction unit that generates the future visual field information as predicted visual field information based on the saliency map. In this case, the rendering section may generate the two-dimensional image data based on the predicted field-of-view information.

The field-of-view information may include at least one of a viewpoint position, a line-of-sight direction, a line-of-sight rotation angle, a position of the user's head, or a rotation angle of the user's head.

The field-of-view information may include the rotation angle of the user's head. In this case, the prediction unit may predict the future head rotation angle of the user based on the saliency map.

The two-dimensional video data may be composed of a plurality of frame images that are continuous in time series. In this case, the rendering section may generate a frame image based on the predicted field-of-view information and output it as a predicted frame image.

An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, in which rendering processing is performed on three-dimensional space data forming a virtual space based on visual field information regarding a user's visual field. to generate two-dimensional image data according to the user's field of view.
A recognition position recognized by the user of a recognition target object recognized by the user is estimated in an area outside the user's field of view in the virtual space.
A saliency map representing saliency in the out-of-field region is generated based on the estimated recognition position of the recognition target object in the out-of-field region.

1 is a schematic diagram showing a basic configuration example of a server-side rendering system; FIG. FIG. 4 is a schematic diagram for explaining an example of a virtual video viewable by a user; FIG. 4 is a schematic diagram for explaining rendering processing; 1 is a schematic diagram showing a configuration example of a server-side rendering system; FIG. FIG. 4 is a schematic diagram for explaining a recognition target object and recognition positions; 4 is a flow chart showing an example of rendering video generation; FIG. 7 is a diagram for explaining the flowchart shown in FIG. 6 and is a schematic diagram showing timings of acquisition and generation of each information. FIG. 4 is a schematic diagram showing an example of generating a saliency map based on bottom-up attention; FIG. 5 is a schematic diagram for explaining a problem of the omnidirectional saliency map of the comparative example; FIG. 10 is a schematic diagram showing an example of information described in a scene description file used as scene description information in Example 2; FIG. 11 is a schematic diagram showing an example of information described in a scene description file used as scene description information in Example 3; FIG. 11 is a schematic diagram showing an example of information described in a scene description file used as scene description information in Example 3; FIG. 11 is a schematic diagram for explaining estimation of a recognition position of a recognition target object in Example 3; FIG. 11 is a schematic diagram for explaining estimation of a recognition position of a recognition target object in Example 3; 7 is a flow chart showing an example of estimating a recognition position of a recognition target object; 10 is a flow chart showing an example of generating an omnidirectional saliency map; FIG. 4 is a schematic diagram showing an example of an omnidirectional saliency map; 1 is a block diagram showing a hardware configuration example of a computer (information processing device) that can implement a server device and a client device; FIG.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

[Server-side rendering system]
A server-side rendering system is configured as an embodiment according to the present technology. First, a basic configuration example and a basic operation example of a server-side rendering system will be described with reference to FIGS. 1 to 3. FIG.
FIG. 1 is a schematic diagram showing a basic configuration example of a server-side rendering system.
FIG. 2 is a schematic diagram for explaining an example of a virtual video viewable by a user.
FIG. 3 is a schematic diagram for explaining rendering processing.
Note that the server-side rendering system can also be called a server-rendering media distribution system.

As shown in FIG. 1, the server-side rendering system 1 includes an HMD (Head Mounted Display) 2, a client device 3, and a server device 4.
HMD 2 is a device used to display virtual images to user 5 . The HMD 2 is worn on the head of the user 5 and used.
For example, when VR video is distributed as virtual video, an immersive HMD 2 configured to cover the field of view of the user 5 is used.
When an AR (Augmented Reality) video is distributed as a virtual video, AR glasses or the like are used as the HMD 2 .
A device other than the HMD 2 may be used as a device for providing the user 5 with virtual images. For example, a virtual image may be displayed on a display provided in a television, a smartphone, a tablet terminal, a PC (Personal Computer), or the like.

As shown in FIG. 2, in this embodiment, 6DoF images are provided as VR images to a user 5 wearing an immersive HMD 2 .
The user 5 can view the video in a range of 360 degrees around the front, back, left, right, and up and down in the virtual space S that is a three-dimensional space. For example, the user 5 freely moves the position of the viewpoint, the line-of-sight direction, etc. in the virtual space S, and freely changes the visual field (visual field range) 7 of the user. The image 8 displayed to the user 5 is switched according to the change in the field of view 7 of the user 5 . The user 5 can view the surroundings in the virtual space S with the same feeling as in the real world by performing actions such as changing the direction of the face, tilting the face, and looking back.
As described above, the server-side rendering system 1 according to the present embodiment can distribute photorealistic free-viewpoint video, and can provide a viewing experience at a free-viewpoint position.

In this embodiment, the HMD 2 acquires field-of-view information.
The visual field information is information about the visual field 7 of the user 5 . Specifically, the field-of-view information includes any information that can specify the field-of-view 7 of the user 5 within the virtual space S. FIG.
For example, the visual field information includes the position of the viewpoint, the line-of-sight direction, the rotation angle of the line of sight, and the like. The visual field information includes the position of the user's 5 head, the rotation angle of the user's 5 head, and the like.
The rotation angle of the line of sight can be defined, for example, by a rotation angle around an axis extending in the line of sight direction. Further, the rotation angle of the head of the user 5 can be defined by a roll angle, a pitch angle, and a yaw angle when the three mutually orthogonal axes set with respect to the head are the roll axis, the pitch axis, and the yaw axis. It is possible.
For example, let the axis extending in the front direction of the face be the roll axis. When the face of the user 5 is viewed from the front, the axis extending in the horizontal direction is defined as the pitch axis, and the axis extending in the vertical direction is defined as the yaw axis. The roll angle, pitch angle, and yaw angle with respect to these roll axis, pitch axis, and yaw axis are calculated as the rotation angle of the head. Note that it is also possible to use the direction of the roll axis as the direction of the line of sight.
In addition, any information that can specify the field of view of the user 5 may be used. As the field-of-view information, one of the information exemplified above may be used, or a plurality of pieces of information may be combined and used.

The method of acquiring visual field information is not limited. For example, it is possible to acquire visual field information based on the detection result (sensing result) by the sensor device (including the camera) provided in the HMD 2 .
For example, the HMD 2 is provided with a camera and a distance measuring sensor whose detection range is around the user 5, an inward facing camera capable of imaging the left and right eyes of the user 5, and the like. Also, the HMD 2 is provided with an IMU (Inertial Measurement Unit) sensor and a GPS.
For example, the position information of the HMD 2 acquired by GPS can be used as the viewpoint position of the user 5 and the position of the user's 5 head. Of course, the positions of the left and right eyes of the user 5 may be calculated in more detail.
It is also possible to detect the line-of-sight direction from the captured images of the left and right eyes of the user 5 .
It is also possible to detect the rotation angle of the line of sight and the rotation angle of the head of the user 5 from the detection result of the IMU.

Further, self-position estimation of the user 5 (HMD 2 ) may be performed based on the detection result by the sensor device provided in the HMD 2 . For example, by estimating the self-position, it is possible to calculate the position information of the HMD 2 and the attitude information such as which direction the HMD 2 faces. View information can be obtained from the position information and orientation information.
Algorithms for estimating the self-position of the HMD 2 are also not limited, and any algorithms such as SLAM (Simultaneous Localization and Mapping) may be used.
Moreover, head tracking for detecting the movement of the head of the user 5 and eye tracking for detecting the movement of the user's 5 right and left line of sight may be performed.

In addition, any device or any algorithm may be used to acquire the field-of-view information. For example, when a smartphone or the like is used as a device for displaying a virtual image to the user 5, the face (head) or the like of the user 5 may be captured, and the visual field information may be obtained based on the captured image. .
Alternatively, a device including a camera, an IMU, or the like may be worn around the head or eyes of the user 5 .
Any machine learning algorithm using, for example, a DNN (Deep Neural Network) or the like may be used to generate the visual field information. For example, by using AI (artificial intelligence) or the like that performs deep learning, it is possible to improve the generation accuracy of view information.
Note that application of machine learning algorithms may be performed for any of the processes within this disclosure.

The HMD 2 and the client device 3 are connected so as to be able to communicate with each other. The form of communication for communicably connecting both devices is not limited, and any communication technique may be used. For example, it is possible to use wireless network communication such as WiFi, short-range wireless communication such as Bluetooth (registered trademark), and the like.
The HMD 2 transmits the field-of-view information to the client device 3 .
Note that the HMD 2 and the client device 3 may be configured integrally. That is, the functions of the client device 3 may be installed in the HMD 2 .

The client device 3 and the server device 4 have hardware necessary for computer configuration, such as CPU, ROM, RAM, and HDD (see FIG. 18). The information processing method according to the present technology is executed by the CPU loading the program according to the present technology prerecorded in the ROM or the like into the RAM and executing the program.
For example, the client device 3 and the server device 4 can be implemented by any computer such as a PC (Personal Computer). Of course, hardware such as FPGA and ASIC may be used.
Of course, the client device 3 and the server device 4 are not limited to having the same configuration.

The client device 3 and the server device 4 are communicably connected via a network 9 .
The network 9 is constructed by, for example, the Internet, a wide area communication network, or the like. In addition, any WAN (Wide Area Network), LAN (Local Area Network), or the like may be used, and the protocol for constructing the network 9 is not limited.

The client device 3 receives the field-of-view information transmitted from the HMD 2 . The client device 3 also transmits the field-of-view information to the server device 4 via the network 9 .

The server device 4 receives the field-of-view information transmitted from the client device 3 . In addition, the server device 4 renders two-dimensional video data (rendered video) corresponding to the field of view 7 of the user 5 by executing rendering processing on the three-dimensional space data that constitutes the virtual space S based on the field-of-view information. Generate.
The server device 4 corresponds to an embodiment of an information processing device according to the present technology. An embodiment of an information processing method according to the present technology is executed by the server device 4 .

As shown in FIG. 3, the 3D spatial data includes scene description information and 3D object data.
The scene description information corresponds to three-dimensional space description data defining the configuration of the virtual space S (three-dimensional space). The scene description information includes various metadata for reproducing each scene of 6DoF content.
The three-dimensional object data is data defining a three-dimensional object in the virtual space S (three-dimensional space). That is, it becomes the data of each object that constitutes each scene of the 6DoF content.
For example, data of three-dimensional objects such as people and animals, and data of three-dimensional objects such as buildings and trees are stored. Alternatively, data of a three-dimensional object such as the sky or the sea that constitutes the background or the like is stored. A plurality of types of objects may be collectively configured as one three-dimensional object, and the data thereof may be stored.
The three-dimensional object data is composed of, for example, mesh data that can be expressed as polyhedral shape data and texture data that is data to be applied to the faces of the mesh data. Alternatively, it consists of a set of points (point cloud) (Point Cloud).

As shown in FIG. 3, the server device 4 reproduces the virtual space S that constitutes each scene by arranging the three-dimensional objects in the three-dimensional space based on the scene description information.
As shown in FIG. 3, an XYZ coordinate system is set in the virtual space S, and a three-dimensional object is placed at a position defined by the coordinate values. The coordinate values correspond to positional information in the virtual space S, and can also be called world coordinates. A method for setting the XYZ coordinate system in the virtual space S is not limited, and any setting method may be adopted.
Using the reproduced virtual space S as a reference, the image viewed by the user 5 is clipped (rendering processing) to generate a rendered image, which is a two-dimensional image viewed by the user 5 .
The server device 4 encodes the generated rendered video and transmits it to the client device 3 via the network 9 .
Note that the rendered image corresponding to the user's field of view 7 can also be said to be the image of the viewport (display area) corresponding to the user's field of view 7 .

The client device 3 decodes the encoded rendered video transmitted from the server device 4 . Also, the client device 3 transmits the decoded rendered video to the HMD 2 .
As shown in FIG. 2 , the HMD 2 reproduces the rendered video and displays it to the user 5 . The image 8 displayed to the user 5 by the HMD 2 may be hereinafter referred to as a rendered image 8 .

[Advantages of server-side rendering system]
Another delivery system for 6DoF video as illustrated in FIG. 2 is a client-side rendering system.
In the client-side rendering system, the client device 3 executes rendering processing on the three-dimensional space data based on the field-of-view information to generate two-dimensional video data (rendering video 8). A client-side rendering system can also be referred to as a client-rendered media delivery system.
In the client-side rendering system, it is necessary to deliver 3D space data (3D space description data and 3D object data) from the server device 4 to the client device 3 .
The three-dimensional object data is composed of mesh data or point cloud data. Therefore, the amount of data distributed from the server device 4 to the client device 3 becomes enormous. In addition, the client device 3 is required to have a considerably high processing capacity in order to execute rendering processing.

On the other hand, in the server-side rendering system 1 according to this embodiment, the rendered image 8 after rendering is distributed to the client device 3 . This makes it possible to sufficiently suppress the amount of distribution data. That is, it is possible to allow the user 5 to experience a 6DoF image in a large space composed of a huge amount of three-dimensional object data with a small amount of distribution data.
In addition, the processing load on the client device 3 side can be offloaded to the server device 4 side, and even when the client device 3 with low processing capability is used, the user 5 can experience 6DoF video. becomes.

[Response delay problem]
In the server-side rendering system 1 , visual field information of the user 5 and rendered video 8 after rendering are transmitted and received via the network 9 . Therefore, there is a possibility that a response delay will occur in displaying the rendered image 8 according to the movement of the viewpoint.
For example, the user 5 changes the field of view 7 by an action such as moving the head. View information is acquired by the HMD 2 and transmitted to the client device 3 . The client device 3 transmits the received field-of-view information to the server device 4 via the network 9 .
The server device 4 executes rendering processing on the three-dimensional space data based on the received field-of-view information of the user 5 to generate a rendered image 8 . The generated rendered image 8 is encoded and transmitted to the client device 3 via the network 9 .
The client device 3 decodes the received rendered image 8 and transmits it to the HMD 2 . The HMD 2 displays the received rendered image 8 to the user 5 .
The server-side rendering system 1 is constructed so as to execute such a processing flow in real time in accordance with changes in the field of view of the user 5 . In this case, there is a possibility that a delay from when the user 5 changes the field of view until the change is reflected in the image of the HMD 2 occurs as a response delay.
Note that this response delay can also be expressed as (Motion-to-Photon Latency: T_m2p). It is desirable that the delay time of this response delay be kept within 20 msec, which is the limit of human perception.

This technique is a very effective technique for solving the above problem of response delay. Hereinafter, an embodiment of the server-side rendering system 1 to which the present technology is applied will be described in detail.
In the following embodiments, a case in which Head Motion information is used as the visual field information of the user 5 will be taken as an example.
The Head Motion information includes Position information (X, Y, Z) representing the positional movement of the head of the user 5 and Orientation information (yaw, pitch, roll) representing the rotational movement of the head of the user 5. .
Position information (X, Y, Z) corresponds to position information in the virtual space S and is defined by coordinate values (world coordinates) of the XYZ coordinate system set in the virtual space S.
Orientation information (yaw, pitch, roll) is defined by roll, pitch, and yaw angles with respect to the mutually orthogonal roll, pitch, and yaw axes set on the head of the user 5 .
Of course, application of the present technology is not limited to the case where Head Motion information (X, Y, Z, yaw, pitch, roll) is used as the user's 5 visual field information. The present technology can be applied even when other information is used as the field-of-view information.

Further, in the following embodiments, the server-side rendering system 1 acquires the field-of-view information of the user 5 in real time, and displays a rendered image to the user 5 .
The time at which the visual field information of the user 5 is acquired by the server-side rendering system 1 will be described as "current time". That is, the time at which the visual field information of the user 5 is acquired by the HMD 2 will be described as the "current time".
As described above, the visual field information acquired at the "current time" is transmitted to the server device 4, the rendered image 8 is generated, and a response delay (T_m2p time) may occur until the HMD 2 displays it. have a nature.
By applying this technology, it is possible to sufficiently suppress the problem of response delays from the "current time", realizing high-quality virtual video distribution.

FIG. 4 is a schematic diagram showing a configuration example of the server-side rendering system 1 according to an embodiment of the present technology.
A server-side rendering system 1 shown in FIG. 4 includes an HMD 2 , a client device 3 and a server device 4 .
HMD2 can acquire the user's 5 visual field information (Head Motion information) in real time. As described above, the time when the Head Motion information is acquired by the HMD 2 is the current time.
The HMD 2 acquires Head Motion information and transmits it to the client device 3 at a predetermined frame rate. Therefore, the "head motion information at the current time" is repeatedly transmitted to the client device 3 at a predetermined frame rate.
Similarly, the “head motion information at the current time” is repeatedly transmitted from the client device 3 to the server device 4 at a predetermined frame rate.

The frame rate for obtaining Head Motion information (the number of times Head Motion information is obtained/second) is set so as to synchronize with the frame rate of the rendering video 8, for example.
For example, the rendered image 8 is composed of a plurality of frame images that are continuous in time series. Each frame image is generated at a predetermined frame rate. The frame rate for Head Motion information acquisition is set so as to synchronize with the frame rate of this rendered image 8 . Of course, it is not limited to this.
Also, as described above, AR glasses or a display may be used as a device for displaying virtual images to the user 5 .

The server device 4 has a data input unit 11 , a head motion information recording unit 12 , a prediction unit 13 , a rendering unit 14 , an encoding unit 15 and a communication unit 16 . The server device 4 also has a saliency map generator 17 , a saliency map recorder 18 , and a recognition position estimator 19 .
These functional blocks are implemented, for example, by the CPU executing the program according to the present technology, and the information processing method according to the present embodiment is executed. In order to implement each functional block, dedicated hardware such as an IC (integrated circuit) may be used as appropriate.

The data input unit 11 reads 3D space data (scene description information and 3D object data) and outputs it to the rendering unit 14 .
Note that the three-dimensional space data is stored, for example, in the storage unit 68 (see FIG. 18) within the server device 4 . Alternatively, the three-dimensional spatial data may be managed by a content server or the like communicably connected to the server device 4 . In this case, the data input unit 11 acquires three-dimensional spatial data by accessing the content server.

The communication unit 16 is a module for performing network communication, short-range wireless communication, etc. with other devices. For example, a wireless LAN module such as WiFi and a communication module such as Bluetooth (registered trademark) are provided.
In this embodiment, communication with the client device 3 via the network 9 is realized by the communication unit 16 .

The head motion information recording unit 12 records the visual field information (head motion information) received from the client device 3 via the communication unit 16 in the storage unit 68 (see FIG. 18). For example, a buffer or the like for recording field-of-view information (Head Motion information) may be configured.
The “head motion information at the current time” transmitted at a predetermined frame rate is accumulated and held in the storage unit 68 .

The prediction unit 13 generates future visual field information as predicted visual field information based on an omnidirectional saliency map (omnidirectional saliency map). In this embodiment, the future Head Motion information of the user 5 is predicted and generated as predicted Head Motion information.
The predicted Head Motion information includes future Position information (X, Y, Z) and future Orientation information (yaw, pitch, roll). That is, in this embodiment, the position of the head and the rotation angle of the head are predicted based on the omnidirectional saliency map.

The rendering unit 14 executes rendering processing illustrated in FIG. That is, the rendering image 8 corresponding to the user's 5 visual field 7 is generated by executing the rendering process on the three-dimensional space data based on the visual field information regarding the user's 5 visual field.
In this embodiment, the rendering unit 14 generates frame images forming the rendered video 8 based on the predicted view information (predicted Head Motion information) generated by the prediction unit 13 . A frame image generated based on the predicted Head Motion information is hereinafter referred to as a predicted frame image 20 .
The rendering unit 14 includes, for example, a reproduction unit that reproduces the virtual space S, a renderer, a parameter setting unit that sets rendering parameters, and the like. Rendering parameters include a resolution map that indicates the resolution of each area.
In addition, any configuration may be adopted as the rendering unit 14 .

The encoding unit 15 performs encoding processing (compression encoding) on the rendered video 8 (predicted frame image 20) to generate distribution data. The distribution data is transmitted to the client device 3 via the communication section 16 .
For example, the encoding process is executed in real time for each area of the rendered video 8 (predicted frame image 20) based on the QP map (quantization parameter).
More specifically, in the present embodiment, the encoding unit 15 switches the quantization precision (QP: Quantization Parameter) within the predicted frame image 20 for each region, so that the points of interest and important points in the predicted frame image 20 are Image quality deterioration due to area compression can be suppressed.
By doing so, it is possible to suppress an increase in distribution data and processing load while maintaining sufficient video quality for areas important to the user 5 . Here, the QP value is a value that indicates the step of quantization in lossy compression efficiency. The higher the QP value, the smaller the encoding amount, the higher the compression efficiency, and the lower the image quality deterioration due to compression. On the other hand, if the QP value is low, the coding amount is large, the compression efficiency is low, and image quality deterioration due to compression can be suppressed.
In addition, any compression encoding technique may be used.
The encoding unit 15 is composed of, for example, an encoder, a parameter setting unit for setting encoding parameters, and the like. Encoding parameters include the above-described QP map and the like.
For example, a QP map is generated based on the resolution map set by the parameter setting section of the rendering section 14 . In addition, any configuration may be adopted as the encoding unit 15 .

The saliency map generator 17 generates an omnidirectional saliency map.
The saliency map generation unit 17 generates not only a saliency map for the field of view representing the saliency of the rendered video (two-dimensional video data) 8 viewed by the user 5, A saliency map is also generated.
The saliency map for the field of view is information that is quantitatively expressed by estimating how easily each pixel of the rendered image 8 attracts attention from the mechanism of human visual attention.
The saliency map in the out-of-field region can be generated as a map in which the saliency of each pixel is calculated when the out-of-field region of the user 5 is expressed as a 2D image.
The generation of the omnidirectional saliency map will be described in detail later. A saliency map is also called a saliency map. In addition, the full sky can also be said to be a full celestial sphere.

The saliency map recording unit 18 records the omnidirectional saliency map generated by the saliency map generating unit 17 in the storage unit 68 (see FIG. 18). For example, a buffer or the like for recording the omnidirectional saliency map may be configured.

The recognition position estimating unit 19 estimates the recognition position of the recognition target object recognized by the user 5 in an area outside the visual field 7 of the user 5 in the virtual space S, which is recognized by the user 5 .
A recognition target object is an object that is assumed to be recognized by the user 5 among objects in the 6DoF content that the user 5 is viewing.
For example, an object that the user 5 has viewed by the current time is set as a recognition target object. That is, an object that has been rendered as a rendering target by the current time is set as a recognition target object.
In addition, an object that has not been viewed by the current time but whose sound is recognized by the user may be set as a recognition target object. For example, when the volume of a sound emitted by an object is greater than a predetermined reference value (threshold), the object that emitted the sound may be regarded as being recognized by the user 5 and may be set as a recognition target object. In this case, the type or content of the sound may be used as a condition for determining whether or not to set the recognition target object.

The recognition position is defined by XYZ coordinate values (world coordinates) on the virtual space S shown in FIG. The position where the user 5 would recognize in his brain that the recognition target object is now is estimated as the recognition position.
Therefore, the recognition position does not necessarily match the position where the recognition target object actually exists in the 6DoF content.
The recognized position can also be said to be a grasped position grasped by the user 5 .

FIG. 5 is a schematic diagram for explaining a recognition target object and recognition positions.
In FIG. 5, in order to make the explanation easier to understand, the image SP for the whole sky of the virtual space S is represented by a horizontally elongated image. In practice, the omnidirectional image SP is often stored as an equirectangular image.
Also, in the following description, the user's field of view 7 to be predicted will simply be described as the user's field of view 7 . Also, the predicted frame image 20 will be described as the frame image 20 .

In the scene shown in FIG. 5, in a virtual space S, three persons P1 to P3 and each object of a blinking lighting device L are arranged. In addition, tree, grass, road, and building objects are also arranged.
At the timing shown in FIG. 5A, it is assumed that the field of view 7 of the user 5 is directed toward the person P1 on the right side. Therefore, a frame image 20 including the person P1 is rendered. At this point, the person P1 is set as the recognition target object. Then, the position (world coordinates) in the virtual space S where the person P1 is arranged is estimated as the recognition position.
In this way, it is possible to estimate the recognition position based on the position of the recognition target object in the two-dimensional image data in which the recognition target object is rendered.

At the timing shown in FIG. 5A, the persons P2 and P3 on the left side and the object of the blinking illumination device L on the right side are arranged in the outside-of-view area 21 that is not included in the user's 5 field of view 7 . Therefore, the persons P2 and P3 and the lighting device L are not rendered. As a result, although the persons P2 and P3 and the lighting device L are arranged in the virtual space S, it is determined that they are not recognized by the user 5, and they are not set as recognition target objects.

Assume that the field of view 7 of the user 5 is turned to the left at the timing shown in FIG. 5B. Persons P2 and P3 are included in the field of view 7 of user 5 and rendered. As a result, the persons P2 and P3 are set as recognition target objects. Then, the positions (world coordinates) in the virtual space S where the persons P2 and P3 are arranged are estimated as the recognition positions.
The person P1 is out of the field of view 7 of the user 5 and is located in the out-of-view area 21. However, since the person P1 is an object that has been rendered in the past, the setting as the recognition target object is maintained.
Since lighting device L is not rendered, it is not set as a recognition target object.

Assume that the person P1 moves toward the illumination device L in the outside-of-view area 21 at the timing shown in FIG. 5C. In this case, in the virtual space S, the position of the person P1 moves.
Here, it is assumed that the user 5 does not recognize the movement of the person P1 in the out-of-field area 21 . In this case, the recognized position of the person P1 in the brain of the user 5 does not match the actual position of the person P1.
The recognition position estimating unit 19 can highly accurately estimate the recognition position in the brain of the user 5 with respect to the movement of the recognition target object in the out-of-field area 21 as shown in FIG. 5C.

In this embodiment, the rendering unit 14 functions as an embodiment of a rendering unit according to the present technology.
The recognition position estimation unit 19 functions as an embodiment of an estimation unit according to the present technology.
The saliency map generator 17 functions as an embodiment of a generator according to the present technology.
The prediction unit 13 functions as an embodiment of a prediction unit according to the present technology.

The client device 3 has a communication section 23 , a decoding section 24 and a rendering section 25 .
These functional blocks are implemented, for example, by the CPU executing the program according to the present technology, and the information processing method according to the present embodiment is executed. In order to implement each functional block, dedicated hardware such as an IC (integrated circuit) may be used as appropriate.

The communication unit 23 is a module for performing network communication, short-range wireless communication, etc. with other devices. For example, a wireless LAN module such as WiFi and a communication module such as Bluetooth (registered trademark) are provided.
The decoding unit 24 executes decoding processing on the distribution data. As a result, the encoded rendered video 8 (predicted frame image 20) is decoded.
The rendering unit 25 executes rendering processing so that the decoded rendered image 8 (predicted frame image 20) can be displayed by the HMD 2. FIG.

[Prediction Accuracy of Head Motion Information]
For example, the server device 4 that has received the "Current Time Head Motion Information" generates future predicted Head Motion information for the response delay (T_m2p time). A predicted frame image 20 is generated based on the predicted Head Motion information and displayed to the user 5 by the HMD 2 .
If the predicted Head Motion information can be generated with very high accuracy, it will be possible to display the rendering image 8 according to the user's 5 field of view 7 in the future for the response delay (T_m2p time) from the "current time", which is the problem of response delay. is sufficiently suppressible.

In order to improve the accuracy of the predicted Head Motion information, the inventor has studied Head Motion prediction.
First, the prediction error of Head Motion prediction tends to increase as the frequency of the head motion signal (sensoring result) increases.
Due to the characteristics of the human body, movements in the rotational direction are capable of rapid changes (movements with high frequency), but in positional movements such as forward/backward, up/down, and left/right, there is a tendency that high-frequency movements with abrupt changes are difficult. It is in.
Therefore, of these two types of motion, the prediction error for motion (X, Y, Z) toward positional movement is low, and the impact on viewing is very small. On the other hand, there is a tendency for the prediction error for motion in the rotational direction (yaw, pitch, roll) to increase, which tends to affect viewing. In other words, it is very important to improve the prediction accuracy for motions in the rotational direction (yaw, pitch, roll).

The present inventor focused on the omnidirectional saliency map in order to improve the accuracy of head motion prediction, especially for motion in the rotational direction (yaw, pitch, roll).
By generating an omnidirectional saliency map with high accuracy and using it for head motion prediction, it is possible to perform prediction accuracy for motion in the rotational direction (yaw, pitch, roll) with extremely high accuracy.

[Generation operation of two-dimensional video data (rendering video)]
An operation example of generating a rendered image using the omnidirectional saliency map by the server device 4 will be described.
FIG. 6 is a flow chart showing an example of rendering video generation.
FIG. 7 is a diagram for explaining the flowchart shown in FIG. 6, and shows the timing of acquisition of Head Motion information, generation of predicted Head Motion information, generation of predicted frame image 20, and generation of the omnidirectional saliency map. It is a schematic diagram.
In this embodiment, in order to make the explanation easier to understand, the field of view information is acquired from the client device 3 at a predetermined frame rate, and the predicted Head Motion information, the predicted frame image 20, and the omnidirectional conspicuous image are obtained at the same frame rate. Assume that each of the gender maps is generated. Of course, the processing is not limited to such processing.
A numbered frame shown in FIG. 7 indicates a frame of each process. FIG. 7 schematically shows the 1st frame to the 25th frame where the processing is started.
In each frame, a frame with a square graphic represents that the data described on the left side has been acquired/generated. Also, the numbers in the square figures indicate which frame the data corresponds to.

First, how much future predicted Head Motion information is to be generated from the "current time" is set.
In this embodiment, the communication unit 16 measures the network delay with the client device 3 and identifies the estimated time of the target (step 101). That is, the response delay (T_m2p time) is measured and T_m2p time is specified as the predicted time.
In this embodiment, head motion information in a frame a predetermined number of frames later than the frame corresponding to the "current time" is predicted and generated as predicted head motion information.
As the predetermined number of frames, the number of frames corresponding to T_m2p time, which is the prediction time, is set.
For example, in this embodiment, it is assumed that Head Motion information five frames ahead is predicted. For example, when the "head motion information at the current time" is acquired in the tenth frame, the head motion information of the fifteenth frame, which is five frames ahead, is predicted and generated as predicted head motion information. Of course, the specific number of frames is not limited and may be set arbitrarily.

The communication unit 16 acquires Head Motion information from the client device 3 (step 102). As shown in FIG. 7, Head Motion information is acquired at a predetermined frame rate from the first frame. The Head Motion information acquired in each frame is used as is as the data corresponding to that frame.

The prediction unit 13 determines whether or not the amount of head motion information required for prediction of the head motion information has accumulated (step 103).
In this embodiment, it is assumed that 10 frames of Head Motion information are required to predict Head Motion information. Of course, the specific number of frames is not limited and may be set arbitrarily.
For example, from the first frame to the ninth frame, the amount of head motion information necessary for predicting the head motion information is not accumulated, so the result in step 103 is No and the process returns to step 102 . Therefore, generation of rendering image 8 (prediction frame image 20) is not executed until the tenth frame.
When the head motion information of the 10th frame is obtained, it is determined that the amount of head motion information required for prediction of the head motion information has accumulated, and the result of step 103 is Yes, and the process proceeds to step 104 .

At step 104, the prediction unit 13 determines whether or not the omnidirectional saliency map corresponding to the "head motion information at the current time" acquired at step 102 has already been generated.
In this embodiment, the historical information of the visual field information (head motion information) up to the current time and the omnidirectional saliency map corresponding to the current time are input, and the predicted visual field information (predicted head motion information) is generated. .
The omnidirectional saliency map corresponding to the current time is the omnidirectional saliency map generated in the past. Specifically, the saliency map for the visual field representing the saliency of the predicted frame image 20 generated based on the predicted visual field information (predicted Head Motion information) predicted in the past, and the predicted visual field information predicted in the past. It is an omnidirectional saliency map including a saliency map in an out-of-field region not included in the field of view 7 (predicted field of view) of the user 5 based on (predicted Head Motion information).

The omnidirectional saliency map corresponding to the "head motion information at the current time" means the omnidirectional saliency map corresponding to the frame from which the "head motion information at the current time" is acquired in the example shown in FIG. do.
That is, if the numbers in the squares representing the Head Motion information and the numbers in the squares representing the omnidirectional saliency map are equal to each other, the corresponding "current time Head Motion information ” and the omnidirectional saliency map.

For example, when the Head Motion information of the 10th frame is acquired, the frame corresponding to the current time is the 10th frame. In step 104, it is determined whether or not an omnidirectional saliency map corresponding to 10 frames (a omnidirectional saliency map represented by a square figure with the number 10 written therein) has been generated. .
As shown in FIG. 7, up to the 10th frame, the predicted Head Motion information has not yet been generated, and the predicted frame image 20 has not yet been generated. Therefore, since the omnidirectional saliency map has not been generated, step 104 results in No, and the process proceeds to step 105 .

In step 105, the prediction section 13 generates predicted visual field information (predicted Head Motion information) based on history information of visual field information (Head Motion information) up to the current time.
In this way, if the omnidirectional saliency map of the frame corresponding to the current time has not been generated, the predicted Head Motion information may be generated based only on the history information of the Head Motion information up to the current time. .
In this embodiment, at frame 10, based on the history information of the head motion information from frame 1 to frame 10, future predicted head motion information for the next five frames is generated. Therefore, as shown in FIG. 7, in the 10th frame, predicted Head Motion information corresponding to 15 frames five frames in the future is generated (predicted Head Motion information represented by a square figure with the number 15 written therein). information).
A specific algorithm for generating predicted Head Motion information based on history information of Head Motion information up to the current time is not limited, and any algorithm may be used. For example, any machine learning algorithm may be used.

The rendering unit 14 executes the rendering process illustrated in FIG. 3 based on the predicted Head Motion information to generate the rendered video 8 (predicted frame image 20) (step 106). In this embodiment, a predicted frame image 20 corresponding to 15 frames is generated based on future predicted Head Motion information five frames ahead.

The recognition position estimation unit 19 estimates the recognition position of the recognition target object in the out-of-field area (step 107). In the present embodiment, in a field outside the field of view 7 (predicted field of view) of the user 5 based on the predicted field of view information (predicted Head Motion information), the recognition target object recognized by the user 5 is estimated.

The saliency map generator 17 generates an omnidirectional saliency map corresponding to 15 frames based on the predicted frame image 20 and the estimated recognition position (step 108).
The generated omnidirectional saliency map is recorded and held by the saliency map recording unit 18 . As illustrated in FIG. 7, in the 10th frame, the omnidirectional saliency map corresponding to the 15th frame is recorded.

Thus, in this embodiment, as shown in FIG. 7, a frame image five frames ahead is generated as the predicted frame image 20 in the frame corresponding to the "current time". Also, in the frame corresponding to the "current time", an omnidirectional saliency map for the next five frames is generated.
In the present disclosure, a frame image generated at the "current time", that is, a frame image generated in a frame corresponding to the "current time" is referred to as a "frame image at the current time". Therefore, in this embodiment, the future predicted frame image 20 generated in the frame corresponding to the "current time" corresponds to the "current time frame image".
On the other hand, the “frame image (predicted frame image) corresponding to the current time” corresponds to a frame image (predicted frame image) generated five frames in the past.

The predictive frame image 20 is encoded by the encoding unit 15 . The communication unit 16 also transmits the encoded predicted frame image 20 to the client device 3 (step 109).
The predicted frame image 20 generated in the tenth frame is transmitted to the HMD 2 via the client device 3 and displayed to the user 5 as the first frame of the 6DoF video content. As a result, distribution of virtual video is started in which the influence of response delay is sufficiently suppressed.
The rendering unit 14 determines whether or not the processing for all frame images has been completed (step 110). Here, it is assumed that processing is executed up to frame 25, as illustrated in FIG.
Therefore, step 110 becomes No and the process returns to step 102 .

From frame 11 to frame 14 shown in FIG. 7, step 104 is No, and the processing flow from step 105 to step 106 is executed.
At frame 15, the omnidirectional saliency map corresponding to frame 15 generated in past frame 10 exists as the omnidirectional saliency map corresponding to the acquired "head motion information at the current time". Therefore, step 104 becomes Yes and the process proceeds to step 111 .

In step 111, the history information of the visual field information (Head Motion information) up to the current time and the omnidirectional saliency map corresponding to the current time are input, and future Head Motion information is predicted as predicted Head Motion information. generated.
A specific algorithm for generating predicted Head Motion information by inputting historical information of Head Motion information and an omnidirectional saliency map is not limited, and any algorithm may be used. For example, any machine learning algorithm may be used.
From then on, until frame 25, step 104 is Yes and the omnidirectional saliency map is used to generate highly accurate predicted Head Motion information.
If processing for all frame images is completed, step 109 becomes Yes, and video generation and distribution processing are completed.

[Discussion on generation of omnidirectional saliency map]
The inventor of the present invention has repeatedly considered the generation of the omnidirectional saliency map.
Human visual attention consists of extrinsic attention due to visual stimuli before recognizing an object (bottom-up attention) and intrinsic attention due to curiosity and curiosity about an object after recognizing an object (top-down attention). ). The keyword salience is used in both bottom-up and top-down attention.

Examples of generating saliency maps based on bottom-up attention include brightness, color, direction, direction of movement, depth, etc. that induce extrinsic attention (bottom-up attention) by visual stimuli before humans recognize objects. are extracted from the input video (2D image). A final saliency map is generated by calculating each feature map so as to assign a high saliency to an area in which the value indicating each feature value is significantly different from the surroundings, and integrating them.

FIG. 8 is a schematic diagram illustrating an example of bottom-up attention-based saliency map generation that can be performed by the system.
In the example shown in FIG. 8, a predicted frame image 20 is input as an input frame.
A feature amount extraction process is performed on the predicted frame image 20, and each feature amount of luminance, color, and direction that attracts bottom-up attention is extracted. Note that the predicted frame image 20 or the like of the previous frame may be used for feature extraction.
A feature image is generated by converting the feature amount into luminance for each feature amount of luminance, color, and direction, and a Gaussian pyramid of the feature image is generated.
Also, a depth map and a motion vector map image are acquired as parameters (rendering information) related to rendering processing from the renderer that configures the rendering unit 14 . A depth map is data including distance information (depth information) to an object to be rendered. A motion vector map image is data containing motion information of an object to be rendered.
A depth map image is used as a depth feature image to generate a Gaussian pyramid. Also, the motion vector map image is used as a motion direction feature image to generate a Gaussian pyramid.
Center-surround difference processing is performed on the Gaussian pyramid of each feature. As a result, a feature map is generated for each feature amount of brightness, color, direction, motion direction, and depth. A saliency map 22 based on bottom-up attention is generated by integrating the feature maps of these feature amounts.
Specific algorithms for feature quantity extraction processing, Gaussian pyramid generation processing, center-surround difference processing, and feature map integration processing for each feature quantity are not limited. For example, each process can be implemented using a well-known technique.

The depth map image obtained from the renderer is not the depth values estimated by performing 2D image analysis or the like on the predicted frame image 20, but the accurate values obtained in the rendering process. Therefore, by directly receiving the depth map image from the renderer and using it as feature information of "depth" for generating the saliency map 22, it is possible to generate the saliency map 22 with high precision and accuracy.
The motion vector map image obtained from the renderer is not the values estimated by performing 2D image analysis or the like on the predicted frame image 20, but the accurate values obtained in the rendering process. Therefore, by directly receiving this depth map image from the renderer and using it for generating the saliency map 22 as the feature information of the "movement direction", it is possible to generate a more accurate and more accurate saliency map.
It should be noted that other feature quantities such as "brightness" and "color" can also be calculated in the rendering process and used as rendering information.
The algorithm for generating a bottom-up attention-based saliency map is not limited and any other algorithm may be used. For example, any machine learning algorithm may be used.
For example, by performing 2D image analysis or the like on the predicted frame image 20, feature information such as "depth" and "motion direction" may be acquired and used.

Saliency is given to objects because top-down attention is directed after object recognition and as attention based on its meaning.
For example, an object such as a human face that is generally likely to attract people's interest is detected from an image and saliency is added. In addition, based on the type of object (idol, baseball player, vehicle, etc.), the importance of the object in 6DoF content, the user's preference for the object (degree of interest or preference, etc.), etc. Salience is given.
The algorithm for generating a top-down attention-based saliency map is not limited and any algorithm may be used. For example, any machine learning algorithm may be used.

It is also possible to collectively generate a saliency map including saliency based on bottom-up attention and salience based on top-down attention. For example, by actually capturing human gaze data, creating training data based on it, and making it learn, we build a machine learning model that can generate a saliency map that includes bottom-up attention to top-down attention. is also possible.

[Method of generating an all-dome saliency map as a comparative example]
Here, a method of generating an omnidirectional saliency map will be described as a comparative example.
A 2D image is rendered for each viewport while changing the viewport for the entire virtual space S at regular intervals (hereinafter, this rendered 2D image is referred to as a viewport image). Then, for each viewport image, a saliency map based on bottom-up attention and a saliency map based on top-down attention as described above are generated and integrated.
The omnidirectional saliency map of this comparative example provides the user 5 with information representing the saliency of all viewport images. That is, for the user 5, the information represents the conspicuity when all areas within the virtual area S are within the field of view.

Therefore, if the omnidirectional saliency map of this comparative example is used, there is a high possibility that the following problems will occur.
FIG. 9 is a schematic diagram for explaining the problem of the omnidirectional saliency map 30 of the comparative example. In FIG. 9, saliency based on top-down attention generated by persons P1 to P3 and saliency based on bottom-up attention generated by lighting device L are schematically illustrated as white areas.

(1) The salience due to bottom-up attention is practically equal to zero with no visual stimulus because it is invisible outside the field of view of the user 5 . The omnidirectional saliency map 30 of the comparative example is created on the premise that all regions in the virtual space S are within the field of view. Therefore, if the omnidirectional saliency map 30 of the comparative example is used as it is for prediction of visual field information, saliency outside the field of view and unrecognizable by the user 5 adversely affects the prediction of visual field information.
For example, at the timing shown in FIG. 5A, the user 5 does not recognize the blinking lighting device L on the right side. In the omnidirectional saliency map 30 of the comparative example, as shown in FIG. 9A, saliency based on bottom-up attention is given to the pixel region of the lighting device L that is blinking. As a result, salience that does not exist in the brain of the user 5 is generated, which adversely affects prediction of visual field information.

(2) Even in the saliency caused by top-down attention, the saliency of an object that is outside the field of view of the user 5 and has not yet been captured (not viewed) in the viewport is not even aware of its existence. equal to zero. Using the comparative omnidirectional saliency map 30, saliency that is outside the field of view and unnoticeable to the user 5 adversely affects prediction of the field of view information.
For example, at the timing shown in FIG. 5A, the persons P2 and P3 on the left side are not recognized by the user 5 . In the omnidirectional saliency map 30 of the comparative example, as shown in FIG. 9A, saliency based on top-down caution is given to the persons P2 and P3. As a result, salience that does not exist in the brain of the user 5 is generated, which adversely affects prediction of visual field information.

(3) In addition, in terms of salience due to top-down attention, the object visually recognized in the past may be moving in the outside-of-field-of-view area 21 . In this case, the user 5 is unaware of the movement of the object. Therefore, the user should recognize that the object will be at a position corresponding to the situation grasped when the object was viewed last time.
For example, if an object were stationary, it would perceive that the object would be where it was last seen. Also, if the object was moving when it was last seen, it is thought that the object would be at a position some distance along the moving direction from the last seen position.
In the omnidirectional saliency map 30 of the comparative example, it is assumed that the moving object and the moved object are also within the field of view. gender is given.
For example, if the object is stationary when last seen, i.e. if the object has not moved since then, salience occurs at a position completely different from the perceived position in the brain (grasp position). do.
If the object was moving when the object was last seen, there is no problem if the perceived position (grasped position) where the object is supposed to be around here in the brain matches the actual position of the object. On the other hand, the possibility that the position expected by the user and the actual position of the object match is not necessarily high. Therefore, there is a high possibility that salience will occur at a position different from the recognition position (grasping position) in the brain.
For example, at the timing shown in FIG. 5C, the user 5 does not recognize the movement of the person P1. In the omnidirectional saliency map 30 of the comparative example, as shown in FIG. 9B, saliency based on top-down caution is given to the person P1 near the lighting device L1. As a result, salience that does not exist in the brain of the user 5 is generated, which adversely affects prediction of visual field information.
Thus, the omnidirectional saliency map 30 of the comparative example does not consider saliency outside the viewport (outside the field of view). Therefore, if the omnidirectional saliency map 30 of the comparative example is used as it is for Head Orientation prediction of where to direct the viewport from the current viewport to the next, the prediction accuracy will conversely decrease, or it will be useless. The purpose of improving prediction accuracy cannot be achieved.
In order to achieve high prediction accuracy, it is important to accurately reflect the actual attention of the user 5 to the inside and outside of the field of view.

[Estimation of Recognition Position of Recognition Target Object]
In this embodiment, the recognition position estimation unit 19 sets the recognition target object and estimates the recognition position of the recognition target object. These processes are executed for each frame.
Several examples of estimating the recognition position of the recognition target object will be described below.
(Example 1)
An object to be rendered by the rendering unit 14 is set as a recognition target object.
For example, at the timing shown in FIG. 5A, the person P1 is set as the recognition target object. At the timing shown in FIG. 5B, persons P1 and P2 are set as recognition target objects.

In each frame, the recognition position of each set recognition target object is estimated each time a "frame image at the current time" (future predicted frame image 20 generated at the current time) is generated.
For the recognition target object included in the "frame image at the current time", the recognition position is estimated based on the position in the virtual space S corresponding to the position of the recognition target object in the frame image.
For example, at the timing shown in FIG. 5A, the person P1 is included in the frame image (predicted frame image) 20, so based on the position in the virtual space S corresponding to the position of the person P1 in the frame image 20, the person P1 is estimated.

The position in the virtual space S corresponding to the position of the person P1 in the frame image 20 becomes the position of the person P1 arranged in the virtual space S when the frame image 20 is rendered. The position of the person P1 is estimated as the recognized position recognized by the user 5 .
If a state in which the person P1 is moving is rendered, that is, if the user 5 recognizes that the person P1 is moving, the virtual space S corresponding to the position of the person P1 in the frame image 20 is displayed. A position shifted along the moving direction from the position in the virtual space S, that is, the position of the person P1 arranged in the virtual space S, may be estimated as the recognition position recognized by the user 5 . The amount of shift may be appropriately set to an amount that the user 5 will predict the destination of movement.
Here, it is assumed that the position in the virtual space S corresponding to the position of the person P1 in the frame image 20 is directly estimated as the recognition position.

For example, at the timing shown in FIG. 5B, the positions in the virtual space S corresponding to the positions of the persons P2 and P3 in the frame image 20 are estimated as the recognition positions recognized by the user 5 with respect to the persons P2 and P3. be done.

For a recognition target object not included in the "frame image at the current time", based on the position in the virtual space S corresponding to the position of the recognition target object in the most recent frame image 20 in which the recognition target object is included , the recognition position is estimated. In this embodiment, the position in the virtual space S corresponding to the position of the recognition target object in the most recent frame image 20 is directly estimated as the recognition position.
Note that the most recent frame image in the past can also be said to be the last rendered frame image.

For example, at the timing shown in FIG. 5B, the frame image 20 does not include the person P1. For the person P1 not included in the frame image 20, the position in the virtual space S corresponding to the position of the person P1 in the frame image 20 at the timing shown in FIG. 5A is estimated as the recognition position.
At the timing shown in FIG. 5C, the person P1 is moving toward the illumination device L in the out-of-field area 21 . In this embodiment, the most recent frame image 20, that is, the position in the virtual space S corresponding to the position of the person P1 in the frame image 20 at the timing shown in FIG. 5A is maintained as the recognition position.
This makes it possible to suppress the occurrence of salience that is not in the brain of the user 5, and it is possible to predict the visual field information with high accuracy.

For example, each time the recognition target object is rendered, the position in the virtual space S corresponding to the position of the recognition target object in the rendered frame image 20 is updated and stored as the recognition position. As a result, for the recognition target object not included in the frame image 20 at the current time, the position in the virtual space S corresponding to the position of the recognition target object in the most recent frame image 20 is held as the recognition position. In this way, updating of the recognized position may be performed for each frame.

(Example 2)
Even if the object is out of sight, the user 5 may perceive the position of the object by listening to sounds such as footsteps and voices emitted from the object. Embodiment 2 is devised assuming such a case. FIG. 10 is a schematic diagram showing an example of information described in a scene description file used as scene description information in this embodiment.
In this embodiment, when generating 6DoF content, each object information described in the scene description file is associated with audio data such as footsteps and voice emitted from the object.
Based on this information, the renderer renders the associated audio data from each object position.

In the example shown in FIG. 10, the following information is stored as object information.
Name...Name of object Position...Position of object Url...Address of 3D object data Audio...Name of audio data of sound emitted from object In the example shown in FIG. information is stored.
Name: Name of audio data Url: Address of audio data

In the example shown in FIG. 10, in a hide-and-seek scene, audio data information of "voice and footsteps 1" and "voice and footsteps 2" are linked to video object information of "hidden person 1" and "hidden person 2". . The audio data of "Voice and footsteps 1" and "Voice and footsteps 2" are, for example, responses such as "It's okay" or "It's okay" to the demon's call "Is it okay?" is mentioned.

The recognition position estimating unit 19 receives, from the renderer, the presence/absence information of audio data linked to each object, the current volume information, and the like. Then, the user 5 currently hears the sound emitted from the object depending on whether or not it exceeds a reference value (threshold), which is the volume level used as a reference for judging that the user 5 can hear and recognize the sound. to determine whether it is aware of Note that the reference volume level may be set arbitrarily.
Assume that there is linked audio data and that the sound volume level exceeds the standard. In this case, the current position of the object within the virtual space S, that is, the position where the sound is generated within the virtual space S is estimated as the recognition position recognized by the user 5 .
That is, in the present embodiment, when it is determined that the user 5 has recognized the sound emitted by the recognition target object that is not included in the "frame image at the current time", the position where the sound is generated in the virtual space S is estimated as the recognition position.

For example, in a scene constructed based on the scene description file shown in FIG. 10, the recognized positions of people hiding are estimated based on the voices of people playing hide-and-seek. For example, it is assumed that "hidden person 1", which is not included in the frame image, utters a voice saying "that's bad". If the volume of the voice exceeds the reference value, it is determined that the voice was heard by the user 5, and the current position in the virtual space S of the "hidden person 1" is estimated as the recognition position.
This makes it possible to suppress the occurrence of saliency that is not in the brain of the user 5, and it is possible to predict visual field information with high accuracy.
Note that when the sound emitted from the recognition target object is interrupted and cannot be heard, the last heard position, that is, the most recently heard position in the past, is maintained as the recognition position.

(Example 3)
If the user 5 has viewed a scene similar to the scene currently viewed in the past, the user 5 may grasp the position of the object by associating it from the memory at that time. The present embodiment 3 is devised assuming such a case, and the recognition position in the brain of the user 5 is estimated based on the past viewing information in the same scene as the present.
11 and 12 are schematic diagrams showing an example of information described in a scene description file used as scene description information in this embodiment.
In this embodiment, when generating 6DoF content, the role information of each object in the current scene and the fixed position information (world coordinates) at the time of the role are stored in each object information described in the scene description file. be done.
That is, the scene description file stores role information representing the role of the object to be recognized and the fixed position (world coordinates) related to the role.

In the examples shown in FIGS. 11 and 12, the following information is stored as object information.
Name...Name of object Position...Position of object Url...Address of 3D object data Role...Role information FixedPos...Fixed position information about role

In the example shown in FIGS. 11 and 12, in a baseball scene, role information and fixed position information are stored for the offensive and defensive players "A Tadashi Ao" and "B Kawa Bsuke". ing. FIG. 11 is a scene description file for attack, and FIG. 12 is a scene description file for defense.
Each time the offense and defense take turns, the attack scene description file and the defense scene description file are updated with each other. Also, when the role of the object changes during attack or defense, each scene description file for attack and defense is updated.
For example, when the scene description file is updated from FIG. 11 to FIG. 12 according to the change of offense and defense, the role of "Ata Ao" changes from the role of "next batter" to the role of "first baseman". "B-gawa Bsuke" changes from the role of "batter" to the role of "pitcher".
The FixedPos of "batter", "next batter", "pitcher", and "first baseman" are assumed to be positions in the world coordinates for the following positions.
"Batter"...Batter's Box "Next Batter"...Next Batter "Pitcher"...Pitcher's Mound "First Baseman"...Position on First Base These FixedPos indicate the general position in the role, and Position and Position indicate the actual object position. is different.
Based on these pieces of information, the recognition position estimation unit 19 determines whether or not the user 5 has seen the same scene as the current scene in the past, and estimates the recognition position.

13 and 14 are schematic diagrams for explaining the estimation of the recognition position of the recognition target object in the third embodiment.
In FIGS. 13 and 14, a baseball stadium is constructed as the virtual space S, and the player 32 "Ao Ada" and the player 33 "Bsuke Bagawa" are also arranged in the virtual space S. As shown in FIG.
The line of sight of the eye in the drawing corresponds to the field of view (predicted field of view) 7 of the user 5, and a frame image (predicted frame image) 20 of the region of the field of view 7 is generated.

In FIG. 13A, first, user 5 is watching a scene in which "Ao Ada" player 32 is playing first base. In other words, the user 5 understands that the "A husband A" 32 is on first base in the defensive scene. Of course, the player 32 "A Tao A" is set as a recognition target object.
After that, due to the change of offense and defense, the player 32 of "A Dao A" becomes the "next batter" and moves to the next batter's circle. As shown in FIG. 13B, the user 5 follows and watches the "Ata Ao" player 32 moving to the next batter's circle.
After that, as shown in FIG. 14A, the user 5 directs the field of view 7 toward the player "Bsuke Bagawa" 33 who has become a "batter" and watches the batting. At this time, the object to be recognized, player A Tao 32, is present outside the field of view and is not included in the frame image 20. FIG.
After that, the offense and defense are changed, and the player 32 of "A Tao A" moves to the position of first base as a "first baseman". On the other hand, as shown in FIG. 14B, the user 5 faces the spectator seats with the field of view 7 and watches the spectators in the cheering seats. A player 32 of "A Tao A" moves to first base in an area outside the field of view of user 5, and user 5 does not see the movement.
The user 5 can grasp that the "A Den A Fu" player 32 is currently on the defense due to the offense-defense change. 13A in the past, he knows that the player 32 is on first base when he is on defense. Therefore, it is conceivable that the user 5 will associate the player 32 in the mind of the user 5 with the player 32 on first base, as in the case of viewing in FIG. 13A, and update the position in the mind.
In accordance with this update in the brain, the recognition position estimation unit 19 estimates the recognition position of the player 32 as the ``first base position'', which is a fixed position related to the role of ``first baseman''. . That is, in the viewing in FIG. 13A , the scene in which the role of player “A Field A” 32 is “first baseman” is viewed in the same way as now (“A Field A” player 32 is displayed in frame image 20). 14A) and the scene update from viewing in FIG. 14A to viewing in FIG. The recognition position in the brain of the user 5 of the "husband" player 32 is estimated to be the "first base position".
As described above, in the third embodiment, the role information of the recognition target object in the scene at the time of viewing (during rendering) in the past is held. is updated and there is a viewing experience during the role, then the recognition position is estimated at the home position of the role.

In the viewing in FIG. 14B, the player 32 "A Tao A" corresponds to a recognition target object for which predetermined role information ("first baseman") that is not included in the "frame image at the current time" is set.
Then, if the player 32 who has been rendered with the same role information (“first baseman”) has been rendered by the current time, the fixed position (“position of first baseman”) related to the role has been rendered. ) is estimated as the recognition position.
As a result, it is possible to suppress the occurrence of saliency that is not in the user's brain, and it is possible to predict visual field information with high accuracy.

It should be noted that viewing the scene in which the role of the player ``A Field A'' player 32 is the ``first baseman'' may be viewed in another baseball game in the past. In other words, not only the game that is currently being watched, but also other games that have been watched in the past, even if a scene in which the role of the "A man A" player 32 is "first baseman" is viewed, the "position of first base" is displayed. ” may be estimated as the recognition position.
That is, as long as the recognition target object with the same role information set is rendered by the current time, it is possible to estimate the fixed position related to the role as the recognition position.

In the past, when a state in which the player "A Tao A" 32 was at the fixed position "first base position" was rendered, the "first base position" may be estimated as the recognized position.
That is, until the current time, the recognition target object ("A man A" player 32) to whom the same role information ("first baseman") is set is in a fixed position ("first base position") related to the role. A home position associated with the role ("first base position") may be inferred as the perceived position if it has been rendered.
As a result, the recognition position can be estimated when the user surely grasps that the player "A Tao A" 32 is at the "first base position" during defense. On the other hand, most of the recognition target objects for which role information is set are in the "fixed position", so there is a high possibility that the state of being in the "fixed position" will be rendered.

It is also possible to integrate the processes of (Example 1) to (Example 3) to estimate the recognition position of the recognition target object. For example, (Example 1), (Example 2), and (Example 3) are prioritized and executed in this order.
First, the information with the highest priority is assumed to be visual information from the eyes, and (Embodiment 1) is executed. That is, the position where the user 5 seems to recognize the recognition target object with his/her eyes is estimated as the recognition position.
When the user 5 visually recognizes the recognition target object, the visually confirmed position is the recognition position of the recognition target object. The recognition position estimation unit 19 determines whether or not the user 5 is visually recognizing the recognition target object based on whether or not the recognition target object has been rendered in the frame image (prediction frame image) 20, and determines whether or not the recognition target object has been rendered. The position of the target object on the virtual space S is estimated as the recognition position.
In this case, since no other information is necessary for estimating the recognition position, the sound information used in (Example 2) and the role information and fixed position information used in (Example 3) are not acquired. . When the object to be recognized is out of the field of view 7 (that is, is no longer rendered), the sound information used in (Example 2) and the role information and fixed position information used in (Example 3) are acquired. In the absence of this information, the last viewed position (last rendered position) is maintained as the perceived position.

The information with the next highest priority is assumed to be sound information emitted from the object to be recognized, and (Embodiment 2) is executed. That is, the recognition position is estimated based on the presence/absence of audio data linked to the recognition target object and the current generation status information (occurrence position, volume information, etc.) of the audio data.
When the object to be recognized is out of the field of view and there is no visual information, the position where the object is supposed to be recognized by sound is estimated as the recognition position.

Role information and fixed position information are acquired as information with the next highest priority, and (Example 3) is executed. That is, the recognition position is estimated based on the past viewing experience information of the user 5 in the same scene as the current scene and the fixed position information of the recognition target object in that scene.
When neither visual information nor sound information is acquired, a position that seems to be associated with the position from the past viewing experience of the same scene as the present is estimated as the recognized position.
If there is no visual information, sound information, role information, or positional information, the user 5 is unaware of the existence of the object, so the user 5 pays zero attention to the object (the object to be recognized is set to no recognition position).

FIG. 15 is a flowchart showing an example of estimating the recognition position of the recognition target object. The process shown in FIG. 15 can also be said to be a process example in which the processes of (Example 1) to (Example 3) are integrated.
Rendering information and scene information for all objects to be recognized in the scene are obtained (step 201).
Rendering information includes any information regarding the rendering of the object to be recognized. Here, the rendering information includes history information of rendering of the recognition target object up to the current time, position information of the recognition target object in the frame image 20, and the like.
The scene information includes scene description information about the object to be recognized. For example, it includes history information of scene description information up to the current time.
Note that in step 301, an object that is rendered for the first time in the "frame image at the current time" is also set as a recognition target object, and rendering information and frame information are acquired.

It is determined whether or not there is an unprocessed object to be recognized (step 202).
If there is an unprocessed object to be recognized (Yes in step 202), one unprocessed object to be recognized is selected, and the processing from step 203 onwards is executed.

It is determined whether or not the selected object to be recognized is included in the "frame image at the current time" (that is, whether or not it has been rendered) (step 203). If the recognition target object is included in the "frame image at the current time" (Yes in step 203), the current role information of the recognition target object is added to the role list (step 204).
The role list is a list to which the role information is input when the recognition target object for which the role information is set has been viewed (that is, rendered) by the current time. If no role information is set for the object to be recognized, addition to the role list is not executed. The role list can also be said to be a role watched list.
A recognition position is estimated to the position of the current object to be recognized (step 205). Here, the current position of the object to be recognized corresponds to the position in the virtual space S corresponding to the position of the object to be recognized in the "frame image at the current time". For an object that is rendered for the first time in the "frame image at the current time", the initial recognition position is estimated in step 205 .
The recognition positions of the recognition target objects whose recognition positions have been estimated by the current time are updated. Of course, there may be cases where the result is the same as the recognition position estimated in the past.
The estimation of the recognition position of this recognition target object is completed, and the process returns to step 202 .

If the object to be recognized is not included in the "frame image at the current time" (No in step 203), audio data is associated with the object to be recognized and the sound volume at the time of current rendering exceeds the reference value. It is determined whether there is (step 206).
If step 206 is affirmative (Yes in step 206), the current role (role information) of the recognized object is added to the role list (step 204).
As described above, in the present embodiment, even if it is determined that the user 5 has recognized a sound emitted by a recognition target object for which the same role information is set by the current time, the recognition target object for which the same role information is set is rendered. Adding the role to the role list is performed as if
A recognition position is estimated to the position of the current object to be recognized (step 205). Here, the current position of the recognition target object corresponds to the position in the virtual space S where the sound emitted from the recognition target object is generated. The recognition positions of the recognition target objects whose estimated positions have been estimated by the current time are updated.
The estimation of the recognition position of this recognition target object is completed, and the process returns to step 202 .

If step 206 is negative (No in step 206), it is determined whether the current role of the object to be recognized has changed since the last time the recognition position was updated (step 207).
If the current role has not changed since the last time the recognized position was updated (No in step 207), the recognized position is not updated (ie, the recognized position has not changed). Then, return to step 202 .

If the current role has changed since the last time the recognition position was updated (Yes in step 207), it is determined whether or not the user 5 has viewed the scene of the recognition target object's current role in the past. (Step 208).
The determination of step 208 is performed by referring to whether the current role information of the object to be recognized is entered in the role list. If the role list is populated with the current role information of the object to be recognized, step 208 is affirmative. If the role list is not populated with current role information for the object to be recognized, step 208 is negative.

If the user 5 has not viewed the scene of the current role of the recognition target object in the past (No in step 208), the recognition position is not updated (that is, the recognition position is unchanged). Then, return to step 202 .
If the user 5 has viewed the scene of the current role of the recognition target object in the past (Yes in step 208), the recognition position is estimated to be the current position of the recognition target object (step 209).
Here, the position of the current object to be recognized corresponds to the home position associated with the role. The recognition positions of the recognition target objects whose estimated positions have been estimated in past frames by the current time are updated.
The estimation of the recognition position of this recognition target object is completed, and the process returns to step 202 .

If there is no unprocessed recognition target object (No in step 202), the process of estimating the recognition positions of all recognition target objects ends (step 210).
As shown in FIG. 15, by using visual information, current sound information, and past viewing information, it is possible to estimate the recognition position of the recognition target object with high accuracy.

In the estimation example shown in FIG. 15, for recognition target objects set with predetermined role information not included in the "frame image at the current time", there are no recognition target objects set with the same role information by the current time. When it was determined that the user 5 recognized the uttered sound, the home position associated with the role was estimated as the recognized position.
Alternatively, if it is determined that the user 5 has recognized a sound emitted while the recognition target object for which the same role information has been set up to the current time is in a fixed position related to the role, the fixed position related to the role is determined. A position may be estimated as a perceived position.

[Generation of omnidirectional saliency map]
Generation of the omnidirectional saliency map by the saliency map generation unit 17 will be described.
FIG. 16 is a flowchart illustrating an example of generating an omnidirectional saliency map.
First, in step 301, a saliency map for the visual field is generated based on the "frame image at the current time". In this embodiment, a saliency map including both saliency based on bottom-up attention and saliency based on top-down attention is generated as a saliency map for the field of view.
For example, saliency based on bottom-up attention and saliency based on top-down attention are each detected separately and added together to generate a saliency map for the field of view corresponding to the final viewport image. .
Also, in the same step 301, a sky omnidirectional saliency map (in this embodiment, an equirectangular image in which all pixels have a value of zero) is prepared, and a field segment saliency map is pasted.
This makes it possible to generate an omnidirectional saliency map so as not to generate saliency based on bottom-up attention in the out-of-field area. That is, it is possible to generate an omnidirectional saliency map in which the saliency based on bottom-up attention is zero in the out-of-field area. As a result, it is possible to prevent unnecessary saliency from occurring, and it is possible to solve the problem point (1) described above.
Any other method may be used as a method for avoiding occurrence of salience based on bottom-up attention in the out-of-field area. For example, once a saliency map for the whole sky (including both bottom-up attention-based saliency and top-down attention-based saliency) is generated, a method of masking the area outside the field of view is adopted. good too.
On the other hand, according to the method of pasting the saliency map for the field of view in the field of view area of the omnidirectional saliency map of the sky, as in the present embodiment, it is possible to reduce the processing load and shorten the processing time. It is also possible to shorten the length.

The recognition positions of all the recognition target objects in the out-of-field region estimated by the recognition position estimation unit 19 are obtained (step 302).
It is determined whether or not there is an object to be recognized (hereinafter referred to as an out-of-field object) in an unprocessed out-of-field area (step 303).
If there is an unprocessed out-of-view object (Yes in step 303), one unprocessed out-of-view object is selected, and the process of step 304 is executed.

In step 304, the position of the out-of-field object in the omnidirectional saliency map (position on the 2D map) is calculated based on the recognized position in the virtual space S of the out-of-field object. Top-down saliency generated by out-of-view objects is placed at the calculated positions.
Out-of-view objects are objects that have been rendered in the past. Therefore, out-of-view objects have previously been saliency detected in step 301 based on top-down attention. For example, the saliency value of each pixel along the shape of the object is detected.
In this embodiment, top-down attention-based salience for the recognized objects detected in step 301 is preserved. Then, at step 304, the retained top-down attention-based saliency (shape and value) is reused and placed on the omnidirectional saliency map.
This top-down attention-based saliency-based placement of out-of-view objects is complete and the process returns to step 303 .

In step 304, a method of generating a saliency map (top-down caution) for the entire omnidirectional area once and then adjusting the saliency occurrence position in the out-of-field region in accordance with the estimated recognition position is adopted. may be
On the other hand, by reusing the saliency based on the top-down attention detected in step 101 as in the present embodiment, the rendering process can be done only in the viewport, and the processing load can be reduced. . Also, the processing time can be shortened.

If there is no unprocessed out-of-view object (No in step 302), the generation process of the omnidirectional saliency map in which top-down attention-based saliency of the out-of-view object is generated based on the recognition position ends. do.

Top-down attention-based saliency is generated at the recognition position that the user 5 recognizes in the brain. That is, a saliency map representing saliency based on top-down attention in the out-of-field region is generated based on the recognition position of the recognition target object in the out-of-field region.
This makes it possible to prevent unnecessary salience from being generated from a position different from the recognition position in the brain, and to solve the above problems (2) and (3).

FIG. 17 is a schematic diagram showing an example of the omnidirectional saliency map generated by this embodiment. FIG. 17A is an example of the omnidirectional saliency map 35 generated at the timing shown in FIG. 5A. FIG. 17B is an example of the omnidirectional saliency map 35 generated at the timing shown in FIG. 5C.
As shown in FIG. 17A, in the omnidirectional saliency map 35 generated at the timing shown in FIG. 5A, saliency based on top-down attention of only the person P1 recognized by the user 5 occurs. No saliency based on the bottom-up attention of the lighting device L that is not recognized by the user 5 occurs. Also, saliency based on top-down attention of persons P1 and P2 who are not recognized by user 5 does not occur.
In addition, as shown in FIG. 17B, for the user 5 who does not recognize the movement of the person P1, salience based on the top-down attention of the person P1 occurs at the position of the person P1 before the movement. In addition, salience based on the bottom-up attention of the lighting device L that is not recognized by the user 5 does not occur.
As described above, the omnidirectional saliency map 35 generated according to the present embodiment avoids the occurrence of saliency that is not in the brain of the user 5, resulting in a highly accurate omnidirectional saliency map. ing.

As described above, in the server-side rendering system 1 according to the present embodiment, the recognition position of the recognition target object in the out-of-view area 21 is estimated. Based on the estimated recognition position, an all-dome saliency map 35 is generated, including a saliency map in the out-of-view region 21 .
This makes it possible to reflect the attention of the user 5 to the outside of the visual field in the omnidirectional saliency map 35 according to the viewing situation at that time. As a result, it is possible to generate a highly accurate omnidirectional saliency map 35 .
Since a highly accurate and accurate omnidirectional saliency map 35 is generated, it is possible to generate prediction Head Motion information (especially Orientation information) with extremely high accuracy, and the problem of response delay (T_m2p time) can be sufficiently resolved. can be suppressed to In other words, it is possible to use the omnidirectional saliency map 35 to deliver high-quality virtual video.
The highly accurate omnidirectional saliency map 35 generated in this embodiment can also be used for other purposes.

<Other embodiments>
The present technology is not limited to the embodiments described above, and various other embodiments can be implemented.

In the above, the case where 6DoF video is distributed as the virtual image is taken as an example. The present technology is not limited to this, and can be applied when 3DoF video, 2D video, or the like is distributed. Also, as the virtual image, instead of the VR video, an AR video or the like may be distributed.
In addition, the present technology can also be applied to stereo images (for example, right-eye images and left-eye images) for viewing 3D images.
The present technology is applicable to content displaying any virtual space in which an out-of-view area may occur. Further, the saliency map of the out-of-view area is not limited to the saliency map of the entire area of the virtual space, and a saliency map of a partial area of the virtual space that is the out-of-view area may be generated.

FIG. 18 is a block diagram showing a hardware configuration example of a computer (information processing device) 60 that can implement the server device 4 and the client device 3. As shown in FIG.
The computer 60 includes a CPU 61, a ROM (Read Only Memory) 62, a RAM 63, an input/output interface 65, and a bus 64 connecting them together. A display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and the like are connected to the input/output interface 65. FIG.
The display unit 66 is a display device using liquid crystal, EL, or the like, for example. The input unit 67 is, for example, a keyboard, pointing device, touch panel, or other operating device. If the input portion 67 includes a touch panel, the touch panel can be integrated with the display portion 66 .
The storage unit 68 is a non-volatile storage device such as an HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable recording medium 71 such as an optical recording medium or a magnetic recording tape.
The communication unit 69 is a modem, router, or other communication equipment for communicating with other devices that can be connected to a LAN, WAN, or the like. The communication unit 69 may use either wired or wireless communication. The communication unit 69 is often used separately from the computer 60 .
Information processing by the computer 60 having the hardware configuration as described above is realized by cooperation of software stored in the storage unit 68 or the ROM 62 or the like and the hardware resources of the computer 60 . Specifically, the information processing method according to the present technology is realized by loading a program constituting software stored in the ROM 62 or the like into the RAM 63 and executing the program.
The program is installed in the computer 60 via the recording medium 61, for example. Alternatively, the program may be installed on the computer 60 via a global network or the like. In addition, any computer-readable non-transitory storage medium may be used.

An information processing method and a program according to the present technology may be executed by a plurality of computers communicably connected via a network or the like to construct an information processing apparatus according to the present technology.
That is, the information processing method and program according to the present technology can be executed not only in a computer system configured by a single computer, but also in a computer system in which a plurality of computers work together.
In the present disclosure, a system means a set of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules within a single housing, are both systems.
The information processing method according to the present technology and the execution of the program by the computer system include, for example, acquisition of visual field information, execution of rendering processing, setting of recognition target objects, estimation of recognition positions, generation of omnidirectional saliency maps, etc. It includes both the case where it is executed by a single computer and the case where each process is executed by different computers. Execution of each process by a predetermined computer includes causing another computer to execute part or all of the process and obtaining the result.
That is, the information processing method and program according to the present technology can also be applied to a configuration of cloud computing in which a plurality of devices share and jointly process one function via a network.

Each configuration of the server-side rendering system, HMD, server device, client device, etc., and each processing flow, etc., which are described with reference to each drawing, are merely one embodiment, and can be arbitrarily modified within the scope of the present technology. It is possible. That is, any other configuration, algorithm, or the like for implementing the present technology may be employed.

In the present disclosure, terms such as “substantially”, “approximately”, and “approximately” are appropriately used to facilitate understanding of the description. On the other hand, there is no clear difference between the use and non-use of words such as "substantially", "approximately", and "approximately".
That is, in the present disclosure, “central,” “central,” “uniform,” “equal,” “identical,” “perpendicular,” “parallel,” “symmetric,” “extended,” “axial,” “cylindrical,” “cylindrical,” and “ring-shaped.” Concepts that define shape, size, positional relationship, state, etc. such as "annular shape" are "substantially centered", "substantially centered", "substantially uniform", "substantially equal", "substantially "substantially orthogonal""substantiallyparallel""substantiallysymmetrical""substantiallyextended""substantiallyaxial""substantiallycylindrical""substantiallycylindrical" The concept includes "substantially ring-shaped", "substantially torus-shaped", and the like.
For example, "perfectly centered", "perfectly centered", "perfectly uniform", "perfectly equal", "perfectly identical", "perfectly orthogonal", "perfectly parallel", "perfectly symmetrical", "perfectly extended", "perfectly Axial,""perfectlycylindrical,""perfectlycylindrical,""perfectlyring," and "perfectly annular", etc. be
Therefore, even when words such as "approximately", "approximately", and "approximately" are not added, concepts that can be expressed by adding so-called "approximately", "approximately", "approximately", etc. can be included. Conversely, states expressed by adding "nearly", "nearly", "approximately", etc. do not necessarily exclude complete states.

In the present disclosure, expressions using "more than" such as "greater than A" and "less than A" encompass both the concept including the case of being equivalent to A and the concept not including the case of being equivalent to A. is an expression contained in For example, "greater than A" is not limited to not including equal to A, but also includes "greater than or equal to A." Also, "less than A" is not limited to "less than A", but also includes "less than A".
When implementing the present technology, specific settings and the like may be appropriately adopted from concepts included in “greater than A” and “less than A” so as to exhibit the effects described above.

It is also possible to combine at least two characteristic portions among the characteristic portions according to the present technology described above. That is, various characteristic portions described in each embodiment may be combined arbitrarily without distinguishing between each embodiment. Moreover, the various effects described above are only examples and are not limited, and other effects may be exhibited.

Note that the present technology can also adopt the following configuration.
(1)
a rendering unit that generates two-dimensional video data corresponding to the user's field of view by executing rendering processing on three-dimensional space data that constitutes a virtual space based on field-of-view information related to the user's field of view;
an estimation unit for estimating a recognition position recognized by the user of a recognition target object recognized by the user in an area outside the user's field of view in the virtual space;
and a generation unit that generates a saliency map representing saliency in the out-of-field region based on the estimated recognition position of the recognition target object in the out-of-field region.
(2) The information processing device according to (1),
The information processing apparatus, wherein the estimation unit sets an object that has been rendered as a target for rendering up to a current time as the recognition target object.
(3) The information processing device according to (1) or (2),
The two-dimensional video data is composed of a plurality of frame images that are continuous in time series,
For the recognition target object not included in the frame image at the current time, the estimation unit determines a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image in which the recognition target object is included. An information processing device that estimates the recognition position based on.
(4) The information processing device according to (3),
The information processing device, wherein the estimation unit estimates a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image as the recognition position.
(5) The information processing device according to (3) or (4),
The estimation unit estimates, as the recognition position, a position shifted along the moving direction of the recognition target object from a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image. processing equipment.
(6) The information processing device according to any one of (3) to (5),
For the recognition target object that is not included in the frame image at the current time, the estimation unit estimates a position in the virtual space where the sound is generated when determining that the user has recognized a sound emitted by the recognition target object. , is estimated as the recognition position. Information processing device.
(7) The information processing device according to any one of (3) to (6),
The three-dimensional space data includes three-dimensional space description data defining the configuration of the virtual space and three-dimensional object data defining a three-dimensional object in the virtual space,
the three-dimensional space description data includes role information representing a role of the object to be recognized and positional information representing a fixed position associated with the role;
The estimating unit confirms that, for the recognition target object set with predetermined role information that is not included in the frame image at the current time, the recognition target object set with the same role information was rendered by the current time. In some cases, the information processing device estimates the home position associated with the role as the recognition position.
(8) The information processing device according to (7),
The estimating unit determines, if the recognition target object for which the same role information is set has been rendered at the fixed position related to the role by the current time, is estimated as the recognition position.
(9) The information processing device according to any one of (3) to (8),
The three-dimensional space data includes three-dimensional space description data defining the configuration of the virtual space and three-dimensional object data defining a three-dimensional object in the virtual space,
the three-dimensional space description data includes role information representing a role of the object to be recognized and positional information representing a fixed position associated with the role;
For the recognition target object set with predetermined role information that is not included in the frame image at the current time, the estimation unit detects the sound emitted by the recognition target object set with the same role information by the current time. an information processing apparatus that estimates the fixed position related to the role as the recognized position when it is determined that the position has been recognized.
(10) The information processing device according to (9),
If the estimation unit determines that the user has recognized a sound emitted while the recognition target object, for which the same role information is set, is in the fixed position related to the role by the current time, information processing apparatus that estimates the fixed position related to the position as the recognition position.
(11) The information processing device according to any one of (1) to (10),
The information processing apparatus, wherein the estimation unit estimates the recognition position based on the position of the recognition target object in the two-dimensional video data in which the recognition target object is rendered.
(12) The information processing device according to any one of (1) to (11),
The information processing apparatus, wherein the generation unit generates the saliency map in which saliency based on bottom-up attention in the out-of-field region is zero.
(13) The information processing device according to any one of (1) to (12),
The information processing apparatus, wherein the generation unit generates the saliency map representing saliency based on top-down attention in the out-of-field region based on the recognition position of the recognition target object in the out-of-field region.
(14) The information processing device according to any one of (1) to (13),
The information processing apparatus, wherein the generation unit generates the saliency map in the out-of-field region and a saliency map representing saliency of the two-dimensional video data.
(15) The information processing device according to any one of (1) to (14), further comprising:
A prediction unit that generates the future visual field information as predicted visual field information based on the saliency map,
The information processing apparatus, wherein the rendering unit generates the two-dimensional video data based on the predicted field-of-view information.
(16) The information processing device according to (15),
The information processing apparatus, wherein the visual field information includes at least one of a viewpoint position, a line-of-sight direction, a line-of-sight rotation angle, a position of the user's head, or a rotation angle of the user's head.
(17) The information processing device according to (16),
The field of view information includes a rotation angle of the user's head,
The prediction unit predicts a future head rotation angle of the user based on the saliency map. Information processing apparatus.
(18) The information processing device according to any one of (15) to (17),
The two-dimensional video data is composed of a plurality of frame images that are continuous in time series,
The information processing apparatus, wherein the rendering unit generates a frame image based on the predicted field-of-view information and outputs it as a predicted frame image.
(19)
generating two-dimensional video data corresponding to the user's field of view by performing rendering processing on three-dimensional space data constituting a virtual space based on field-of-view information related to the user's field of view;
estimating a recognition position recognized by the user of a recognition target object recognized by the user in an area outside the user's field of view in the virtual space;
An information processing method in which a computer system generates a saliency map representing saliency in the out-of-field region based on the estimated recognition position of the recognition target object in the out-of-field region.

S... Virtual space 1... Server side rendering system 2... HMD
3... Client device 4... Server device 5... User 7... User's visual field 8... Rendered image 13... Predictor 14... Renderer 17... Saliency map generator 18... Saliency map recorder 19... Recognized position estimator 20... Predicted frame image (frame image)
35... All-dome saliency map 60... Computer

Claims (19)

a rendering unit that generates two-dimensional video data according to the user's field of view by executing rendering processing on three-dimensional space data that constitutes a virtual space based on field-of-view information related to the user's field of view;
an estimating unit for estimating a recognition position recognized by the user of a recognition target object recognized by the user in an area outside the user's field of view in the virtual space;
and a generating unit that generates a saliency map representing saliency in the out-of-field region based on the estimated recognition position of the recognition target object in the out-of-field region. The information processing device according to claim 1,
The information processing apparatus, wherein the estimation unit sets an object that has been rendered as a target for rendering up to a current time as the recognition target object. The information processing device according to claim 1,
The two-dimensional video data is composed of a plurality of frame images that are continuous in time series,
For the recognition target object not included in the frame image at the current time, the estimation unit determines a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image in which the recognition target object is included. An information processing device that estimates the recognition position based on. The information processing device according to claim 3,
The information processing device, wherein the estimation unit estimates a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image as the recognition position. The information processing device according to claim 3,
The estimation unit estimates, as the recognition position, a position shifted along the moving direction of the recognition target object from a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image. processing equipment. The information processing device according to claim 3,
For the recognition target object that is not included in the frame image at the current time, the estimation unit estimates a position in the virtual space where the sound is generated when determining that the user has recognized a sound emitted by the recognition target object. , is estimated as the recognition position. Information processing device. The information processing device according to claim 3,
The three-dimensional space data includes three-dimensional space description data defining the configuration of the virtual space and three-dimensional object data defining a three-dimensional object in the virtual space,
the three-dimensional space description data includes role information representing a role of the object to be recognized and positional information representing a fixed position associated with the role;
The estimating unit confirms that, for the recognition target object set with predetermined role information that is not included in the frame image at the current time, the recognition target object set with the same role information was rendered by the current time. In some cases, the information processing device estimates the home position associated with the role as the recognition position. The information processing device according to claim 7,
The estimating unit determines, if the recognition target object for which the same role information is set has been rendered at the fixed position related to the role by the current time, is estimated as the recognition position. The information processing device according to claim 3,
The three-dimensional space data includes three-dimensional space description data defining the configuration of the virtual space and three-dimensional object data defining a three-dimensional object in the virtual space,
the three-dimensional space description data includes role information representing a role of the object to be recognized and positional information representing a fixed position associated with the role;
For the recognition target object set with predetermined role information that is not included in the frame image at the current time, the estimation unit detects the sound emitted by the recognition target object set with the same role information by the current time. an information processing apparatus that estimates the fixed position related to the role as the recognized position when it is determined that the position has been recognized. The information processing device according to claim 9,
If the estimation unit determines that the user has recognized a sound emitted while the recognition target object, for which the same role information is set, is in the fixed position related to the role by the current time, information processing apparatus that estimates the fixed position related to the position as the recognition position. The information processing device according to claim 1,
The information processing apparatus, wherein the estimation unit estimates the recognition position based on the position of the recognition target object in the two-dimensional video data in which the recognition target object is rendered. The information processing device according to claim 1,
The information processing apparatus, wherein the generation unit generates the saliency map in which saliency based on bottom-up attention in the out-of-field region is zero. The information processing device according to claim 1,
The information processing apparatus, wherein the generation unit generates the saliency map representing saliency based on top-down attention in the out-of-field area based on the recognition position of the recognition target object in the out-of-field area. The information processing device according to claim 1,
The information processing apparatus, wherein the generation unit generates the saliency map in the out-of-field region and a saliency map representing saliency of the two-dimensional video data. The information processing apparatus according to claim 1, further comprising:
A prediction unit that generates the future visual field information as predicted visual field information based on the saliency map,
The information processing apparatus, wherein the rendering unit generates the two-dimensional video data based on the predicted field-of-view information. The information processing device according to claim 15,
The information processing apparatus, wherein the visual field information includes at least one of a viewpoint position, a line-of-sight direction, a line-of-sight rotation angle, a position of the user's head, or a rotation angle of the user's head. The information processing device according to claim 16,
The field of view information includes a rotation angle of the user's head,
The information processing apparatus, wherein the prediction unit predicts a future head rotation angle of the user based on the saliency map. The information processing device according to claim 15,
The two-dimensional video data is composed of a plurality of frame images that are continuous in time series,
The information processing apparatus, wherein the rendering unit generates a frame image based on the predicted field-of-view information and outputs it as a predicted frame image. generating two-dimensional image data corresponding to the user's field of view by performing rendering processing on three-dimensional space data constituting a virtual space based on field-of-view information related to the user's field of view;
estimating a recognition position recognized by the user of a recognition target object recognized by the user in an area outside the user's field of view in the virtual space;
An information processing method in which a computer system generates a saliency map representing saliency in the out-of-field region based on the estimated recognition position of the recognition target object in the out-of-field region.

Images