WO2020045141A1

WO2020045141A1 - Information processing device, information processing method, and program

Info

Publication number: WO2020045141A1
Application number: PCT/JP2019/032260
Authority: WO
Inventors: 京二郎永野; 富士夫荒井; 秀憲青木; 靖子石原; 新太郎筒井
Original assignee: ソニー株式会社
Priority date: 2018-08-31
Filing date: 2019-08-19
Publication date: 2020-03-05

Abstract

[Problem] To provide an information processing device, information processing method, and program with which it is possible to reduce a processing load. [Solution] An information processing device is equipped with a control unit. The control unit: determines the degree of noticeability of each of first surrounding information in a first region and second surrounding information in a second region in an actual space acquired by a sensor, the degree of noticeability differing from the view field information corresponding to the view field of a user; and determines, based on the degree of noticeability, the degree of priority of a first process relating to a first virtual object disposed in the first region, and a second process relating to a second virtual object disposed in the second region.

Description

Information processing apparatus, information processing method, and program

The present technology relates to an information processing apparatus, an information processing method, and a program related to processing relating to a virtual object displayed on a transmission type head mounted display or the like.

Patent Document 1 describes a system that shares a virtual reality space between a non-transmissive head-mounted display used by a first user and a non-transmissive head-mounted display used by a second user. In this system, a virtual reality space image displayed on a head-mounted display used by the first user is generated based on the line-of-sight information of the second user transmitted via the network.

Further, a transmission type head mounted display (HMD) that is mounted on the head of the user U and superimposes and displays an image such as a virtual object on the field of view of the user U while allowing the user U to visually recognize the outside world is presented to the user U. Are known.

JP-A-2017-78891

処理 In processing related to virtual objects displayed on a head-mounted display, it is desired to reduce the processing load.

In view of the circumstances described above, an object of the present technology is to provide an information processing device, an information processing method, and a program capable of reducing a processing load.

In order to achieve the above object, an information processing device according to an embodiment of the present technology includes a control unit.
The control unit invites the first peripheral information of the first area and the second peripheral information of the second area of the real space obtained from the sensor, which are different from the visual field information corresponding to the visual field of the user. A first process related to a first virtual object arranged in the first area and a second process related to a second virtual object arranged in the second area based on the attraction degree. Determine the priority of processing.

According to such a configuration, it is possible to predict an area that the user may see from the peripheral information of the user and determine the degree of attraction in the first area and the second area in the non-viewing area. Then, based on the degree of attraction, the priority of the first processing for the first virtual object arranged in the first area and the priority of the second processing for the second virtual object arranged in the second area are determined. Can be determined.

The control unit may be configured to perform the first processing with higher priority than the second processing when determining a lower degree of interest for the second peripheral information than the first peripheral information. The priority may be determined.

The control unit determines the priority such that the first processing relating to the rendering of the first virtual object has a higher processing load than the second processing relating to the rendering of the second virtual object. You may.

The sensor includes a first illuminance sensor that detects illuminance of the first area as the first peripheral information, and a second illuminance sensor that detects illuminance of the second area as the second peripheral information. May be included.

制御 The control unit may determine the degree of attraction in consideration of the environment of the real space.

環境 The environment of the real space may be outdoors under sunlight.

When at least one of the first illuminance sensor and the second illuminance sensor detects an illuminance in the real space to which sunlight having an illuminance higher than an illuminance in a region corresponding to the user's field of view is applied. The control unit sets the degree of attraction to the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to which the sunlight is irradiated, using the illuminance of the area corresponding to the user's field of view and the illuminance sensor. The illuminance difference between the detected sunlight and the illuminance in the real space to be irradiated may be determined to be lower when the difference is equal to or larger than the threshold than when the difference is smaller than the threshold.

環境 The real space environment may be indoors.

The control unit may determine the degree of attraction of the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to be irradiated with sunlight so that the priority is the lowest.

When at least one of the first illuminance sensor and the second illuminance sensor detects the illuminance of the real space to which the light other than the sunlight is irradiated, the control unit sets the light other than the sunlight to The attractiveness of the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to be illuminated, the illuminance of the area corresponding to the user's field of view and the light other than the sunlight detected by the illuminance sensor The illuminance difference between the illuminance to be irradiated and the illuminance in the real space may be determined to be lower when the difference is less than the threshold than when the difference is equal to or greater than the threshold.

The first sensor detects a loudness of the sound in the first area as the first peripheral information, and detects a loudness of the sound in the second area as the second peripheral information. A second sound sensor may be included.

The control unit, when the loudness of the sound in the second area is smaller than the loudness of the sound in the first area, sets the degree of attraction for the second peripheral information to the degree of attraction for the first peripheral information. May be determined to be low.

The sensor detects a odor intensity of the first area as the first peripheral information, and detects an odor intensity of the second area as the second peripheral information. A second odor sensor may be included.

The control unit, when the odor intensity of the second area is weaker than the odor of the first area, sets the degree of attraction to the second peripheral information to be lower than the degree of attraction to the first peripheral information. It may be determined.

The sensor may be a camera that acquires image information around the user as the surrounding information.

The image information includes positional relationship information between the first region and the second region and the user, and the control unit determines the degree of attraction using the positional relationship information. You may.

The information processing device can be mounted on the head of the user, and can present the first virtual object and the second virtual object in the field of view of the user while allowing the user to visually recognize the outside world. It may be a head mounted display configured.

In order to achieve the above object, an information processing method according to an embodiment of the present technology is configured to provide first peripheral information of a first area in a real space different from visual field information corresponding to a visual field of a user and second information of a second area. The second peripheral information is obtained from the sensor, the degree of attraction of each of the first peripheral information and the second peripheral information is determined, and the first attraction arranged in the first area is determined based on the degree of attraction. The priority of the first process related to the virtual object and the priority of the second process related to the second virtual object arranged in the second area are determined.

In order to achieve the above object, a program according to an embodiment of the present technology includes first peripheral information of a first area in a real space different from visual field information corresponding to a visual field of a user, and second peripheral information of a second area. Obtaining peripheral information from a sensor; determining the degree of attraction of each of the first peripheral information and the second peripheral information; and determining a degree of attraction in the first area based on the degree of attraction. The information processing apparatus is configured to execute a process including a step of determining a priority of a first process related to one virtual object and a second process related to a second virtual object arranged in the second area.

FIG. 6 is a block diagram of an information processing device according to first to fourth embodiments of the present technology. FIG. 13 is a flowchart illustrating an example of a process regarding a virtual object in the information processing apparatuses according to the first to third embodiments. FIG. 13 is a flowchart illustrating an example of a process regarding a virtual object in the information processing apparatuses according to the first to third embodiments. FIG. 13 is a flowchart illustrating an example of a process regarding a virtual object in the information processing apparatuses according to the first to third embodiments. It is a figure showing the hardware constitutions of the above-mentioned information processor. FIG. 9 is a schematic diagram for explaining a relationship between an area around the user U and the number of priority processing points when the situation around the user U is not a situation that induces the movement of the line of sight of the user U. It is a figure for explaining a 1st embodiment, and when a surrounding situation of user U is a situation which induces eye movement of user U, it is a field of a surrounding of user U, and a priority processing score. It is a schematic diagram for demonstrating a relationship. It is a figure for explaining a 2nd embodiment, and when a surrounding situation of user U is a situation which induces a line-of-sight movement of user U, an area around user U and a priority processing score are shown. It is a schematic diagram for demonstrating a relationship. It is a figure for explaining a 3rd embodiment, and when a surrounding situation of user U is a situation which induces a gaze shift of user U, it is a field of a surrounding of user U, and a priority processing score. It is a schematic diagram for demonstrating a relationship. It is a figure for explaining a 4th embodiment, and a relation between an area around user U1 and a priority processing point when a situation surrounding user U1 is not a situation which induces a user's line of sight movement. It is a schematic diagram for demonstrating. It is a figure for explaining a 4th embodiment, and is a mimetic diagram for explaining judgment of a priority processing point using peripheral information. It is a flow figure explaining an example of processing about a virtual object in an information processor concerning a 4th embodiment.

(Overview of information processing device)
The information processing device according to an embodiment of the present technology is a head-mounted display (HMD), which is a display device mounted on the head of the user U, and is one of wearable computers. The shape of the HMD is typically an eye mirror type or a hat type.

The HMD of the present embodiment is a transparent HMD, for example, for reproducing a game content and superimposing and presenting a virtual object (display image) corresponding to the content to the user U in the external world, which is a real space. Can be configured. The transmission type HMD includes an optical transmission type, a video transmission type, and the like.

The transmissive HMD has a display arranged in front of the user U when worn on the head of the user U. The display includes an image display element and an optical element.

In the optical transmission system, a display image displayed on the image display element is presented to the user U via an optical element such as a holographic optical element or a half mirror disposed in front of the user U. In the optical transmission system, the user U can see the state of the outside world through the optical element in a see-through manner, and a display image such as a virtual object displayed on the image display element is arranged and superimposed on the state of the external world that is a real space. Is presented to the user U.
The image includes a still image and a moving image.

In the video transmission system, the user cannot directly see the external world when worn, but an image in which the virtual object is superimposed on the external image obtained by the camera is displayed on the display, so that the user U can visually recognize the external world. The virtual object can be presented while the virtual object is being displayed.

(4) In the HMD, if processing related to drawing is performed with the same processing load for all virtual objects arranged in the non-viewing area of the user U wearing the HMD, the processing load of the HMD increases, which is inefficient.

Therefore, the HMD according to the present embodiment predicts a region in a real space that is likely to be in the user's field of view based on the peripheral information of the user. Based on the result of the prediction, the processing content regarding the drawing of the virtual object arranged in the real space is changed. Specifically, the processing load on the virtual object placed in an area that is unlikely to be in the field of view of the user U is lower than the processing load on the virtual object placed in an area that is likely to be in the field of view of the user U. I do.

The peripheral information includes the peripheral situation information and the space information, but may be only the peripheral situation information.
The peripheral situation information is information on the peripheral situation of the user, for example, illuminance information, sound information, odor information, image information, and the like.
The spatial information is the orientation, position information, and the like of the user.

In the present embodiment, the real space around the user is divided into a plurality of regions, and the user's degree of attraction is determined for each region based on the peripheral information acquired for each region. The degree of attraction indicates a possibility of entering the field of view of the user U. The area that is highly likely to be in the user's field of view is an area where the user U is likely to be caught by the eye, and is an area where the degree of attraction is high.

Then, the processing loads for the virtual objects arranged in the region with a high degree of attraction are made different from each other so that the processing for the virtual objects arranged in the region with a low degree of attraction is prioritized over the processing with respect to the virtual objects arranged in the region with a low degree of attraction. I have. Thus, efficient processing can be performed.
The details will be described below.

(Configuration of information processing device)
The information processing apparatus will be described with reference to FIG. FIG. 1 is a block diagram of a head mounted display (HMD) 1 as an information processing device.

As shown in FIG. 1, the HMD 1 includes a control unit 10, an input unit 20, an output unit 30, a storage unit 46, and a support (not shown).

The support can be mounted on the head of the user U. The support is supported by the displays 31R and 31L, which are the output units 30, in front of the user at the time of wearing. In addition, the supporter supports each sensor, which will be described later, which is the input unit 20. The shape of the support is not particularly limited, and may be, for example, a hat shape as a whole.

[Input section]
The input unit 20 is a sensor unit group including a plurality of sensor units. The detection result detected by each sensor unit is input to the control unit 10. The detection result detected by each sensor unit includes the surrounding situation information and the space information.

The peripheral situation information is information around the user U in the real space where the user U wearing the HMD 1 is located. Specifically, the surrounding situation information is illuminance information, sound information, odor information, image information, and the like in the real space.
These pieces of information can be acquired from an illuminance sensor 22, a sound sensor 23, an odor sensor 24, and a camera 25 described later.

The space information includes image information captured by a camera 25 described later mounted on the HMD 1, acceleration information, angular velocity information, azimuth information, and the like of the HMD 1 detected by a nine-axis sensor 21 described later. It is possible to detect the position, orientation, movement, and posture (walking, running, stopping, etc.) of the user U from these spatial information.

The input unit 20 includes a nine-axis sensor 21, an illuminance sensor 22, a sound sensor 23, an odor sensor 24, and a camera 25. These sensors are mounted on the HMD1.

The 9-axis sensor 21 includes a 3-axis acceleration sensor, a 3-axis gyro sensor, and a 3-axis compass sensor. The nine-axis sensor 21 can detect the acceleration, angular velocity, and orientation of the HMD 1 in three axes, and can detect the position, orientation, movement, and posture (walking, running, stopping, and the like) of the user U. it can.
The detection result detected by the 9-axis sensor 21 is output to the control unit 10 as spatial information.

The illuminance sensor 22 has a light receiving element, and converts the light incident on the light receiving element into a current to detect brightness (illuminance). In the present embodiment, four illuminance sensors 22 are provided. The four illuminance sensors 22 detect illuminance in each area when the real space where the user U is located is divided into four areas.

FIG. 6 is a diagram illustrating a positional relationship between each area and the user U when the real space where the user U is located is divided into four areas. FIG. 6 and FIGS. 7 to 11 described below correspond to views of the

real space

60 or 70 where the user U moves while wearing the HMD 1 as viewed from above the user U.

As shown in FIG. 6, the real space 60 is divided into a front area 60F on the front side of the user U, a rear area 60B on the back side, a right area 60R on the right side, and a left area on the left side with the user U as a center. It is divided into four 60L regions.

The four illuminance sensors 22 include an illuminance sensor 22F that detects illuminance in the front area 60F, an illuminance sensor 22R that detects illuminance in the right area 60R, an illuminance sensor 22L that detects illuminance in the left area 60L, and a rear area 60B. An illuminance sensor 22B that detects illuminance.

In each of the right region 60R, the rear region 60B, and the left region 60L, the detection result (illuminance information) detected by each of the illuminance sensors 22R, 22B, and 22L is the surrounding situation information in each region, and the field of view of the user U. Is different from the visual field information corresponding to.
The detection results detected by the four illuminance sensors 22 are output to the control unit 10 as peripheral situation information.

The sound sensor 23 detects the loudness of the sound around the user U. In the present embodiment, four sound sensors 23 are provided. Like the illuminance sensor 22, the four sound sensors 23 respectively detect the sound volume in each area when the real space where the user U is located is divided into four areas. Using the detection results of the four sound sensors 23, the direction in which the sound source as viewed from the user U exists can be specified.

The four sound sensors 23 include a sound sensor 23F that detects the volume of the front area 60F, a sound sensor 23R that detects the volume of the right area 60R, a sound sensor 23L that detects the volume of the left area 60L, and a sound sensor 23L that detects the volume of the rear area 60B. A sound sensor 23B for detecting a volume.
The detection results detected by the four sound sensors 23 are output to the control unit 10 as surrounding situation information.

In each of the right area 60R, the rear area 60B, and the left area 60L, the detection result (sound information) detected by each of the sound sensors 23R, 23B, and 23L is the surrounding situation information in each area, and the field of view of the user U. Is different from the visual field information corresponding to.

The odor sensor 24 detects the intensity of the odor around the user U. In the present embodiment, four odor sensors 24 are provided. Using the detection results of the four odor sensors 24, the direction in which the source of the odor as viewed from the user U exists can be specified.

The four odor sensors 24 include an odor sensor 24F that detects the intensity of the odor in the front region 60F, an odor sensor 24R that detects the odor in the right region 60R, an odor sensor 24L that detects the odor in the left region 60L, and a rear. The odor sensor 24B detects the odor of the area 60B.
The detection results detected by the four odor sensors 24 are output to the control unit 10 as peripheral situation information.

In each of the right region 60R, the rear region 60B, and the left region 60L, the detection result (odor information) detected by each of the odor sensors 24R, 24B, and 24L is the surrounding situation information in each region, and the field of view of the user U. Is different from the visual field information corresponding to.

として As the odor sensor 24, for example, an oxide semiconductor sensor, a crystal oscillator (QCM) sensor having a film having molecular selectivity formed on the oscillator surface, or the like can be used.

The camera 25 includes a right-eye camera 251 and a left-eye camera 252. The right-eye camera 251 and the left-eye camera 252 capture an image of the front area 60F of the user U corresponding to the visual field area of the user U, and acquire a captured image as image information.

The right-eye camera 251 and the left-eye camera 252 are arranged at a predetermined interval in the horizontal direction on the front surface of the head mounted display 1. The right-eye camera 251 captures a right-eye image, and the left-eye camera 252 captures a left-eye image.

The right-eye image and the left-eye image captured by the right-eye camera 251R and the left-eye camera 252 include spatial information such as the position and orientation of the user U wearing the HMD1.
The right-eye image and the left-eye image are output to the control unit 10 as spatial information.

[Output unit]
The output unit 30 has a display 31R for the right eye and a display 31L for the left eye, and these displays are mounted on the HMD1.

The right-eye display 31R and the left-eye display 31L are arranged in front of the right and left eyes of the user U, respectively.
The right-eye display 31R (the left-eye display 31L) has a right-eye image display element 311R (a left-eye image display element 311L) and a right-eye optical element 312R (a left-eye optical element 312L).

Hereinafter, when there is no particular need to distinguish between left and right, the display 31R for the right eye and the display 31L for the left eye are referred to as the display 31, the image display element 311R for the right eye and the image display element 311L for the left eye are referred to as the image display element 311, and the display element for the right eye. The optical element 312R and the left-eye optical element 312L may be referred to as an optical element 312 in some cases.

(4) The image display element 311 includes an organic EL display element, a liquid crystal display element as a light modulation element, and the like. The image display element 311 forms a display image such as a virtual object based on the image signal output from the control unit 10 and emits display light. When the display light enters the eyes of the user U via the optical element 312, the virtual object can be presented to the user U in the viewing area of the user U.

The optical element 312 is a holographic optical element, a half mirror, or the like, and is arranged in front of the user U. The optical element 312 is configured to diffract light emitted from the image display element 311 and guide the light to the left and right eyes of the user.

Furthermore, the optical element 312 is configured to transmit light from the outside world. Thereby, in the HMD 1, it is possible to present the display image formed by the image display element 311 to the user U by superimposing the display image on the light from the outside.

The control unit 10 includes a communication control unit 11, a surrounding situation information management unit 12, a spatial information acquisition unit 13, a spatial information management unit 14, a priority processing point determination unit 15, a drawing processing load determination unit 16, It includes an image generation unit 17, an output image control unit 18, and a drawing processing load management unit 19.

The communication control unit 11 communicates with the various sensors 21 to 25 and the display 31 mounted on the HMD 1 to transmit and receive various information. Specifically, the communication control unit 11 receives the detection results of the various sensors 21 to 25 as the surrounding situation information and the space information, and transmits an image signal or the like to the display 31.

(4) Further, the communication control unit 11 communicates with an HMD worn by another user or an external peripheral device, and transmits and receives various information. The communication control unit 11 can acquire image information and the like captured by a camera mounted on the HMD worn by another user. Image information captured by a camera mounted on another user's HMD includes positional relationship information between another user U wearing the HMD and a priority processing point determination target area described later.

The peripheral situation information management unit 12 obtains the detection results obtained by the illuminance sensor 22, the sound sensor 23, and the odor sensor 24 via the communication control unit 11, and the image information (detection results) obtained from the HMD of another user. ) Is stored in a peripheral situation database (not shown) in a time-series manner as peripheral situation information, and data is updated and managed so as to always be the latest peripheral situation information.

The space information acquisition unit 13 acquires, as space information, the detection results acquired by the 9-axis sensor 21 and the camera 25 acquired via the communication control unit 11.

The spatial information management unit 14 uses the detection results detected by the 9-axis sensor 21 and the camera 25 and the position and orientation information of the camera 25 obtained based on the detection results as spatial information as a spatial information database (not shown). And updates and manages the data so that it is always the latest spatial information.

The priority processing point determination unit 15 determines the priority processing points of each area based on the surrounding situation information managed by the surrounding situation information management unit 12 and the position and orientation information of the camera 25 managed by the spatial information management unit 14. The degree of attraction).

The priority processing score is obtained by converting the priority of processing relating to drawing of a virtual object (display image) currently arranged in the non-viewing area of the user U into a score, and is an attractiveness determined for the peripheral information. . An area that is likely to be viewed by the user U is assigned a high priority processing score as an area with a high degree of attraction. The higher the number of assigned priority processing points, the higher the priority of processing relating to drawing of a virtual object.

The priority processing points are divided into a front area 60F, a rear area 60B, a right area 60R, and a left area obtained by dividing a 360-degree periphery of the user U centering on the user U wearing the HMD1 existing in the real space into four parts as shown in FIG. It is required every 60L. In the present embodiment, an example in which the real space is divided into four parts will be described. However, the present invention is not limited to this.

The ranges of the front region 60F, the rear region 60B, the right region 60R, and the left region 60L can be determined in consideration of a general normal human visual field region.
In general, the visual field of a normal person is said to be about 60 degrees on the nose side with one eye and about 90 to 100 degrees on the ear side, and the range that can be seen simultaneously by both eyes is about 120 degrees left and right.

In consideration of this general visual field area, as shown in FIG. 6, a range in the left-right direction of the front area 60F in a 360-degree area around the user U along the direction connecting the right eye and the left eye of the user U is defined. the angle theta _F 120 °, may be set the angle theta _R and theta _L defining the range of the right side area 60R, and the left region 60L respectively 40 °, the angle theta _B to define the scope of the rear region 60B to 160 degrees . An imaginary line bisecting the angle θ _F defining the range of the front region 60F is located at the center of the front of the user U.

6 to 9 used in the description of the present embodiment, the angle θ _F defining the range of the front region 60F and the angle θ _B defining the range of the rear region 60B are the same for easy viewing. The angle does not match the above numerical range.

FIG. 6 shows an example of the number of priority processing points for each area when the surrounding situation is not a situation that induces the movement of the user's line of sight.

In FIG. 6, the priority processing point P1 of the front area 60F, which is the visual field area of the user U, is the highest 10 points as compared with other areas.
In the right region 60R and the left region 60L, it is predicted that whether the user U directs his / her gaze to the right or to the left has a similar possibility, and is less likely than the front region 60F. Since it is predicted, the priority processing points P2 and P4 are five.
In order for the user U to turn his or her gaze to the rear area 60B, the user U needs to change his or her head direction by almost 180 degrees, so that it is predicted that the user U is less likely to turn rearward than to the right and left sides. Therefore, the priority processing score P3 is three.

In the example illustrated in FIG. 6, the peripheral information (second peripheral information) of the rear area 60B in the real space makes the eyes of the user U more noticeable than the peripheral information (first peripheral information) of the right area 60R and the left area 60L. It is difficult to attract and has low attraction.

On the other hand, in a case where the surrounding situation of the user U is a situation that induces the movement of the line of sight of the user U, the priority processing score indicating the degree of attraction of the user U in each of the

areas

60F, 60B, 60R, and 60L is shown in FIG. It is different from the case shown.

The priority processing score is obtained based on the peripheral information in each of the

areas

60F, 60B, 60R, and 60L. As described above, the peripheral information includes the peripheral situation information and the space information. If the peripheral situation of the user U is a situation that induces the user U's line of sight, the priority processing points and the points shown in FIG. Will change.

The surrounding situation information includes illuminance information detected by four illuminance sensors 22F, 22B, 22R, and 22L mounted on the HMD 1 so as to acquire the surrounding situation of each of the

areas

60F, 60B, 60R, and 60L, and four sounds. The sound information detected by the sensors 23F, 23B, 23R, and 23L, the odor information detected by the four odor sensors 24F, 24B, 24R, and 24L, the camera 25 mounted on the HMD 1 worn by the user U and another user U At least one of the pieces of image information captured in step (1).

(4) The drawing processing load determination unit 16 determines the processing load related to the drawing of the virtual object placed in each of the

areas

60F, 60B, 60R, and 60L according to the priority processing points determined by the priority processing point determination unit 15.

(4) The load of processing (first processing) related to a virtual object (first virtual object) arranged in an area having a relatively high priority processing point (an area of the real space corresponding to the first peripheral information) increases. Relatively, the load of the process (the second process) related to the virtual object (the second virtual object) arranged in the region where the number of priority processing points is low (the region of the real space corresponding to the second peripheral information) is reduced. .

As an example of the processing load determination, when the priority processing point P is 0 <P ≦ 3, the drawing processing load determination unit 16 determines that the processing X with a low processing load such as moving the virtual object once every 100 msec is performed. do.
When the priority processing point P is 3 <P ≦ 8, the drawing processing load determination unit 16 determines to perform the processing Y with a medium processing load such as moving the virtual object once every 50 msec.
When the priority processing point P satisfies 8 <P ≦ 10, the drawing processing load determination unit 16 determines to perform the processing Z with a high processing load such as moving the virtual object once every 16 msec.
Here, an example in which the processing load is divided into three stages has been described, but the processing load is not limited to this.

In the example shown in FIG. 6, the priority processing score of the front area 60F is 10, and therefore, the drawing processing load determination unit 16 determines that the drawing processing of the virtual object arranged in the front area 60F is performed in the processing Z.
Since the priority processing points of the right area 60R and the left area 60L are 5, the rendering processing load determination unit 16 determines that the rendering processing of the virtual objects arranged in the right area 60R and the left area 60L is performed in the processing Y. .
Since the number of priority processing points of the rear area 60B is three, the rendering processing load determination unit 16 determines that the rendering processing of the virtual object arranged in the rear area 60B is performed in the process X.

Accordingly, the load of the rendering processing of the virtual object arranged in the front area 60F is the highest. The load of the rendering processing of the virtual objects arranged in the right area 60R and the left area 60L is medium. The load of the rendering processing of the virtual object arranged in the rear area 60B is the lowest.

As a result, the processing load on the information processing device (HMD) can be reduced as compared with the case where the drawing processing is executed with the same processing load on all the virtual objects arranged in the non-viewing area, and the efficiency can be reduced. Good processing can be performed.
For example, if the HMD 1 is not powered by wire but powered by a battery, a reduction in processing load leads to an increase in battery life due to a reduction in power consumption.

処理 In the non-viewing area of the user U, the processing Y on the virtual objects (first virtual objects) arranged in the right area 60R and the left area 60L (first area) corresponds to the first processing. The processing X for the virtual object (second virtual object) arranged in the rear area 60B (second area) corresponds to the second processing.

As described above, in the present embodiment, the processing priority is changed by changing the processing content so that the processing load is different according to the priority processing score (attraction level).

(4) The output image generation unit 17 changes the processing content of the CPU and the GPU based on the processing content determined by the drawing processing load determination unit 16, and generates a virtual object (display image).

If the processing content determined by the drawing processing load determination unit 16 is processing X, the output image generation unit 17 generates a virtual object so that the virtual object moves once every 100 msec.
In the case of processing Y, the output image generation unit 17 generates a virtual object such that the virtual object moves once every 50 msec.
In the case of processing Z, the output image generation unit 17 generates a virtual object such that the virtual object moves once every 16 msec.

(4) The output image control unit 18 outputs the virtual object generated by the output image generation unit 17 as an image signal so that the virtual object can be displayed on the right-eye display 31R and the left-eye display 31L of the HMD 1.

The drawing processing load management unit 19 stores the processing content determined by the drawing processing load determination unit 16 in a time series in a drawing processing load database (not shown) in association with the surrounding situation information, the space information, and the number of priority processing points. Update and manage data so that it is always up-to-date.

[Storage unit]
The storage unit 46 stores a program for causing the HMD 1 as an information processing device to execute a series of information processing on the virtual object performed by the control unit 10.

(Information processing method)
Next, an example of a series of information processing methods related to a virtual object drawing process executed by the HMD 1 will be described with reference to FIG.
FIG. 2 is a flowchart for explaining processing relating to a virtual object in the HMD 1.

As shown in FIG. 2, when the process starts, the surrounding state information is acquired by the communication control unit 11 (S1). The peripheral situation information is stored in the peripheral situation database. The surrounding situation information is a detection value detected by the illuminance sensor 22, a detection value detected by the sound sensor 23, a detection value detected by the odor sensor 24, and the like, and at least one of them is acquired.
An information processing method in a case where image information captured by the camera 25 is acquired as the peripheral situation information will be described in a fourth embodiment described later.

Next, the detection value detected by the 9-axis sensor 21 and the image information (photographed image) photographed by the camera 25 are acquired by the spatial information acquisition unit 13 via the communication control unit 11, and the camera information is acquired based on these information. Information on the position and orientation of the 25 is obtained (S2).
Spatial information such as a detection value detected by the 9-axis sensor 21, image information captured by the camera 25, and information on the position and orientation of the camera 25 obtained based on these information is stored in a spatial information database.

Next, the priority processing point determination unit 15 performs the priority processing for each of the front area 60F, the rear area 60B, the right area 60R, and the left area 60L of the user U based on the surrounding situation information and the position and orientation information of the camera 25. A score P is obtained (S3).

Next, the processing content is determined by the drawing processing load determination unit 16 based on the priority processing score obtained in S3, and the output image generation unit 17 generates a virtual object (display image) based on the determined processing content. Generation processing is executed (S4).

Next, the output image control unit 18 converts the virtual object generated by the output image generation unit 17 into an image signal so that it can be displayed on the display 31, and outputs the image signal to the output unit 30 (S5).

The virtual object is displayed on the display 31 based on the image signal output from the control unit 10 and presented to the user U.

A series of processes related to the drawing process described above may be performed at regular intervals, or may be performed each time the direction of the user U changes and the direction of the HMD 1 changes.
Hereinafter, an example in which a series of processes related to the drawing process is executed at regular time intervals with reference to FIG. 3 will be described. A series of processes related to the drawing process will be executed by changing the direction of the HMD 1 with reference to FIG. An example will be described.

A case where a series of processes related to the drawing process is executed at regular intervals will be described.
As shown in FIG. 3, when the process is started, in the first process, it is considered that a predetermined time has elapsed in S11, and the process proceeds to S12 and thereafter. In the second and subsequent processes, it is determined whether a predetermined time has elapsed from the previous drawing process (S11).
If it is determined that the predetermined time has elapsed in S11 (Yes), the process proceeds to S12.
If it is determined in S11 that the predetermined time has not elapsed (No), the process proceeds to S16.

In S12, the communication control unit 11 acquires the surrounding situation information. The acquired surrounding situation information is stored in the surrounding situation database.

Next, the detection value detected by the 9-axis sensor 21 and the image information (photographed image) photographed by the camera 25 are acquired by the spatial information acquisition unit 13 via the communication control unit 11, and the camera information is acquired based on these information. Information on the position and orientation of the 25 is obtained (S13).
Spatial information such as a detection value detected by the 9-axis sensor 21, image information captured by the camera 25, and information on the position and orientation of the camera 25 obtained based on these information is stored in a spatial information database.

Next, the priority processing point determination unit 15 performs the priority processing for each of the front area 60F, the rear area 60B, the right area 60R, and the left area 60L of the user U based on the surrounding situation information and the position and orientation information of the camera 25. A score P is obtained (S14).

Next, the processing contents are determined by the drawing processing load determination unit 16 based on the priority processing points obtained in S14 (S15).

Next, a process of generating a virtual object (display image) by the output image generation unit 17 is performed based on the determined processing content (S16). Here, if it is determined in S11 that the predetermined time has not elapsed and the process proceeds to S16, a virtual object generation process is executed in S16 based on the processing content determined in the previous process.

Next, the output image control unit 18 converts the virtual object generated by the output image generation unit 17 into an image signal so that it can be displayed on the display 31, and outputs the image signal to the output unit 30 (S17).

Next, a case will be described in which a series of processes related to the drawing process are executed by changing the direction of the HMD 1.

4. As shown in FIG. 4, when the process starts, the rotation angle of the HMD 1 is acquired from the detection value detected by the 9-axis sensor 21 by the spatial information acquisition unit 13 via the communication control unit 11 (S21).

Next, in the first process, it is considered that the rotation amount exceeds the threshold value in S22, and the process proceeds to S23. In the second and subsequent processes, the rotation amount of the HMD 1 is calculated by the control unit 10 from the rotation angle of the HMD 1 acquired in the previous drawing process and the rotation angle of the HMD 1 acquired in S21, and the rotation amount is equal to or larger than the threshold value. It is determined whether or not it is (S22). The threshold is set in advance.

If it is determined in S22 that the value is equal to or larger than the threshold (Yes), the process proceeds to S23.
If it is determined in S22 that the value is less than the threshold value (No), the process proceeds to S27.

In S23, the surrounding situation information is acquired by the communication control unit 11, and the surrounding situation information is stored in the surrounding situation database.

Next, the detection value detected by the 9-axis sensor 21 and the image information (photographed image) photographed by the camera 25 are acquired by the spatial information acquisition unit 13 via the communication control unit 11, and the camera information is acquired based on these information. Information on the position and orientation of the position No. 25 is obtained (S24).
Spatial information such as a detection value detected by the 9-axis sensor 21, image information captured by the camera 25, and information on the position and orientation of the camera 25 obtained based on these information is stored in a spatial information database.

Next, the priority processing point determination unit 15 performs the priority processing for each of the front area 60F, the rear area 60B, the right area 60R, and the left area 60L of the user U based on the surrounding situation information and the position and orientation information of the camera 25. A score P is obtained (S25).

Next, the content of the processing is determined by the drawing processing load determination unit 16 based on the priority processing score obtained in S25 (S26).

Next, a process of generating a virtual object (display image) by the output image generation unit 17 is performed based on the determined processing content (S27). If the rotation amount is determined to be less than the threshold value in S22 and the process proceeds to S27, a virtual object generation process is performed in S27 based on the processing content determined in the previous process.

Next, the output image control unit 18 converts the virtual object generated by the output image generation unit 17 into an image signal so that it can be displayed on the display 31, and outputs the image signal to the output unit 30 (S28).

As described above, the control unit 10 controls the processing related to the virtual object currently arranged in the non-visual area of the user U based on the surrounding situation information and the space information acquired from the various sensors of the input unit 20.

Specifically, the control unit 10 predicts an area having a high possibility of entering the field of view of the user U using the surrounding situation information and the spatial information. Then, the control unit 10 performs the rendering process of the virtual object in the region that is likely to be in the predicted field of view of the user U, that is, the region where the user U has a high degree of attraction, and performs the drawing in another region with a low degree of attraction. Control is performed such that the processing is performed with higher priority than the processing.

(Hardware configuration)
FIG. 5 is a diagram for explaining a hardware configuration of the HMD 1. The information processing in the HMD 1 as the information processing apparatus described above is realized by cooperation between software and hardware of the HMD 1 described below.

As shown in FIG. 5, the HMD 1 includes a CPU (Central Processing Unit) 51, a RAM (Random Access Memory) 52, a ROM (Read Only Memory) 53, a GPU (Graphics Processing Unit) 54, and a communication device 55. , A sensor 56, an output device 57, a storage device 58, and an imaging device 59, which are connected via a bus 61.

The CPU 51 controls the overall operation of the HMD 1 according to various programs.
The ROM 53 stores programs used by the CPU 51, operation parameters, and the like.
The RAM 52 temporarily stores a program used in the execution of the CPU 51, a parameter appropriately changed in the execution, and the like.
The GPU 54 performs various processes related to generation of a display image (virtual object).

The communication device 55 is a communication interface configured with a communication device or the like for connecting to the communication network 62. The communication device 55 may include a communication device compatible with a wireless LAN (Local Area Network), a communication device compatible with LTE (Long Term Evolution), a wire communication device that performs wired communication, or a Bluetooth (registered trademark) communication device.

The sensor 56 detects various data related to the surrounding situation information and the space information. The sensor 56 corresponds to the nine-axis sensor 21, the illuminance sensor 22, the sound sensor 23, and the odor sensor 24 described with reference to FIG.

The output device 57 includes a display device such as a liquid crystal display device and an organic EL (Electroluminescence) display device. Further, the output device 57 includes a sound output device such as a speaker or headphones. The display device displays a captured image, a generated image, and the like. On the other hand, the sound output device converts a sound signal into sound and outputs the sound. The output device 57 corresponds to the display 31 described with reference to FIG.

The storage device 58 is a device for storing data. The storage device 58 may include a recording medium, a recording device that records data on the recording medium, a reading device that reads data from the recording medium, a deletion device that deletes data recorded on the recording medium, and the like. The storage device 58 stores programs executed by the CPU 51 and the GPU 54 and various data. The storage device 58 corresponds to the storage unit 46 described with reference to FIG.

The imaging device 59 includes an imaging optical system such as a photographing lens and a zoom lens that collects light, and a signal conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging optical system collects light emitted from the subject and forms a subject image on the signal conversion unit. The signal conversion element converts the formed subject image into an electric image signal. The imaging device 59 corresponds to the camera 25 described with reference to FIG.

A program constituting the software includes a step of acquiring first peripheral information and second peripheral information in a real space different from visual field information corresponding to a visual field of the user from the sensor; and a step of acquiring the first peripheral information and the second peripheral information. Determining the degree of attraction of each piece of information; first processing and second processing relating to the first virtual object arranged in the first area of the real space corresponding to the first peripheral information based on the degree of attraction This is for causing the information processing apparatus to execute a process including a step of determining the priority of the second process of the second virtual object arranged in the second area of the real space corresponding to the surrounding information of the second virtual object.

In the following embodiments, specific surrounding situation information will be described as an example, and how the priority processing score is determined in the processing relating to the virtual object in the HMD 1 as the information processing apparatus described above will be described.

In the first embodiment, an example in which the detection result of the illuminance sensor 22 is used as the peripheral situation information when the priority processing point is determined is described. In the second embodiment, an example using the detection result of the sound sensor 23 will be described. In the third embodiment, an example using the detection result of the odor sensor 24 will be described. In the fourth embodiment, an example in which a captured image (detection result) captured by the camera 25 is used will be described.

In the first to third embodiments, only the specific example of the peripheral situation information to be obtained is different, and the processing relating to the virtual object can be executed in accordance with the flowcharts of FIGS. 2 to 4 described above.

Also, in the above-described example of the HMD 1, the illuminance sensor 22, the sound sensor 23, and the odor sensor 24 for acquiring the surrounding situation information are all mounted on the HMD 1, but the invention is not limited to this. At least one of an illuminance sensor 22, a sound sensor 23, an odor sensor 24, and a camera capable of acquiring image information may be used to acquire the surrounding situation information. Further, the sensor for acquiring the surrounding situation information may be an external device without being mounted on the HMD 1.

<First embodiment>
In the first embodiment, an example will be described in which a detection result (illuminance information) acquired from the illuminance sensor 22 as surrounding situation information is used to determine a priority processing point (attraction level).

The control of the processing related to the virtual object based on the illuminance information as the peripheral situation information acquired by the illuminance sensor 22 can be applied to, for example, the HMD 1 that reproduces the game content. When the user U wears the HMD 1, the HMD 1 can present the user U with a virtual object (display image) corresponding to the content superimposed on the external world that is a real space.

The user U can enjoy the game while freely moving in the real space while wearing the HMD. When playing a game while moving in the real space, it is assumed that the game is enjoyed outdoors or indoors.

Normally, people have empirically learned that even if they look in a dazzling direction such as the direction in which the sun is located, their eyes are not immediately accustomed to their brightness and are difficult to see.

Therefore, in an outdoor game, since the sunlight is dazzling, the behavior of the user U such as avoiding moving the line of sight in the direction of the sun where the illuminance is higher than the illuminance of the area corresponding to the current visual field is predicted. Is done.
Further, when the user U is facing in the direction where the dazzling sunlight is located at the present time, it is predicted that the user U will take an action of not being dazzled or moving his / her gaze to a region with lower illuminance.

In an indoor game, the behavior of the user U such as avoiding moving his line of sight to an area irradiated with sunlight, which is predicted to be difficult to see by sunlight entering indoors, is predicted as in the case of outdoors.
On the other hand, in an indoor game, when there is light other than sunlight, such as an indoor light, if the light is brighter than the current brightness in the visual field of the user U, the predicted action against the sunlight is opposite. In addition, it is predicted that the user U moves his / her line of sight in a direction that emits brighter light.

In the present embodiment, the priority processing points are determined by taking into account the real space environment such as the outdoor or indoor environment described above in addition to the illuminance information as the peripheral situation information acquired from the illuminance sensor 22. The environment of the real space is, for example, whether it is outdoors or indoors, if it is outdoors, whether it is sunshine when the sun is out, if it is indoors, it is whether there is a window through which the sunlight enters etc. is there.

The illuminance information as the peripheral situation information is information that occurs accidentally, such as sunlight, an outdoor light, an indoor light, and the like, which is different from information preset in the game content.
Hereinafter, the details will be described.

(In the case of outdoors in sunshine)
First, a case where the environment in which the user U exists is outdoors in the sunshine will be described.
FIG. 7 shows a determination example of the priority processing score for each area in a situation where the user U is located outdoors in the sunshine and the sun 65 is located in the right area 60R of the user U. The situation shown in FIG. 7 is a situation in which the sunlight is dazzling and induces the movement of the line of sight of the user U.
In addition, the time of sunshine refers to a state in which the direct sunlight of the sun illuminates the ground surface to the extent that a shadow of an object is formed.

屋外 Outdoors during sunshine, the illuminance sensor 22 mainly detects the illuminance in a real space where sunlight is irradiated. In addition to sunlight, it is assumed that artificial light such as an outdoor light is detected.

屋外 Outdoors in bright sunlight are bright environments. Normally, people have empirically learned that in such a bright environment, even if they look in the dazzling direction where the sun is located, their eyes are not immediately accustomed to the brightness and it is difficult to see.

In other words, a region that is brighter than the brightness (illuminance) of the front region 60F corresponding to the field of view of the user U at the present time and in which the sun where the illuminance is assumed to be greatly different from the illuminance of the front region 60F is located. In other words, it is assumed that the user U does not dare to look at the screen because it is too dazzling.

On the other hand, for a region that is higher than the illuminance of the front region 60F corresponding to the field of view of the user U at the current time but the illuminance does not change much from the illuminance of the front region 60F, the user U does not become too dazzling. The area is not specifically avoided and may be viewed as a normal course of action.

Therefore, in the outdoor environment under the sunshine, the attractiveness of the area where the illuminance is higher than the front area 60F corresponding to the current visual field of the user U in the non-visual area of the user U is determined as follows.

That is, first, the illuminance of the front region 60F corresponding to the current visual field of the user U detected by the illuminance sensor 22F and the illuminance sensors 22R (22B, 22L) detecting the illuminance of the non-visual region of the user U are detected. Of the right region 60R (the rear region 60B and the left region 60L).
Then, the attractiveness of the right region 60R (the rear region 60B, the left region 60L) is determined to be lower when the illuminance difference is equal to or larger than the threshold than when the illuminance difference is smaller than the threshold.

That is, an area where the illuminance difference is significantly different from the front area 60F is an area that is too dazzling for the user U, and it is assumed that the user U is unlikely to see the area, and the priority processing score (attraction degree) is determined to be low.
On the other hand, an area where the illuminance difference is small from the front area 60F is an area that is not too dazzling for the user U, is assumed to be likely to be seen by the user U, and has a large illuminance difference (in the case of a threshold or more). The priority processing score (attraction level) is determined to be higher than that.

In the example illustrated in FIG. 7, the sun is located on the right side of the user U, and the illuminance value detected by the illuminance sensor 22R that detects the illuminance in the right side region 60R is the highest among the four illuminance sensors 22, and the front region 60F and the left side This shows a case where the illuminance values detected in the area 60L and the rear area 60B are the same. In FIG. 7, it is assumed that the difference (illuminance difference) between the illuminance value of the front region 60F and the illuminance value of the right region 60R is equal to or larger than the threshold value.

In the example illustrated in FIG. 7, the user U is predicted to have the lowest possibility of directing his or her gaze to the dazzling right region 60 </ b> R where the sun is located, and thus the priority processing score P <b> 2 is one. On the other hand, the front area 60F, the left area 60L, and the rear area 60B are predicted to have lower illuminance than the right area 60R and a higher possibility of the user U turning their eyes, and the respective priority processing points P1, P4, and P3 are: There are eight points.

In FIG. 7, in the non-viewing area, the peripheral information (second peripheral information) of the right area 60R (second area) of the real space is the peripheral information (second area information) of the rear area 60B and the left area 60L (first area). It is harder to catch the eyes of the user U and has a lower degree of attraction than the first peripheral information).

In FIG. 7, based on the processing content determined based on the determined priority processing score, the processing X (second virtual object) is performed on the virtual object (second virtual object) arranged in the right area 60R (second area). Is performed based on the above process. With respect to the virtual objects (first virtual objects) arranged in the left region 60L, the front region 60F, and the rear region 60B (first region), drawing is performed based on the process Y (first process).

On the other hand, even when the sun is located on the right side of the user U and the illuminance of the right region 60R is the highest and the illuminance of the front region 60F, the left region 60L, and the rear region 60B is the same, the front region 60F and the right region 60R. If the illuminance difference is less than the threshold value, a different priority processing score is determined than when the difference is equal to or greater than the threshold value.

The user U may see an area where the illuminance difference of the non-viewing area is less than the threshold value as a normal flow of action without particular avoidance.

As an example, when the illuminance difference is less than the threshold value, as shown in FIG. 6, the priority processing point is determined to be a case where the surrounding situation is not a situation that induces the user U's gaze movement, and the right processing area is determined. The priority processing score P2 of 60R is determined to be five. This is a higher priority processing score than the case where the illuminance difference is equal to or larger than the threshold value shown in FIG.

In the example shown in FIG. 7, an example is given in which the left region 60L and the rear region 60B have the same illuminance. However, if the illuminance is different, the priority processing score is determined according to the illuminance value. Then, according to the priority processing points determined in the left region 60L, the rear region 60B, and the right region 60R corresponding to the non-viewing region, the priority of the process regarding the virtual object arranged in each region in the non-viewing region is determined. Is done.

状況 Examples of situations in which the illuminance difference is less than the threshold include the case of evening, the case where the outdoor light is turned on outdoors in the sunshine, and the case where the user U is in the shadow of a building.

In these cases, even if there is light such that an illuminance difference occurs between different regions, since the illuminance difference is small, the user U does not particularly avoid an area in which the illuminance value is detected high. It doesn't pay special attention and may be viewed as a normal course of action. In such a case, for example, the priority processing score is determined to be a case where the surrounding situation is not a situation that induces the user U's line of sight as shown in FIG.

For example, the brightness of the sun at about 10:00 am in fine weather is about 65,000 lux, whereas the brightness of the sun one hour before the sunset of fine weather is about 1000 lux, and the evening is compared with the daytime. The illuminance difference tends to be small.
Thus, in the case of the evening, the glare of the sun is less likely to be felt than in the daytime, and the user U may see the direction in which the sun is located without particularly avoiding an area where the illuminance is high.

When the outside light is on outdoors in the sunshine, the illuminance detected in the direction where the outside light is located and the illuminance of the situation where the user U is currently placed, that is, the front area 60F corresponding to the field of view of the user U are displayed. It is assumed that the illuminance does not change much. This situation is not considered to be a situation that would induce the user U to move his / her gaze.

例えば Also, for example, when the user U is in the shadow of a building even in a sunny daytime outdoors, the detection values detected by the four illuminance sensors do not change so much, and the illuminance difference is assumed to be small. Even in such a case, it is considered that the situation does not induce the user U's gaze movement.

In addition, when the sun is located in the front area 60F corresponding to the current visual field of the user U, it is assumed that the user U moves his or her line of sight to a less dazzling area with lower illuminance.
On the other hand, when the sun is located in the front area 60F corresponding to the current visual field of the user U, even if there is a low illuminance area due to the non-visual field area, the illuminance does not change much as much as the front area 60F. The user U does not dare to look at the area, but may see it as a normal flow of action.

Therefore, in the outdoor environment under the sunshine, the attractiveness of the area where the illuminance is lower than the front area 60F corresponding to the current visual field of the user U in the non-visual area of the user U is determined as follows.

That is, first, the illuminance of the front region 60F corresponding to the current visual field of the user U detected by the illuminance sensor 22F and the illuminance sensors 22R (22B, 22L) detecting the illuminance of the non-visual region of the user U are detected. Of the right region 60R (the rear region 60B and the left region 60L).
Then, the degree of attraction of the right region 60R (the rear region 60B, the left region 60L) is determined to be lower when the illuminance difference is less than the threshold than when it is greater than or equal to the threshold.

That is, an area where the illuminance difference is largely different from the front area 60F is an area that is not too dazzling for the user U, and it is assumed that the user U has a high possibility to see in order to avoid dazzling at the present time. ) Is determined to be high.
On the other hand, the area where the illuminance difference from the front area 60F is small is glare that is not so different from the front area 60F for the user U, and it is assumed that the user U may see it as a normal action flow. The priority processing score (attraction level) is determined to be lower than when it is larger (when it is equal to or more than the threshold).

In addition, even during the daytime, it is cloudy and outdoors where the direct sunlight of the sun does not illuminate the surface of the ground to the extent that shadows of objects can be created, and outdoors at night when the sun does not come out, in the case of dark indoors described later The processing is the same as that described above.

(In the case of indoor)
Next, a case where the environment where the user U exists is indoors will be described.
When the user U is indoors, it is determined whether the illuminance detected by the illuminance sensor 22 is caused by sunlight entering the room from the outside.

In the case of indoors, for a real space area where sunlight is irradiated, the user U has a very low possibility of seeing, and has the lowest priority, that is, the number of priority processing points so that the processing load is the lowest. Is determined to be low.

In the case of indoors, it is basically determined that the user U has a high possibility of seeing a real space region irradiated with light other than sunlight and the priority processing score is increased.
Hereinafter, the case where the light applied to the real space area is the sunlight and the case where the light is not applied will be described.

[In case of sunlight]
The area pointed by the dazzling sunlight has a large difference in illuminance with respect to the average luminance (illuminance) of the indoor environment where the user U is located at the present time and is bright. User U knows empirically that he is not accustomed to brightness and it is difficult to see things clearly. Thereby, it is assumed that the user U does not actively look at the region pointed to by sunlight.

Therefore, indoors, the priority processing point (attraction degree) for the real space area irradiated with sunlight has the lowest priority of the processing relating to the virtual objects arranged in the area, that is, the processing load is the lowest. Is determined as follows.
In the example of this embodiment, the processing load is divided into three stages, and the processing X is the processing with the lowest processing load. Therefore, indoors, the priority processing points for the real space area irradiated with sunlight are given. Is determined to be 3 points or less so that the determination of the processing X is made.

[When not in sunlight]
On the other hand, when the illuminance detected by the illuminance sensor 22 is not caused by sunlight, the illuminance detected indoors is recognized by the user U as being caused by artificially generated light such as an interior light or a spotlight. You.

If there is a bright area where the illuminance differs greatly from the average luminance (illuminance) of the indoor environment where the user U is currently placed, the user U is artificially created, as opposed to the case of sunlight. It is assumed that a bright area caused by the reflected light is positively viewed.

However, even if there is a region where the illuminance value is detected to be higher than other regions due to light other than sunlight, the illuminance does not change much from the illuminance of the front region 60F corresponding to the current visual field of the user U. On the other hand, it is assumed that the user U is unlikely to notice the light.
In such a case, even in a region where the illuminance value is detected to be high, the user U does not pay particular attention to the region, and may see it as a normal flow of action.

Therefore, in an area where indoors other than sunlight is irradiated, the priority processing point (attraction degree) is determined as follows.

That is, first, the illuminance of the front region 60F corresponding to the current visual field of the user U detected by the illuminance sensor 22F and the illuminance sensors 22R (22B, 22L) detecting the illuminance of the non-visual region of the user U are detected. Of the right region 60R (the rear region 60B and the left region 60L).
Then, it is determined that the attractiveness of the right region 60R (the rear region 60B, the left region 60L) is lower when the illuminance difference is less than the threshold than when it is greater than or equal to the threshold.

That is, when it is assumed that the illuminance difference from the front area 60F is small and is not an area that induces the user U's line of sight, the priority processing score (attraction degree) is determined to be low.
On the other hand, when it is assumed that the illuminance difference with the front area 60F is large and the area where light other than sunlight is located is an area that induces the user U's line of sight, the illuminance difference is small (threshold). (In the case of less than the value), the priority processing score (attraction level) is determined to be higher.

When the illuminance difference is less than the threshold value, the user U may view the real space area irradiated with light other than the sunlight as a normal flow of action, and the priority processing score is relatively high. However, the priority processing score is determined to be lower than the case where the priority processing score is equal to or greater than the threshold value.

For example, a region in the real space where light other than sunlight whose illuminance difference is equal to or larger than the threshold is irradiated is determined as a region where the possibility of the user U to see is considerably high, and the priority processing score is considerably increased. .
On the other hand, if the illuminance difference is less than the threshold value and the area is not an area that induces the gaze movement of the user U, but if it is likely to be viewed as a normal action flow, a medium priority processing point is set. Is determined.

例 As an example of a situation where the illuminance difference is equal to or larger than the threshold value, there is a case where a bright indoor is locally illuminated with high illuminance light such as a spotlight. The area where such high illuminance light is irradiated is easy to catch the user U's eyes even when the whole room is bright, and the area with such light is an area where the degree of attraction is high.

Another example of a situation where the illuminance difference is equal to or larger than the threshold value is a case where a dark room is locally illuminated with a spotlight, brighter illumination light, downlight, or the like. Even if the area irradiated with such light is dark light, the user U can easily catch the eyes in a dark indoor.

Examples of a situation where the illuminance difference is less than the threshold include a case where a bright indoor is locally illuminated with ordinary room lighting or downlight, or a shadow area of an object such as a table. The area to which such light is radiated and the shadow area are not so different from the average brightness of the real space, and therefore do not particularly attract the user U's eyes.

の他 Another example of a situation where the illuminance difference is less than the threshold value is a dark area where there is no particular light inside a dark room. Such an area is not so different from the average brightness of the real space, and therefore does not particularly attract the user U's eyes.

Here, a bright indoor refers to, for example, an indoor with a brightness of 30 lux or more, and a dark indoor refers to, for example, an indoor with a brightness of less than 30 lux.

[Judgment of sunlight]
Whether or not the illuminance detected by the illuminance sensor 22 is due to sunlight can be determined using the position information of the user U, date and time / weather information, indoor information acquired in advance, and the like.

The position information of the user U can be acquired by a GPS (Global Positioning System) mounted on the HMD 1 or the like.
The date / time / weather information is, for example, the sun altitude (elevation angle) and azimuth angle of each place and date / time, which can be acquired by the HMD 1 communicating with an application server that provides date / time / weather information on an external network. Location information and weather information such as fine rain.
The indoor information includes window position information such as the presence / absence of an indoor window, the direction in which the window is located indoors, and the position of the window with respect to the wall.

Whether the user U is indoor or outdoors can be detected from the position information of the user U.
Further, when the user U is indoors, whether or not the light detected by the illuminance sensor is caused by sunlight can be determined based on the window presence / absence information, which is indoor information.
When there is no window, it is determined that the detected light is not light caused by sunlight, and the above-described processing when there is no sun can be performed.
On the other hand, when there is a window, the irradiation position of the sunlight entering the room through the window can be obtained from the position of the sun from the date / time / weather information and the position of the window from the indoor information. It can be determined whether or not the detected light is caused by sunlight.

Here, an example has been described in which the position information, date and time, and weather information of the user U are used to determine whether the light detected indoors is due to sunlight, but the present invention is not limited to this. The position information of the user U and the date / time / weather information can be used in other ways.

For example, as described above, it is possible to grasp whether the user U is indoors or outdoors from the position information of the user U, and therefore, based on the position information, either processing when indoors or processing when outdoors Can be determined.

{Circle around (4)} In the outdoor processing, the weather information at the place where the user U is currently located can be grasped from the date / time / weather information and the position information of the user U. This makes it possible to turn off the illuminance sensor when it is not necessary to execute information processing to prioritize processing of a virtual object arranged in a non-viewing area using illuminance information of light caused by sunlight, for example, when it is cloudy. The power consumption can be reduced.

In the above, the case where the user U plays a game while moving indoors or outdoors has been described as an example. In such a game, the user U may move from indoor to outdoor and from outdoor to indoor according to the content of the game.

For example, when the user U goes from indoor to outdoor in the flow of the game, the user U may move with his / her gaze to a dazzling area where sunlight is irradiated. Since the action of the user U going from indoors to outdoors can be assumed in advance according to the content of the game content, in addition to the above-described peripheral situation information, the user's action predicted according to the content is taken into consideration, The priority of the virtual object drawing process may be determined.

The threshold value of the illuminance difference is set in advance for each different environment such as outdoors or indoors. Further, the threshold is appropriately set for each brightness of the environment.

Alternatively, the average brightness of the environment in which the user U is placed, the illuminance value detected by each illuminance sensor, the illuminance difference between the front region 60F and each of the non-viewing regions, and the user U Data linking the behavior pattern of the user U such as whether or not a high region has been seen may be accumulated as needed. Then, based on these accumulated data, statistically, the relationship between the average brightness of the environment where the user U is placed, the behavior pattern of the user U, and the threshold value is established, and more appropriate A threshold may be set.

As described above, in the HMD 1 according to the present embodiment, the surrounding situation information is obtained by the illuminance sensor, and the behavior of the user U is predicted using the surrounding situation information. Is determined. Based on the degree of attraction, the priority of the process regarding the virtual object arranged in each of the non-viewing areas of the user U is determined. Then, based on the determined priority, the processing is executed so as to reduce the drawing processing load of the virtual object arranged in the area where the degree of attraction of the user U is low, so that efficient processing can be performed.

<Second embodiment>
In the second embodiment, a case will be described as an example in which sound information that is a detection result obtained by the sound sensor 23 is used as surrounding situation information.
FIG. 8 shows an example of determining the priority processing score for each area in a situation where the explosion sound 66 has occurred in the right area 60R of the user U. The situation illustrated in FIG. 8 is a situation that induces the movement of the line of sight of the user U.

In the present embodiment, the priority processing score of each of the

regions

60F, 60B, 60R, and 60L is obtained based on the sound volume that is the sound information detected by each of the sound sensors 23F, 23R, 23L, and 23B.

Generally, people tend to look in the direction that seems to be the source of sound when something sounds around. Therefore, in the present embodiment, the priority processing point is determined according to the volume so that the priority processing point of the region where the detected volume is the largest is higher.

The four sound sensors 23F, 23R, 23L, 23B detect the volume of sounds around the user U. In the example shown in FIG. 8, since the explosion sound 66 is generated on the right side of the user U, the volume value detected by the sound sensor 23R that detects the sound information in the right side region 60R is the largest.

When the user U faces the right region 60R, which is the sound source, the gaze direction of the user U from the state in which the user U is facing the front region 60F to the right region 60R, which is the sound source. Moves from the front area 60F to the right area 60R. The assumed line-of-sight movement area of the user U is an area where the user U is likely to see.
Therefore, in the example shown in FIG. 8, the priority processing points P1 and P2 of the front area 60F and the right area 60R are determined to be 10 points higher than the other areas.

On the other hand, since the left area 60L, which is the exact opposite of the right area 60R, which is the sound source, is assumed to be the area where the possibility that the user U turns his or her gaze is the lowest, the priority processing point P3 is determined to be three points. Is done.
The rear area 60B is an area adjacent to the right area 60R, and is more likely to enter the field of view of the user U than the left area 60L. Therefore, the priority processing score P3 is determined to be four.

In FIG. 8, the peripheral information (second peripheral information) of the rear area 60B and the left area 60L (second area) of the real space is the peripheral information (first peripheral information) of the right area 60R (first area). ), It is harder to catch the eyes of the user U and the degree of attraction is low.

In FIG. 8, drawing is performed based on the processing Z for the virtual objects arranged in the right area 60R and the front area 60F based on the determined priority processing points. With respect to the virtual object arranged in the left area 60L, drawing is performed based on the processing X. With respect to the virtual object arranged in the rear area 60B, drawing is performed based on the processing Y.

Assuming that the process Z relating to the virtual object (first virtual object) arranged in the right region 60R (first region) is the first process, the virtual object (second region) arranged in the left region 60L (second region) Processing X relating to the second virtual object) and processing Y relating to the virtual object (the second virtual object) arranged in the rear area 60B (the second area) correspond to the second processing.

Looking only at the relationship between the rear area 60B and the left area 60L, the processing Y for the virtual object (first virtual object) arranged in the rear area 60B (first area) corresponds to the first processing. Processing X relating to the virtual object (second virtual object) arranged in the left area 60L (second area) corresponds to the second processing.

In addition, the sound information as the surrounding situation information referred to here is environmental sounds such as a nearby human voice, music sound flowing around, construction sound, and explosion sound, which occur accidentally. Sounds generated in advance are excluded.

For sounds that are set and generated in advance in the content, virtual objects arranged in a non-viewing area that are executed using the degree of attraction determined based on accidentally generated peripheral information described in the present embodiment No processing is performed.

In addition, even if it is an environmental sound that occurs accidentally, an environmental sound such as a crowd, which is a situation where a person does not pay attention even if the sound occurs, is exceptionally obtained from the sound information acquired by the sound sensor 23. The noise may be appropriately canceled.

The control of the process regarding the virtual object using the sound information acquired from the sound sensor 23 can be applied to, for example, an HMD that reproduces the game content. The user U can enjoy the game while wearing the HMD and freely moving in the real space. When playing a game while actually moving in the real space, environmental sounds other than sounds generated in advance in the game content may be generated in the real space.

In the present embodiment, the direction in which the sound source is located can be specified by installing a sound sensor on the HMD and detecting sound information of the surrounding situation. Since the user U is assumed to face a certain direction of the sound source, the HMD 1 performs a process of drawing a virtual object arranged in an area from a region in front of the user U to a certain region of the sound source at the time of sound generation. It is performed with higher priority than the rendering processing of the virtual object arranged in another area.

Thereby, the user U can experience a realistic game. Further, in the HMD 1, since the processing is executed while reducing the drawing processing load of the virtual object in the area where the degree of attraction of the user U is low, efficient processing is possible.

<Third embodiment>
In the third embodiment, a case where a detection result (smell information) acquired by the scent sensor 24 is used as the surrounding situation information will be described as an example.
FIG. 9 shows an example of determining the priority processing score for each area in a situation where the source of the smell 67 is in the right area 60R of the user U. The situation shown in FIG. 9 is a situation that induces the movement of the line of sight of the user U.

In the present embodiment, the priority processing points of the

areas

60F, 60B, 60R, and 60L are obtained based on the odor intensities detected by the odor sensors 24F, 24R, 24L, and 24B, respectively.

Generally, when a person senses an odor in the vicinity, he or she tends to turn in the direction that is considered to be the source of the odor. Therefore, in the present embodiment, the priority processing point is determined such that the priority processing point (attraction degree) of the region having the highest detected odor intensity is high.

The four odor sensors 24F, 24R, 24L, 24B detect the odor intensity around the user U. In the example shown in FIG. 9, since the source of the odor 67 is located in the right direction of the user U, the value of the odor intensity detected by the odor sensor 24R that detects the odor intensity in the right region 60R is the largest.

Further, during a period from the state in which the user U is facing the front area 60F to the state in which the user U is facing the right area 60R, which is the source of the odor, the line of sight of the user U is from the front area 60F of the user U when the odor is detected to the right area. Move over 60R.
Therefore, in the example shown in FIG. 9, the priority processing points P1 and P2 of the front area 60F and the right area 60R are determined to be 10 points higher than the other areas.

On the other hand, since the left area 60L, which is directly opposite to the right area 60R where the odor source is located, is assumed to be the area where the user U is unlikely to look, the priority processing score P4 is determined to be four. .
The rear area 60B is an area adjacent to the right area 60R, and is more likely to be in the field of view of the user U than the left area 60L. Therefore, the priority processing score P3 is determined to be five.

In FIG. 9, the peripheral information (second peripheral information) of the rear area 60B and the left area 60L (second area) of the real space is the peripheral information (first peripheral information) of the right area 60R (first area). ), It is harder to catch the eyes of the user U and the degree of attraction is low.

制御 The control of the processing related to the virtual object based on the odor information acquired by the odor sensor 24 can be applied to, for example, an HMD for a simulated disaster experience. The user U wearing the HMD can have a simulated experience of a fire by presenting a virtual object (display image) such as fire or smoke in a superimposed manner on the external world that is a real space.

Then, an odor is generated in the real space, and the direction in which the source of the odor is located is specified based on the detection result of the odor sensor mounted on the HMD. Here, as the odor generated in the real space, a gas that is harmless to the body and can be detected by the odor sensor can be used.

Since it is assumed that the user U faces a certain direction of the odor source, the rendering process of the virtual object arranged in the region from the front region 60F of the user U to the region with the odor source at the time of the odor detection is performed by another process. This is performed prior to the rendering processing of the virtual object arranged in the area.

This allows the user U to have a realistic simulated disaster experience. Further, in the HMD 1, since the processing is executed while reducing the drawing processing load of the virtual object in the area where the degree of attraction of the user U is low, efficient processing is possible.

<Fourth embodiment>
In the fourth embodiment, image information detected by the camera 25 mounted on the HMD 1 is acquired as peripheral situation information, and the priority processing score is determined using the image information.

Here, as an example, it is assumed that there are three users U equipped with the HMD, and these three people play using the same game content. It is assumed that the three users U are a user U1, a user U2, and a user U3, respectively. The HMD worn by each of the users U1 to U3 is the HMD1 having the same structure shown in FIG.

In the present embodiment, the processing in the HMD 1 worn by the user U1 will be described, but the same processing is performed in the other users U2 and U3. The HMDs 1 worn by the users U1 to U3 are configured to be able to communicate with each other, and can transmit / receive various information including image information acquired by each HMD to / from another HMD.

FIG. 10 is a schematic diagram for explaining the relationship between the area around the user U1 and the priority processing score when the surrounding situation of the user U1 is not a surrounding situation that induces the movement of the user U1's gaze.
FIG. 11 is a schematic diagram for explaining the determination of the priority processing score using the peripheral situation information.
FIG. 12 is a flowchart illustrating an example of a process related to a virtual object in the information processing apparatus according to the present embodiment.

As shown in FIG. 10, it is assumed that the real space 70 where the user U actually moves in the game content is divided into a plurality of areas A1 to A16, and the user U1 is located in the area A11.

In the example shown in FIG. 10, the priority processing point P10 is highest in the area A10 located in front (front) of the user U1, and is determined to be 10 points. In the areas A1, A2, A5, A6, A9, A10, A13, and A14 in front of the user U1, the priority processing points are determined such that the longer the distance from the user U1, the lower the priority processing points.

In FIG. 10, areas A7 and A15 immediately to the right or left of the user U1 have lower priority processing points than A10, which is a front area, and the priority processing points P7 and P15 are determined to be six.
Further, in the rightward direction and the leftward direction of the user U1, the priority processing points are determined so as to become lower as the distance from the user U1 increases, and the priority processing points P3 of the area A3 are determined to be three points.

In FIG. 10, the area A12 immediately behind the user U1 is determined as the area where the user U1 is least likely to see, and the priority processing score A12 is determined to be 0. Also, in the other areas A4, A8, and A16 behind the user U1, the priority processing score is determined to be relatively low, as the possibility that the user U1 sees is low.

In the present embodiment, the peripheral situation information used when determining the priority processing point is positional relationship information between each of the users U1 to U3 and the area to be subjected to the priority processing point determination. The positional relationship information between the user U and the priority processing point determination target area includes the line-of-sight information of the user U and distance information between the user U and the priority processing point determination target area. The line-of-sight information of the user U is information on the direction of the user U with respect to the priority processing point determination target area.

視 The line-of-sight information and distance information of each of the users U1 to U3 are detected based on images (image information) captured by the camera 25 mounted on the HMD 1 worn by each of the users U1 to U3.

In the example illustrated in FIG. 11, captured images (image information) acquired from the cameras 25 of the HMDs 1 of the other users U2 and U3 are image information in which a non-visual area of the user U1 is captured, and are in the visual field of the user U1. This is peripheral information different from the corresponding visual field information. The image captured by the camera 25 mounted on the HMD worn by the user U1 is visual field information corresponding to the visual field of the user U1.

In the example shown in FIG. 11, the user U1 is located in the area A11, the user U2 is located in the area A4, and the user U3 is located in the area A2. The virtual object A is an object arranged in the area A7, and the virtual object B is an object arranged in the area A15. The users U2 and U3 pay attention to the virtual object A. The virtual object A and the virtual object B are objects located around the user U1 and arranged in the non-viewing area of the user U1.

In a direction in which many users are paying attention among a plurality of users U participating in the game, there is a high possibility that a virtual object to which the user pays attention exists, and the user does not face a certain direction of the virtual object at present. It is highly likely that user U will see the virtual object.
In the example shown in FIG. 11, the users U2 and U3 are paying attention to the virtual object A, and the user U1 who is not facing the area where the virtual object A is present at the present time is highly likely to see the virtual object A.

In the present embodiment, for each area around the user U1, all the users U (users U1 to U3 in FIG. 11) playing in the same real space using the same game content and the user U and the priority processing point determination target are determined. The priority processing score is determined using the positional relationship information with the area.

As described above, the positional relationship information is information on the user's orientation (the line-of-sight information of the user U) with respect to the priority processing point determination target area and distance information between the user U and the priority processing point determination target area. Here, a high priority processing score is determined for an area where many users are looking and an area closer to the user is closer.

In the example illustrated in FIG. 11, the virtual object A around the user U1 is located in the area A7 where the users U2 and U3 other than the user U1 are focused, and is closer to the users U2 and U3 than the virtual object B. In position.
On the other hand, the virtual object B around the user U1 is not noticed by any user, and is located in an area A15 farther from the positions of the users U2 and U3 than the virtual object A.

In such an example, the area A7 in which the virtual object A is arranged has a higher priority processing score than the area A15 in which the virtual object B is arranged.
Hereinafter, the determination of the priority processing score will be described in detail with reference to the flowchart of FIG.

First, as shown in FIG. 12, the camera 25 mounted on the HMD 1 worn by each of the users U1 to U3 as the peripheral situation information via the communication control unit 11 by the peripheral situation information management unit 12 of the HMD 1 of the user U1. The information of the image photographed at is obtained (S31).
From the image information, the distance information between the user U and the priority processing point determination target area, and the user's direction information (gaze information) with respect to the priority processing point determination target area are obtained for each of the users U1 to U3.

Next, the priority processing point determination unit 15 determines the priority processing point (attraction degree) for each peripheral area of the user U1 from the image information based on the above-described distance information and line-of-sight information (S32).

In S32, the priority processing points of the priority processing point determination target area are calculated for each user U, and the total of the priority processing points calculated for each user is set as the priority processing point of the priority processing point determination target area.

Here, the determination of the priority processing points of the area A7 where the virtual object A is arranged and the area A11 where the virtual object B is arranged in FIG. 11 will be specifically described as an example. The number of processing points is determined.

The priority processing points of the area A7 (A11) where the virtual object A (virtual object B) is arranged in FIG. 11 are obtained from the sum of the priority processing points obtained for each of the users A1 to A3. The priority processing score of each user is obtained by the following equation.

In the above equation, P is the number of priority processing points, l (m) is the distance between the camera 25 mounted on the HMD 1 worn by each user U and the area for which priority processing points are to be determined, and θ (rad) is the horizontal plane. The angle at which the priority processing point determination target area is located from the front of each user U, and E indicates a priority processing point coefficient.

Here, a case will be described in which the determination of the priority processing point in the processing relating to the drawing of the virtual object on the HMD 1 worn by the user U1 is performed. Therefore, the priority processing point coefficient E is a coefficient based on the user U1. Since the priority processing coefficient E is determined based on the positional relationship between the user U and the priority processing point determination target area, the priority processing coefficient differs for each user U even if the priority processing point determination target area is the same.

In the determination of the priority processing score of the area A7 in which the virtual object A is arranged shown in FIG. 11, if the distance 1 between the user U1 and the area A7 is 1, the distance between the users U2 and U3 and the area A7 is a square root of 2. expressed.
The angle between the user U1 and the area A7 can be represented by π / 2 radians (about 1.57 radians), and the angle between the users U2 and U3 and the area A7 can be represented by 0 radians.

Here, when the priority processing point coefficient E is set to 10 and the priority processing point of the area A7 viewed from each user is calculated, P = 1 for the user U1, and P = 1.41 for the users U2 and U3.
Therefore, the priority processing score PA in the processing related to the drawing of the virtual object A arranged in the area A7 on the HMD 1 worn by the user U1 is the sum of the priority processing scores determined for each user U.
PA = 1 + 1.41 + 1.41 = 3.82
Becomes

On the other hand, in the determination of the priority processing score of the area A15 where the virtual object B is arranged, the distance 1 between the user U1 and the area A15 is 1, and the distance between the users U2 and U3 and the area A15 is represented by a square root of 10. .
The angle between the user U1 and the area A15 can be represented by π / 2 radians (about 1.57 radians), and the angle between the users U2 and U3 and the area A15 can be represented by about 0.46 radians.

Here, when the priority processing point coefficient of the area A15 viewed from each user is calculated with the priority processing point coefficient E set to 10, P = 1 for the user U1, and P = 0.38 for the users U2 and U3.
Therefore, the priority processing score Pb in the processing related to the drawing of the virtual object B arranged in the area A15 on the HMD 1 worn by the user U1 is the sum of the priority processing scores determined for each user U.
Pb = 1 + 0.38 + 0.38 = 1.76
Becomes

Next, based on the priority processing score determined by the priority processing score determination unit 15, the rendering processing load determination unit 16 determines the load of the processing related to the rendering of the virtual object placed in the peripheral area of the user U <b> 1. A virtual object generation process is executed by the output image generation unit 17 based on the processed content (S33).

In the present embodiment, the threshold value Pt of the priority processing point for determining the processing load is set to 2.0.
The rendering processing load determination unit 16 performs a comparatively heavy load such that the virtual object placed in the area where the priority processing score determined by the priority processing score determination unit 15 is less than the threshold is moved once every 100 msec. It is determined that a low process X is performed.
The rendering processing load determination unit 16 performs a comparatively heavy load such as moving the virtual object placed in the area where the priority processing point calculated by the priority processing point determination unit 15 is equal to or greater than the threshold value once every 16 msec. It is determined that high processing Y is performed.
Here, an example in which the processing load is divided into two stages has been described, but the present invention is not limited to this.

In the above example, since the value of Pa is 3.82 for the virtual object A arranged in the area A7, the drawing processing load determination unit 16 determines to perform the processing Y. For the virtual object B arranged in the area A15, the value of PB is 1.76, so that it is determined that the process X is performed.

{Circle around (2)} The output image generation unit 17 executes a virtual object generation process based on the processing content determined by the drawing processing load determination unit 16.

Next, the virtual object generated by the output image generation unit 17 is converted into an image signal so that it can be displayed on the display 31 by the output image control unit 18 and output to the output unit 30 (S34).

As a result, the processing load related to the drawing processing of the virtual object A that is likely to be noticed by the user U1 increases, and the processing load related to the drawing processing of the virtual object B that is unlikely to be noticed by the user U1 decreases. Good processing can be performed.

Here, an example in which a captured image of a camera mounted on the HMD is used has been described, but a captured image captured by an external camera as an external device disposed in a real space that is not mounted on the HMD may be used. The external camera may be fixed or may be configured to be movable, as long as the external camera can acquire image information having positional relationship information between the user U and the priority processing point determination target area.

In this manner, the positional relationship information between the user U and the priority processing point determination target area is obtained from the image information, and based on this information, the processing related to the drawing processing of the virtual object arranged in the non-visual area of the user U1 is controlled. You may.

As described above, in the information processing apparatus (HMD) according to the present technology, the processing regarding the virtual object arranged in the non-viewing area is performed based on the degree of attraction determined based on the peripheral information of the real space acquired from the sensor. Are determined, and the process related to the virtual object is executed.
Thereby, the processing relating to the virtual object arranged in the area with a high degree of attraction can be performed prior to the processing relating to the virtual object arranged in the area with a low degree of attraction, and efficient processing can be performed. .

<Other embodiments>
Embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

For example, in the above-described embodiment, an example has been described in which 360 degrees around the user are divided into four parts in the left-right direction as viewed from the user, but the division may be further made in the vertical direction.

In general, the visual field of a normal person is said to be about 60 degrees on the upper side and about 70 degrees on the lower side in one eye.
As an example, each of the four regions divided in the left-right direction in the above-described embodiment may be further divided into three in the up-down direction, and divided into 12 regions as a whole. When each area obtained by dividing the 180-degree area in the vertical direction into three is defined as an upper area, an intermediate area, and a lower area in order from the top, the angle defining the range in the vertical direction of the upper area is 30 degrees, and the vertical direction of the intermediate area is May be 130 degrees, and the angle that defines the range in the vertical direction of the lower region may be 20 degrees.

In the case of further dividing in the vertical direction as described above, a sensor is mounted on the HMD so that the surrounding situation in each area can be individually detected.

For example, when a sound is generated at the feet or overhead of the user U, it is assumed that the user U looks at the feet or overhead where the sound is generated.
In such a case, by providing a sound sensor that detects sound information as the surrounding situation information in the up-down direction, as in the above-described embodiment, based on the degree of attraction determined based on the detection result of the sound sensor, It is possible to determine the priority of processing of the virtual object arranged in the non-viewing area in the direction. Accordingly, it is possible to control the processing relating to the virtual object arranged in the non-viewing area in the vertical direction, and it is possible to perform efficient processing.

Further, in the above-described embodiment, an example is described in which sensors such as the illuminance sensor 22, the sound sensor 23, and the odor sensor 24 which detect the surrounding situation are mounted on the HMD. It may be installed as an external device different from.
The external device may be installed so as to be able to detect the surrounding situation of the user U. As an example, a wristband-type device equipped with a sensor for detecting the surrounding situation of each of the right area and the left area of the user U is used. be able to.
The external device and the control unit of the HMD are configured to be able to communicate with each other, and the control unit of the HMD is configured to be able to acquire the detection result of the external device.

Further, in each of the above-described embodiments, an example in which the drawing process of the virtual object arranged in the non-viewing area is controlled using the illuminance information, the sound information, the odor information, or the image information as the peripheral situation information of the user U. However, the drawing process of the virtual object may be controlled using a combination of these pieces of information.

Further, in the above-described embodiment, when judging the number of priority processing points, an example is given in which both the surrounding situation information and spatial information such as the position and orientation information of the camera are used, but only the surrounding situation information may be used.
Since the illuminance sensor 22, the sound sensor 23, and the odor sensor 24 for detecting the surrounding state are respectively installed to detect the

respective areas

60F, 60B, 60R, and 60L, for example, the sensor for detecting the surrounding state of the front area 60F is used. If specified, the orientation of the user U, that is, the position and orientation of the camera can be grasped.

Note that the present technology may have the following configurations.
(1)
Determining the degree of attraction for each of the first peripheral information of the first area and the second peripheral information of the second area in the real space obtained from the sensor, which is different from the visual field information corresponding to the user's visual field; Based on the degree of attraction, the priority of the first processing for the first virtual object arranged in the first area and the priority of the second processing for the second virtual object arranged in the second area are determined. An information processing device including a control unit for determining.

(2)
The information processing apparatus according to the above (1),
The control unit may be configured to perform the first processing with higher priority than the second processing when determining a lower degree of interest for the second peripheral information than the first peripheral information. An information processing device that determines the priority.

(3)
The information processing apparatus according to (2),
The control unit determines the priority such that the processing load of the first process related to the drawing of the first virtual object is higher than that of the second process related to the drawing of the second virtual object. Information processing device.

(4)
The information processing apparatus according to any one of (1) to (3),
The sensor includes a first illuminance sensor that detects illuminance of the first area as the first peripheral information, and a second illuminance sensor that detects illuminance of the second area as the second peripheral information. Information processing device including.

(5)
The information processing apparatus according to (4),
The information processing device, wherein the control unit determines the degree of attraction in consideration of an environment of the real space.

(6)
The information processing apparatus according to (5),
The environment of the real space is outdoors under the sunshine.

(7)
The information processing apparatus according to the above (6),
When at least one of the first illuminance sensor and the second illuminance sensor detects an illuminance in the real space to which sunlight having an illuminance higher than an illuminance in a region corresponding to the user's field of view is applied. The control unit sets the degree of attraction to the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to which the sunlight is irradiated, using the illuminance of the area corresponding to the user's field of view and the illuminance sensor. An information processing apparatus that determines that an illuminance difference between the detected sunlight and the illuminance in the real space that is equal to or larger than the threshold is lower than a case where the illuminance difference is smaller than the threshold.

(8)
The information processing apparatus according to (5),
The environment of the real space is an indoor information processing device.

(9)
The information processing apparatus according to (8),
The information processing device, wherein the control unit determines an eye-catching degree of the peripheral information acquired from the illuminance sensor that detects an illuminance of the real space to be irradiated with sunlight so that the priority becomes the lowest.

(10)
The information processing apparatus according to (8),
When at least one of the first illuminance sensor and the second illuminance sensor detects the illuminance of the real space to which the light other than the sunlight is irradiated, the control unit sets the light other than the sunlight to The attractiveness of the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to be illuminated, the illuminance of the area corresponding to the user's field of view and the light other than the sunlight detected by the illuminance sensor An information processing apparatus which determines that an illuminance difference between the illuminance and the illuminance in the real space to be irradiated is lower when the illuminance difference is less than the threshold than when the illuminance difference is equal to or greater than the threshold.

(11)
The information processing apparatus according to any one of (1) to (10),
The first sensor detects a loudness of the sound in the first area as the first peripheral information, and detects a loudness of the sound in the second area as the second peripheral information. An information processing apparatus including a second sound sensor that performs the processing.

(12)
The information processing apparatus according to (11),
The control unit, when the loudness of the sound in the second area is smaller than the loudness of the sound in the first area, sets the degree of attraction for the second peripheral information to the degree of attraction for the first peripheral information. Information processing device that also determines low.

(13)
The information processing apparatus according to any one of (1) to (12),
The sensor detects a odor intensity of the first area as the first peripheral information, and detects an odor intensity of the second area as the second peripheral information. An information processing device including a second odor sensor.

(14)
The information processing apparatus according to (13),
The control unit, when the odor intensity of the second area is weaker than the odor of the first area, sets the degree of attraction to the second peripheral information to be lower than the degree of attraction to the first peripheral information. Judge information processing device.

(15)
The information processing apparatus according to any one of (1) to (14),
The information processing device, wherein the sensor is a camera that acquires image information around the user as the surrounding information.

(16)
The information processing apparatus according to (15),
The image information includes positional relationship information between the first area and the second area and the user,
The information processing device, wherein the control unit determines the degree of attraction using the positional relationship information.

(17)
The information processing apparatus according to any one of (1) to (16),
The information processing device can be mounted on the head of the user, and can present the first virtual object and the second virtual object in the field of view of the user while allowing the user to visually recognize the outside world. An information processing device that is a head-mounted display configured.

(18)
Acquiring, from a sensor, first peripheral information of a first area and second peripheral information of a second area in a real space different from visual field information corresponding to a visual field of a user;
Judgment degree of each of the first peripheral information and the second peripheral information is determined,
Based on the degree of attraction, the priority of the first processing for the first virtual object arranged in the first area and the priority of the second processing for the second virtual object arranged in the second area are determined. Judgment Information processing method.

(19)
Acquiring, from a sensor, first peripheral information of a first area and second peripheral information of a second area in a real space different from visual field information corresponding to a visual field of a user;
Determining the degree of attraction of each of the first peripheral information and the second peripheral information;
Based on the degree of attraction, the priority of the first processing for the first virtual object arranged in the first area and the priority of the second processing for the second virtual object arranged in the second area are determined. A program for causing an information processing apparatus to execute a process including a determining step.

1. HMD (information processing device)
10: control unit 22: illuminance sensor (sensor)
23 ... Sound sensor (sensor)
24 ... Odor sensor (sensor)
25 ... Camera (sensor)
60, 70: real space 60B: rear area (first area, second area)
60L... Left side area (first area, second area)
60R right side area (first area, second area)
65 ... sun 66 ... explosion sound (sound)
67: smell A: virtual object A (first virtual object)
B: virtual object B (second virtual object)
A7... Area A7 (first area)
A15... Area A15 (second area)
U, U1 to U3: User P1 to P16: Priority processing points (attraction level)

Claims

Determining the degree of attraction for each of the first peripheral information of the first area and the second peripheral information of the second area in the real space obtained from the sensor, which is different from the visual field information corresponding to the user's visual field; Based on the degree of attraction, the priority of the first process for the first virtual object arranged in the first area and the priority of the second process for the second virtual object arranged in the second area are determined. An information processing device including a control unit for determining.
The information processing device according to claim 1,
The control unit is configured to perform the first process with higher priority than the second process when determining a lower degree of interest for the second peripheral information than the first peripheral information. An information processing device that determines the priority.
The information processing apparatus according to claim 2, wherein
The control unit determines the priority such that the first processing relating to the rendering of the first virtual object has a higher processing load than the second processing relating to the rendering of the second virtual object. Information processing device.
The information processing apparatus according to claim 3, wherein
A first illuminance sensor for detecting illuminance of the first area as the first peripheral information; and a second illuminance sensor for detecting illuminance of the second area as the second peripheral information Information processing device including.
The information processing apparatus according to claim 4, wherein
The information processing device, wherein the control unit determines the degree of attraction in consideration of an environment of the real space.
The information processing apparatus according to claim 5, wherein
The environment of the real space is outdoors under sunshine.
The information processing device according to claim 6,
When at least one of the first illuminance sensor and the second illuminance sensor detects the illuminance of the real space to which sunlight having an illuminance higher than the illuminance of an area corresponding to the user's field of view is applied. The control unit sets the degree of attraction to the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to which the sunlight is irradiated, with the illuminance of the area corresponding to the user's field of view and the illuminance sensor. An information processing apparatus which determines that an illuminance difference between the detected illuminance and the illuminance in the real space to which the sunlight is applied is lower when the illuminance is equal to or larger than the threshold than when the illuminance difference is smaller than the threshold.
The information processing apparatus according to claim 5, wherein
The environment of the real space is indoors.
The information processing apparatus according to claim 8, wherein:
The information processing device, wherein the control unit determines an eye-catching degree of the peripheral information acquired from the illuminance sensor that detects an illuminance of the real space to be irradiated with sunlight so that the priority becomes the lowest.
The information processing apparatus according to claim 8, wherein:
When at least one of the first illuminance sensor and the second illuminance sensor detects the illuminance of the real space to which the light other than sunlight is applied, the control unit sets the light other than the sunlight to light. The attractiveness of the peripheral information acquired from the illuminance sensor that detects the illuminance of the real space to be illuminated, the illuminance of the area corresponding to the user's field of view and the light other than the sunlight detected by the illuminance sensor are An information processing apparatus which determines that an illuminance difference between the illuminance and the illuminance in the real space to be irradiated is lower when the illuminance is less than the threshold than when the illuminance difference is greater than or equal to the threshold.
The information processing apparatus according to claim 3, wherein
The first sensor detects a loudness of the sound in the first area as the first peripheral information, and detects a loudness of the sound in the second area as the second peripheral information. An information processing apparatus including a second sound sensor that performs the processing.
The information processing apparatus according to claim 11, wherein
The control unit, when the loudness of the sound in the second area is smaller than the loudness of the sound in the first area, sets the degree of attraction for the second peripheral information to the degree of attraction for the first peripheral information. Information processing device that also determines low.
The information processing apparatus according to claim 3, wherein
The sensor detects a odor intensity of the first area as the first peripheral information, and detects an odor intensity of the second area as the second peripheral information. An information processing device including a second odor sensor.
The information processing apparatus according to claim 13, wherein
When the intensity of the odor of the second area is weaker than the odor of the first area, the controller sets the degree of attraction to the second peripheral information to be lower than the degree of attraction to the first peripheral information. Judge information processing device.
The information processing apparatus according to claim 3, wherein
The information processing device, wherein the sensor is a camera that acquires image information around the user as the surrounding information.
The information processing apparatus according to claim 15, wherein
The image information includes information on a positional relationship between each of the first area and the second area and the user,
The information processing device, wherein the control unit determines the degree of attraction using the positional relationship information.
The information processing apparatus according to claim 3, wherein
The information processing apparatus can be mounted on the head of the user, and can present the first virtual object and the second virtual object in the field of view of the user while allowing the user to visually recognize the outside world. An information processing device that is a head-mounted display configured.
Acquiring, from a sensor, first peripheral information of a first area and second peripheral information of a second area in a real space different from visual field information corresponding to a visual field of a user;
Determining the degree of attraction of each of the first peripheral information and the second peripheral information;
Based on the degree of attraction, the priority of the first process for the first virtual object arranged in the first area and the priority of the second process for the second virtual object arranged in the second area are determined. Judgment Information processing method.
Acquiring, from a sensor, first peripheral information of a first area and second peripheral information of a second area in a real space different from visual field information corresponding to a visual field of a user;
Determining the degree of attraction of each of the first peripheral information and the second peripheral information;
Based on the degree of attraction, the priority of the first process for the first virtual object arranged in the first area and the priority of the second process for the second virtual object arranged in the second area are determined. A program for causing an information processing apparatus to execute a process including a determining step.