JP6984645B2 - Image processing equipment, image processing methods, and image systems - Google Patents

Description

The present technology relates to an image processing device, an image processing method, and an image system, and more particularly to an image processing device, an image processing method, and an image system capable of reducing user discomfort such as "VR sickness". ..

In recent years, head-mounted displays and head-up displays have become capable of presenting images with a wide viewing angle, and the direction and orientation of the images to be displayed naturally change and follow (tracking) according to the posture of the body or head. ), It has become possible to realize a more realistic first-person perspective experience. The first-person perspective is becoming more and more popular in the world of video games, especially as first-person shooters (F.P.S.).

On the other hand, it is known that there are cases where some users experience discomfort called "VR sickness" or "video sickness" when experiencing virtual reality (VR) using a head-mounted display or head-up display. Has been done. It is said that this occurs when there is a discrepancy between the sensation that the user experiences or predicts from experience and the actual sensation (for example, visual stimulus).

A device for reducing the discomfort of "VR sickness" and "video sickness" has been proposed (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-31439

However, there is a further demand for measures against such discomfort such as "VR sickness".

This technology was made in view of such a situation, and makes it possible to reduce user discomfort such as "VR sickness".

The image processing device of one aspect of the present technology generates an avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world as the first presented image to be presented to the user, and detects the movement of the user. When there is a posture difference between the actual posture of the user based on the result and the posture of the avatar, an image generation unit that generates a second presented image different from the viewpoint image of the avatar is provided , and the posture is the head position. The second presented image corresponds to a posture in which at least one of the avatar's posture and the elevation is set as the value of the user's actual posture. It is an image .

In the image processing method of one aspect of the present technology, the image processing apparatus generates an avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world as the first presented image to be presented to the user, and the user's image processing method is performed. When there is a posture difference between the actual posture of the user based on the detection result of detecting the movement and the posture of the avatar, a second presentation image different from the viewpoint image of the avatar is generated, and the posture is the head position. The second presented image corresponds to a posture in which at least one of the avatar's posture and the elevation is set as the value of the user's actual posture. It is an image .

The image system of one aspect of the present technology generates an avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world as the first presented image to be presented to the user, and the detection result of detecting the movement of the user. When there is a posture difference between the actual posture of the user and the posture of the avatar based on the above, an image generation unit that generates a second presentation image different from the avatar viewpoint image, the first presentation image, and the second presentation image. The posture is defined by a head position and a line-of-sight direction represented by an azimuth angle and an elevation angle, and the second presentation image is the orientation of the avatar's posture. It is an image corresponding to the posture in which at least one of the angle or the elevation angle is set as the value of the actual posture of the user .

In one aspect of the present technology, an avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presentation image to be presented to the user, and is based on the detection result of detecting the movement of the user. When there is a posture difference between the actual posture of the user and the posture of the avatar, a second presentation image different from the avatar viewpoint image is generated. The posture is defined by a head position and a line-of-sight direction represented by an azimuth and an elevation angle, and in the second presented image, at least one of the azimuth or elevation angle of the avatar's posture is the actual posture of the user. The image corresponds to the posture set to the value of.

The image processing device of one aspect of the present technology can be realized by causing a computer to execute a program.

Further, in order to realize the image processing apparatus of one aspect of the present technology, a program to be executed by a computer can be provided by transmitting via a transmission medium or by recording on a recording medium.

The image processing device may be an independent device or an internal block constituting one device.

According to one aspect of the present technology, it is possible to reduce user discomfort such as "VR sickness".

The effects described herein are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure which shows the structural example of one Embodiment of the image system to which this technique is applied. It is a figure which shows the example of the content image of a boxing game. It is a block diagram which shows the detailed configuration example of the image system of FIG. It is a figure explaining the coordinate system of a virtual world. It is a figure explaining three postures of a user posture, an avatar posture, and a presentation image posture. It is a flowchart explaining the presentation image generation process. It is a figure explaining the example of the content which shares a virtual world with a plurality of users. It is a block diagram which shows the structural example of one Embodiment of the computer to which this technique is applied.

Hereinafter, embodiments for implementing the present technology (hereinafter referred to as embodiments) will be described.

<Image system configuration example>
FIG. 1 shows a configuration example of an embodiment of an image system to which the present technology is applied.

The image system 1 of FIG. 1 is composed of a head-mounted display 11 (hereinafter referred to as HMD 11), an image processing device 12, and a controller 13.

The HMD 11 displays an image on the virtual world generated by the image processing device 12 and presents it to the user. The image processing device 12 generates an image to be displayed on the HMD 11 and supplies the image to the HMD 11. The controller 13 includes a plurality of operation buttons (not shown), accepts the user's operation, and supplies a predetermined instruction according to the user's operation to the image processing device 12.

Between the HMD 11 and the image processing device 12, between the image processing device 12 and the controller 13, for example, a wired cable such as HDMI (registered trademark) (High Definition Multimedia Interface) or MHL (Mobile High-definition Link), or a wired cable, or It is connected by wireless communication such as Wi-Fi (Wiless Fidelity), wirelessHD and Miracast.

In this embodiment, for example, the image processing apparatus 12 displays an image as shown in FIG. The image of FIG. 2 is an image of a boxing game (content) from a first-person viewpoint with a user wearing the HMD 11 as a player. The image displayed on the HMD 11 may be a 2D image, or a 3D image that displays an image for the right eye to be shown to the user's right eye and an image for the left eye to be shown to the user's left eye for three-dimensional visual recognition. It may be an image.

In the image system 1 configured as described above, the image processing device 12 reduces discomfort such as "VR sickness" in a state where the user visually recognizes the image displayed on the HMD 11 and experiences the virtual world. An image is generated so as to be displayed on the HMD 11.

The HMD 11, the image processing device 12, and the controller 13 may be combined as necessary, and the image system 1 may be composed of one or two devices.

<Detailed configuration example of image system>
FIG. 3 is a block diagram showing a detailed configuration example of the image system 1.

The HMD 11 includes a sensor 41 and a display unit 42.

The sensor 41 is composed of a combination of one or more sensor elements such as a gyro sensor, an acceleration sensor, and a geomagnetic sensor, and detects the position and direction of the user's head. In the present embodiment, the sensor 41 is a sensor capable of detecting a total of 9 axes of a 3-axis gyro sensor, a 3-axis acceleration sensor, and a 3-axis geomagnetic sensor. The sensor 41 supplies the detected result to the image processing device 12.

The display unit 42 is composed of, for example, an organic EL (Electro-luminescence) element, a liquid crystal display, or the like, and displays a predetermined image based on an image signal supplied from the image processing device 12.

The HMD 11 may include a speaker that outputs voice, a microphone that acquires the voice of the user, and the like.

The controller 13 includes an operation unit 81 including a plurality of operation buttons, receives the user's operation, and supplies a predetermined instruction according to the user's operation to the image processing device 12.

The image processing device 12 has at least a storage unit 61 and an image generation unit 62, and the image generation unit 62 includes an avatar motion control unit 71, a user posture detection unit 72, a posture difference calculation unit 73, and a presentation image generation unit 74. It is composed of.

The storage unit 61 is composed of, for example, a hard disk, a non-volatile memory, or the like, and stores a program for controlling the operation of the image processing device 12, a content (program) for displaying an image on the HMD 11, and the like.

The avatar motion control unit 71 controls the motion (posture) of the avatar corresponding to the user in the image (content image) on the virtual world displayed on the display unit 42. In controlling the movement (posture) of the avatar, the actual posture of the user (actual posture of the user) detected by the user posture detecting unit 72 is also taken into consideration. The operation of the avatar changes, for example, depending on the operation selected by the user, the action performed by the user, or the like in the content image of the virtual world displayed on the display unit 42.

When an avatar other than the player himself / herself appears in the image displayed on the display unit 42, for example, as in the opponent of the boxing game shown in FIG. 2, the avatar motion control unit 71 moves the avatar of another person ( Posture) is also controlled. In the following, in order to distinguish between the avatar of the user himself / herself and the avatar of another person, the avatar of another person is referred to as the avatar of another person, and the avatar of the user himself / herself is referred to as the avatar of the user himself / herself. do.

The user posture detection unit 72 detects the actual posture of the user based on the detection result of the position and direction of the head of the user supplied from the sensor 41.

The posture difference calculation unit 73 calculates the posture difference between the posture of the avatar determined by the avatar motion control unit 71 and the actual posture of the user detected by the user posture detection unit 72.

The presentation image generation unit 74 generates a presentation image to be presented to the user, which is an image of the content acquired from the storage unit 61, and supplies the presentation image to the display unit 42. The presented image shows the movements of the user's own avatar and another avatar determined by the avatar motion control unit 71, the actual posture of the user determined by the user posture detection unit 72, and the avatar determined by the posture difference calculation unit 73. It is generated based on the posture difference between the posture and the actual posture of the user. In a normal scene, the presentation image generation unit 74 adjusts the posture of the avatar to the actual posture of the user, and generates a presentation image (first presentation image) so that the image from the viewpoint of the avatar is presented to the user. Under predetermined conditions, in order to reduce user discomfort such as "VR sickness", an image different from the image from the avatar viewpoint is generated as a presentation image (second presentation image) and displayed on the display unit 42. ..

<Explanation of the presented image>
The presentation image generated by the presentation image generation unit 74 will be further described with reference to FIGS. 4 and 5.

First, with reference to FIG. 4, a virtual world in the content displayed on the display unit 42 and a coordinate system expressing the posture of the avatar on the virtual world will be described.

As shown in A of FIG. 4, assuming that the virtual world in the content displayed on the display unit 42 is a three-dimensional space consisting of the X-axis, the Y-axis, and the Z-axis, the posture of the avatar is a three-dimensional position ( It can be expressed by five parameters (x, y, z, θ, φ) consisting of x, y, z) and a two-dimensional direction (θ, φ).

Here, the three-dimensional position (x, y, z) corresponds to the position of the avatar's head in the virtual world, and the two-dimensional direction (θ, φ) corresponds to the line-of-sight direction (head) of the avatar in the virtual world. Corresponds to the direction in which the part faces). As shown in B of FIG. 4, the direction θ is an angle formed by the line-of-sight direction with respect to a predetermined reference axis (Z axis in the example of B in FIG. 4) on the XZ plane, and is a so-called azimuth angle. As shown in C of FIG. 4, the direction φ is an angle in the Y-axis direction formed by the line-of-sight direction with respect to the XZ plane, and is a so-called elevation angle. Therefore, the posture in the present specification includes not only the head position on the virtual world but also the line-of-sight direction.

In the virtual world of the content displayed on the display unit 42, _{the three postures of the user posture pos u} , the avatar posture pos _a , and the presented image posture pos _v are defined as follows.
User posture pos _u = (x _u , _yu , z _u , θ _u , φ _u )
Avatar attitude _{pos a = (x a, y} a, z a, θ a, φ a)
Presented image posture pos _v = (x _v , y _v , z _v , θ _v , φ _v )

The user position pos _u, is the actual posture of the user as determined by the detection result supplied from the sensor 41 that is projected onto the virtual world. The avatar posture pos _a is a representation of the avatar's posture in coordinates on the virtual world. The presented image posture pos _v is a virtual posture corresponding to the presented image displayed on the display unit 42.

With reference to FIG. 5, _{the three postures of the user posture pos u} , the avatar posture pos _a , and the presented image posture pos _v will be specifically described.

This will be described with an example of the contents of the boxing game shown in FIG.

For example, in a normal situation, the posture of the avatar is controlled according to the actual posture of the user detected by the user posture detecting unit 72, so that the avatar posture pos _a and the user posture pos _u are the same. That is, the avatar posture pos _a = the user posture pos _u . Further, since the presented image displayed on the display unit 42 is an image of the virtual world from the viewpoint of the avatar, the presented image posture pos _v = avatar posture pos _a .

Therefore, in normal situations where no special treatment is done to reduce "VR sickness",
Presented image posture pos _v = avatar posture pos _a = user posture pos _u
Will be.

Next, as an example, assume a scene in which the down feather (falls to the ground) is brought down by the punch of the opponent.

Assuming that the actual user is not lying on the ground and is in a standing posture as shown in the posture 101 of FIG. 5, the user posture pos _u = (x _u , _yu , z _{u) calculated by the user posture detecting unit 72.} , Θ _u , φ _u ) is the posture 101.

On the other hand, avatar attitude is controlled by the avatar action control unit _{71 pos a = (x a,} y a, z a, θ a, φ a) is shown in Figure 5, the orientation 102 fallen on the ground.

In a normal scene, as described above, the image of the virtual world from the viewpoint of the avatar is used as the presentation image, but if the presentation image is changed significantly according to the sudden movement of the avatar, the user causes "VR sickness". There is a risk.

Therefore, presenting image generator 74, rather than the image corresponding to the avatar position pos _a (position 102 in FIG. 5), for example, an image 111 corresponding to the position 103 close to the actual posture of the user (position 101 in FIG. 5) It is generated as a presentation image and displayed on the display unit 42. The image 111 as the presentation image is an image in which the change in the visual field is gradual as compared with the image corresponding to _{the avatar posture pos a.} The posture 103 corresponding to the image 111 to be the presentation image is the presentation image posture pos _v = (x _v , y _v , z _v , θ _v , φ _v ).

In the example of FIG. 5, in the scene where the processing for reducing VR sickness is performed, the presented image posture pos _v is different from both the avatar posture pos _a and the user posture pos _u , but the presented image is at least a normal scene. It may be an image different from the avatar viewpoint image which is the presentation image of. For example, the presented image may be an image corresponding to the actual posture of the user.

<Presentation image generation process>
Next, the presentation image generation process by the image processing device 12 will be described with reference to the flowchart of FIG. This process is started, for example, when an operation of displaying a predetermined content image on the display unit 42 of the HMD 11 is performed.

First, in step S1, the avatar motion control unit 71 determines the _{avatar motion (avatar posture pos a) in the virtual world displayed on the display unit 42.} In addition, when another person's avatar other than the player himself appears in the virtual world, the action (posture) of the other person's avatar is also determined. After step S1, the determination of the action of the avatar is always executed according to the content of the content.

In step S2, the user posture detector 72, supplied from the sensor 41, based on the sensor detection result indicating the position and direction of the user's head, to detect the actual attitude pos _u users. Detection of the actual attitude pos _u of the user based on the sensor detection results is performed at all times subsequent. Actual attitude pos _u of the detected user is supplied to the avatar action control unit 71, avatar behavior (posture) are according to the user's operation.

In step S3, the presentation image generation unit 74 generates an avatar viewpoint image, which is an image from the viewpoint of the avatar, as a presentation image, and supplies the image to the display unit 42 of the HMD 11 for display. The presented image posture pos _v here is equal to the avatar posture pos _a and the user posture pos _u (presented image posture pos _v = avatar posture pos _a = user posture pos _u ).

In step S4, the posture difference calculation unit 73 calculates the posture difference between the avatar posture pos _{a determined by the avatar motion control unit 71 and the user's actual posture pos u} detected by the user posture detection unit 72. It is supplied to the presentation image generation unit 74.

In step S5, the presentation image generation unit 74 _{determines whether the posture difference between the avatar posture pos a} and the user's actual posture pos _u is equal to or greater than a predetermined first range.

Here, as a first range in step S5, for example, a threshold thres azimuthal theta _theta and sets a threshold thres _phi elevation phi, either of the following conditions of Equation (1) or (2) When satisfied, the presentation image generation unit 74 can determine that the posture difference between _{the avatar posture pos a} and the user's actual posture pos _{u is equal to or greater than the first range.} | D | in the equations (1) and (2) represents the absolute value of d.
｜ θ _a −θ _u ｜ ＞ thres _θ・ ・ ・ (1)
｜ φ _{a −} φ _u ｜ ＞ thres _φ・ ・ ・ (2)

_{The thres θ in the} equation (1) is the threshold value of the azimuth angle θ, and the thres _{φ in the} equation (2) is the threshold value of the elevation angle φ. Arbitrary values can be set for the threshold value thres _θ and the threshold value thres _φ , but they can be set to, for example, 30 degrees. This value of 30 degrees moves the line of sight by moving only the eyeball if it is within 30 degrees when the average person looks at an object in a direction deviated from the front, but the direction deviated further. It is a numerical value that when you look at an object in, you naturally move your head.

_{Further, the threshold value thres θ} and the threshold value thres _φ can be set at one time when strict determination conditions are used to prevent sickness. This is because the range of the central region of the retina called the fovea, which is used when gazing at the details of things, is said to be about 1 degree in the average human being. The threshold value thres _θ and the threshold value thres _φ can be changed to the values desired by the user by using a setting screen or the like. By changing these thresholds, the degree of sickness prevention can be controlled.

The above-mentioned determination conditions are mainly conditions for determining that the direction of the head has been moved, but when the case where the avatar can be moved greatly throughout the body is also taken into consideration, the equations (1) and (2) are taken into consideration. ), The condition of the following equation (3) can be added. That is, when any of the conditions of the equations (1) to (3) is satisfied, it is determined that the posture difference between _{the avatar posture pos a} and the user's actual posture pos _{u is equal to or greater than the first range.
Δx au> thres x ··· (3} )

[Delta] x _au in the formula (3), the head position of the actual attitude pos _u user _{(x u, y u, z} u) and the head position of the avatar position _{pos a (x a, y a} , z a) and It represents the difference, as the [Delta] x _au, can employ one-dimensional norm of the two-dimensional norm and of formula (4) (5). thres _x represents the threshold of Δx _au.

As the determination condition in the other step S5, the moving speed of the head position of the avatar or the angular velocity in the line-of-sight direction may be used. Specifically, avatar gaze direction (θ _a, φ _a) Δθ angular velocities, and [Delta] [phi, the head position of the avatar _{(x a, y a, z} a) time variation of the (movement speed) _{v a} As a result, when any of the conditions of the equations (6) to (8) is satisfied, it may be determined that the posture difference between _{the avatar posture pos a} and the user's actual posture pos _{u is equal to or greater than the first range.} ..
_{| Δθ a -Δθ a |> thres} Δθ ··· (6)
｜ Δφ _a −Δφ _a ｜ ＞ thres _Δφ・ ・ ・ (7)
v _a > thres _v ... (8)

Further, in step S5, when the _{phenomenon in which the posture difference between the avatar posture pos a} and the user's actual posture pos _u is determined to be equal to or greater than the first range occurs not only once but also a predetermined number of times or more occurs. Alternatively, it may be determined that the posture difference is equal to or greater than the first range when the occurrence occurs continuously for a predetermined time.

If it is determined in step S5 that _{the posture difference between the avatar posture pos a} and the user's actual posture pos _u is not equal to or greater than the first range, the process returns to step S3, and the processes of steps S3 to S5 described above are repeated. Is done. As a result, _{when the posture difference between the avatar posture pos a} and the user's actual posture pos _u is smaller than the first range, the avatar viewpoint image from the avatar's viewpoint is displayed as a presentation image, and the presentation image posture pos _v = The avatar posture becomes _{pos a.}

On the other hand, _{if it is determined in step S5 that the posture difference between the avatar posture pos a} and the user's actual posture pos _u is equal to or greater than the first range, the process proceeds to step S6, and the presentation image generation unit 74 presents. As an image, a mitigation processing image for reducing discomfort such as "VR sickness" is generated, and is supplied to the display unit 42 for display.

Presenting the image generator 74, as reduction process image, for example, generated and displayed an image corresponding to the user position pos _u. In this case, the presented image posture pos _v becomes the user posture pos _u .
pos _v = (x _v , y _v , z _v , θ _v , φ _v ) = (x _u , _yu , z _u , θ _u , φ _u )

Alternatively, presenting the image generator 74, as reduction process image, for example, a presentation image orientation pos _v the head position of the avatar attitude pos _a for the head position of the presentation image orientation pos _v user orientation for viewing direction of An image with the line-of-sight direction of pos _{u is generated and displayed.} That is, the presented image posture pos _v is as follows.
_{pos v = (x v, y} v, z v, θ v, φ v) = (x a, y a, z a, θ u, φ u)

_{Alternatively, the presentation image generation unit 74 sets the head position and azimuth θ of the presentation image posture pos v as} the head position and azimuth θ of the avatar posture pos _a as the mitigation processing image, and sets the presentation image posture pos v. _For the elevation angle φ of v, an image with the elevation angle φ of the user posture pos _u is generated and displayed. That is, the presented image posture pos _v is as follows.
_{pos v = (x v, y} v, z v, θ v, φ v) = (x a, y a, z a, θ a, φ u)
This means that the user is more likely to get drunk if the field of view is unintentionally changed in the vertical direction, and therefore the user's line-of-sight direction is adopted only in this direction.

On the contrary, as the mitigation processed image, for example, _{the head position and elevation angle φ of the presented image posture pos v} are the head position and elevation angle φ of the avatar posture pos _a , and the azimuth angle θ of the presented image posture pos _{v is the user posture.} An image with the azimuth θ of pos _{u may be generated and displayed.} That is, the presented image posture pos _v is as follows.
_{pos v = (x v, y} v, z v, θ v, φ v) = (x a, y a, z a, θ u, φ a)
This means that the user's line-of-sight direction is adopted only for the azimuth angle θ.

Next, in step S7, the presentation image generation unit 74 _{determines whether the posture difference between the avatar posture pos a} and the user's actual posture pos _u is within a predetermined second range.

In step S7, _{the posture difference between the avatar posture pos a} and the user's actual posture pos _u is not yet within the second range, that is, the posture difference between _{the avatar posture pos a} and the user's actual posture pos _{u is the first.} If it is determined that the range is larger than the range of 2, the process returns to step S6, and the above-mentioned process is repeated. That is, the mitigation processed image is continuously generated as the presented image and displayed on the display unit 42.

On the other hand, if it is determined in step S7 that _{the posture difference between the avatar posture pos a} and the user's actual posture pos _u is within the second range, the process proceeds to step S8, and the presentation image generation unit 74 is reduced. the processed image generated so as to gradually approach the avatar viewpoint image (the image corresponding to the avatar position pos _a), on the display unit 42. After the processing of step S8, the presented image posture pos _v = avatar posture pos _a = user posture pos _u .

This is a process of step S8, the control of the presentation image orientation pos _v of the presentation image (reduction processing image) avatar viewpoint image close continuously, for example, a simple linear movement, curved motion such as Bezier, body model It is possible to use the calculation based on the motion within the drive range allowed by the geometric constraint based on.

After step S8, the process is returned to step S3, and steps S3 to S8 described above are executed again. The values in the first range and the second range may be the same value or may be different values.

The above processing is executed as the presentation image generation processing.

As described above, in the presentation image generation process by the image system 1, when the presentation image generation unit 74 has _{a posture difference between the avatar posture pos a} and the user's actual posture pos _u smaller than a predetermined threshold value (first range). , An avatar viewpoint image ( _{an image corresponding to the avatar posture pos a} ) from the viewpoint of the avatar is generated as the first presentation image, _{and the posture difference between the avatar posture pos a} and the user's actual posture pos _u is a predetermined threshold value (first). If there is a posture difference of a certain amount or more between the avatar posture pos _a and the user's actual posture pos _u , a mitigation processed image different from the avatar viewpoint image is generated and displayed as the second presentation image. It is displayed on the unit 42. This makes it possible to reduce user discomfort such as "VR sickness".

In the generation of the mitigation processed image in step S6 described above, the mitigation processed image is generated and displayed using the content image on the virtual world. For example, a black image (blackout screen) in which the entire screen is entirely black or , An image different from the displayed content image may be displayed, such as a warning screen explaining that the screen is being changed to prevent "sickness".

In this case, in the process of continuously approaching the avatar viewpoint image, which is executed in step S8, it is possible to return to the avatar viewpoint image by using, for example, an alpha blending process of alpha blending the black image and the avatar viewpoint image.

Assuming that the avatar viewpoint image is I _a and the black image is I _b , the mitigation processed image I _u in step S8 is represented by the following equation (9) using the blend ratio α (0 ≦ α ≦ 1).
I _u = αI _a + (1-α) I _b ... (9)

By continuously changing the blend ratio α of the formula (9) from 0 to 1 with the passage of time, it is possible to control the black image to gradually return to the original avatar viewpoint image.

<Example of content that shares a virtual world with multiple users>
The above-mentioned example is an example in which one user experiences a virtual world, but there are cases where a plurality of users share and experience the same virtual world.

For example, as shown in FIG. 7, the avatar 131B of the other user 121B is displayed on the virtual world (presentation image) displayed on the HMD 11 of one user 121A, and is displayed on the HMD 11 of the other user 121B. It is assumed that there is a scene in which the avatar 131A of one user 121A is displayed on the virtual world (presentation image).

The avatar 131A of the user 121A operates in the virtual world based on the sensor detection result of the user 121A. The avatar 131B of the user 121B operates in the virtual world based on the sensor detection result of the user 121B.

Therefore, basically, the avatar 131 (131A, 131B) behaves the same as the operation of the user 121 (121A, 121B), but as a content design, the operation that does not match the operation of the user 121 can be performed. be.

For example, consider a situation in which user 121A bows a greeting to avatar 131A of user 121A.

The actual user 121A may bow, but since the actual user 121A cannot be seen by the user 121B, the avatar 131A seen by the user 121B in the virtual world may be bowed. .. Therefore, the user 121A gives an operation instruction to make the avatar 131A bow by operating the controller 13, voice input, or the like.

In such situations, as described above, since the user posture pos _u and avatar position pos _a different situation occurs, presentation image generation processing described in FIG. 6 can be applied as well.

That is, when the user 121A instructs the bowing operation by the operation button or the like, the presentation image in which the avatar 131A of the user 121A performs the bowing operation is displayed on the display unit 42 of the HMD 11 worn by the user 121B. When the avatar 131A of the user 121A performs a bowing motion, _{the posture difference between the avatar posture pos a} and the actual posture pos _u of the user 121A becomes equal to or more than the first range. The unit 74 generates a mitigation processed image and displays it on the display unit 42 of the HMD 11 worn by the user 121A. In other words, when the avatar viewpoint image from the viewpoint of the avatar 131A performing the bowing motion is displayed on the display unit 42 of the HMD 11 worn by the user 121A, the visual field change is large and "sickness" occurs. Can be switched.

Then, when the bowing motion is completed _{and the posture difference between the avatar posture pos a} and the actual posture pos _u of the user 121A is within the second range, the avatar posture of the avatar 131A of the presented image is processed as the process of step S8 in FIG. The pos _a is controlled so as to gradually approach the actual posture pos _{u of the user 121A.}

When the operation command is such that the user 121 can select what kind of operation is to be performed by his / her avatar 131, it is determined in advance whether or not to present the mitigation processing image for each operation command, and an instruction is given. A pre-prepared presentation image may be displayed according to the operation command. That thereby, displays the actual attitude pos _u of the user based on the detection result of detecting the motion of the user, with respect to the operation command when the posture difference between avatar position pos _a results, a different image and an avatar of the viewpoint image Therefore, VR sickness can be suppressed.

<Computer configuration example>
The series of processes described above can be executed by hardware or software. When a series of processes are executed by software, the programs constituting the software are installed in the computer. Here, the computer includes a microcomputer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 8 is a block diagram showing an example of hardware configuration of a computer that executes the above-mentioned series of processes programmatically.

In a computer, a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203 are connected to each other by a bus 204.

An input / output interface 205 is further connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

The input unit 206 includes a keyboard, a mouse, a microphone, and the like. The output unit 207 includes a display, a speaker, and the like. The storage unit 208 includes a hard disk, a non-volatile memory, and the like. The communication unit 209 includes a network interface and the like. The drive 210 drives a removable recording medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 201 loads the program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executes the above-mentioned series. Is processed.

In a computer, the program can be installed in the storage unit 208 via the input / output interface 205 by mounting the removable recording medium 211 in the drive 210. The program can also be received by the communication unit 209 and installed in the storage unit 208 via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In addition, the program can be pre-installed in the ROM 202 or the storage unit 208.

The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in the present specification, in parallel, or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

The steps described in the flowchart are not necessarily processed in chronological order as well as in chronological order in the order described, but are required timings such as in parallel or when calls are made. May be executed in.

In the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

In the above-described embodiment, an example of a head-mounted display (HMD) has been described as a display device for displaying a presented image presented to a user, but a head-up display (HUD) may be used as the display device of the image system 1. However, a dome-shaped (hemispherical) display or the like may be used. Any display device that displays an image so as to cover the user's field of view may be used.

For example, a form in which some of the above-described embodiments are appropriately combined can be adopted.

This technology can be configured as cloud computing in which one function is shared by a plurality of devices via a network and jointly processed.

Further, each step described in the above-mentioned flowchart may be executed by one device or may be shared and executed by a plurality of devices.

Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

It should be noted that the effects described in the present specification are merely exemplary and not limited, and may have effects other than those described in the present specification.

The present technology can also have the following configurations.
(1)
An avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presentation image to be presented to the user, and the actual posture of the user based on the detection result of detecting the movement of the user and the above-mentioned An image processing device including an image generation unit that generates a second presented image different from the avatar viewpoint image when a posture difference from the avatar's posture occurs.
(2)
The posture is defined by the head position and the line-of-sight direction.
The image generation unit generates a second presentation image different from the avatar viewpoint image when the difference between the azimuth angle or the elevation angle between the actual posture of the user and the posture of the avatar is equal to or more than the first threshold value (1). ). The image processing apparatus.
(3)
The posture is defined by the head position and the line-of-sight direction.
In the image generation unit, the difference between the azimuth angle or elevation angle between the user's actual posture and the avatar's posture is equal to or greater than the first threshold value, or the head position between the user's actual posture and the avatar's posture The image processing apparatus according to (1) above, which generates a second presented image different from the avatar viewpoint image when the difference is equal to or greater than the first threshold value.
(4)
The image processing device according to any one of (1) to (3) above, wherein the second presented image is an image corresponding to the posture of the user.
(5)
The image processing device according to any one of (1) to (3) above, wherein the second presented image is an image corresponding to the head position of the posture of the avatar and the line-of-sight direction of the posture of the user.
(6)
The image processing device according to any one of (1) to (3) above, wherein the second presented image is an image corresponding to the head position and azimuth of the avatar's posture and the elevation angle of the user's posture.
(7)
The image processing device according to any one of (1) to (3) above, wherein the second presented image is an image corresponding to the head position and elevation angle of the avatar's posture and the azimuth angle of the user's posture.
(8)
The image generation unit is different from the avatar viewpoint image when the posture difference between the actual posture of the user and the posture of the avatar is equal to or greater than the first threshold value for a predetermined number of times or for a predetermined time. The image processing apparatus according to any one of (1) to (7) above, which generates the presented image of.
(9)
Further, when the posture difference between the actual posture of the user and the posture of the avatar is within the second threshold value, the image generation unit further displays the second presented image so as to gradually approach the avatar viewpoint image. The image processing apparatus according to any one of (1) to (8) above.
(10)
Further provided with an operation unit for instructing the operation of the avatar corresponding to the user,
The image generation unit presents a second image different from the avatar viewpoint image when the posture difference between the actual posture of the user and the posture of the avatar occurs due to the operation of the avatar in response to the instruction from the operation unit. The image processing apparatus according to any one of (1) to (9) above, which generates an image.
(11)
The image processing device
An avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presentation image to be presented to the user, and the actual posture of the user based on the detection result of detecting the movement of the user and the above-mentioned An image processing method including a step of generating a second presented image different from the avatar viewpoint image when a posture difference from the avatar's posture occurs.
(12)
An avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presentation image to be presented to the user, and the actual posture of the user based on the detection result of detecting the movement of the user and the above-mentioned When there is a posture difference from the avatar's posture, an image generation unit that generates a second presentation image different from the avatar viewpoint image, and an image generation unit.
An image system including the first presented image and a display unit for displaying the second presented image.

1 image system, 11 HMD, 12 image processing device, 13 controller, 41 sensor, 42 display unit, 62 image generation unit, 71 avatar motion control unit, 72 user attitude detection unit, 73 attitude difference calculation unit, 74 presentation image generation unit. , 81 Operation unit, 201 CPU, 202 ROM, 203 RAM, 206 input unit, 207 output unit, 208 storage unit, 209 communication unit, 210 drive

Claims (11)

An avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presentation image to be presented to the user, and the actual posture of the user based on the detection result of detecting the movement of the user and the above-mentioned An image generation unit that generates a second presentation image different from the avatar viewpoint image when there is a posture difference from the avatar's posture.
Equipped with
The posture is defined by the head position and the line-of-sight direction represented by the azimuth and elevation angles.
The second presented image is an image processing device that corresponds to a posture in which at least one of the azimuth or elevation angle of the avatar's posture is set as the value of the actual posture of the user. The image generation unit generates a second presentation image different from the avatar viewpoint image when the difference between the azimuth angle or the elevation angle of the user's actual posture and the avatar's posture is equal to or more than the first threshold value. The image processing apparatus according to. In the image generation unit, the difference between the azimuth angle or elevation angle between the user's actual posture and the avatar's posture is equal to or greater than the first threshold value, or the head position between the user's actual posture and the avatar's posture The image processing apparatus according to claim 1, wherein when the difference is equal to or greater than the first threshold value, a second presented image different from the avatar viewpoint image is generated. The image processing device according to claim 1, wherein the second presented image is an image corresponding to the head position of the posture of the avatar and the line-of-sight direction of the actual posture of the user. The image processing device according to claim 1, wherein the second presented image is an image corresponding to the head position and azimuth of the avatar's posture and the elevation angle of the user's actual posture. The image processing device according to claim 1, wherein the second presented image is an image corresponding to the head position and elevation angle of the avatar's posture and the azimuth angle of the user's actual posture. The image generation unit is different from the avatar viewpoint image when the posture difference between the actual posture of the user and the posture of the avatar is equal to or greater than the first threshold value for a predetermined number of times or for a predetermined time. The image processing apparatus according to claim 1, which generates the presented image of the above. Further, when the posture difference between the actual posture of the user and the posture of the avatar is within the second threshold value, the image generation unit further displays the second presented image so as to gradually approach the avatar viewpoint image. The image processing apparatus according to claim 1 to be generated. Further provided with an operation unit for instructing the operation of the avatar corresponding to the user,
The image generation unit presents a second image different from the avatar viewpoint image when the posture difference between the actual posture of the user and the posture of the avatar occurs due to the operation of the avatar in response to the instruction from the operation unit. The image processing apparatus according to claim 1, which generates an image. The image processing device
An avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presented image to be presented to the user, and the actual posture of the user based on the detection result of detecting the movement of the user and the above-mentioned When there is a posture difference from the avatar's posture, a second presentation image different from the avatar viewpoint image is generated .
The posture is defined by the head position and the line-of-sight direction represented by the azimuth and elevation angles.
The second presented image is an image processing method that corresponds to a posture in which at least one of the azimuth or elevation angle of the avatar's posture is set as the value of the actual posture of the user. An avatar viewpoint image from the viewpoint of the avatar corresponding to the user in the virtual world is generated as the first presentation image to be presented to the user, and the actual posture of the user based on the detection result of detecting the movement of the user and the above-mentioned When there is a posture difference from the avatar's posture, an image generation unit that generates a second presentation image different from the avatar viewpoint image, and
With a display unit that displays the first presented image and the second presented image
Equipped with
The posture is defined by the head position and the line-of-sight direction represented by the azimuth and elevation angles.
The second presented image is an image system that corresponds to a posture in which at least one of the azimuth or elevation angle of the avatar's posture is the value of the actual posture of the user.

Images