JP2021177580A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To provide an information processing device, an information processing method and a program, each of which enables a two-dimensional image to be seen as a three-dimensional image in a pseudo manner in a three-dimensional space.SOLUTION: An information processing apparatus is provided, comprising a control unit performing display control for arranging a two-dimensional image obtained by photographing a real object as a stereoscopic image in a three-dimensional space, and display control for maintaining a state where a straight line connecting a prescribed axis on the stereoscopic image and a user viewpoint is substantially perpendicular to the stereoscopic image, with the prescribed axis on the stereoscopic image as a reference.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to information processing devices, information processing methods, and programs.

In a VR (Virtual Reality) application that has become widespread in recent years, a user can view a virtual space in which a 3D model is arranged from an arbitrary viewpoint. Such a VR world can be provided mainly by using a non-transparent HMD (Head Mounted Display) that covers the user's field of view with a display unit.

The reflection of the user's operation in the virtual space is performed, for example, by acquiring information related to the real world from a photographed image of the real world. Specifically, a method is used as an example in which a user's body or marker is photographed with a camera, an area of the image is detected, operation input information such as a user's movement is acquired, and the information is reflected in a virtual space.

It is desirable that such information related to the real world be acquired more accurately. For example, in Patent Document 1 below, an image of the real world is presented to the user as auxiliary data in addition to the main data, and information processing is performed. A technique for showing the user what kind of recognition the real space is performed on the device side and giving awareness of the defect is disclosed.

Japanese Unexamined Patent Publication No. 2016-140017

Here, it is assumed that a 3D model of a more realistic real-world person or the like is used as the viewing target. However, in order to make a person or the like into a 3D model from a live-action image, it was necessary to shoot in a special shooting studio equipped with a large number of cameras. Further, the image quality of the 3D model generated in this way is significantly deteriorated as compared with the image quality of a person or the like who normally watches on a television or the like.

Therefore, the present disclosure proposes an information processing device, an information processing method, and a program capable of making a two-dimensional image appear as a pseudo three-dimensional image in a three-dimensional space.

According to the present disclosure, display control for arranging a two-dimensional image obtained by photographing a real object as a stereoscopic image in a three-dimensional space and a predetermined axis on the stereoscopic image are used as a reference. We propose an information processing apparatus including a control unit that maintains a state in which a straight line connecting an axis and a user's viewpoint is substantially perpendicular to the stereoscopic image.

According to the present disclosure, the processor arranges a two-dimensional image obtained by photographing a real object as a stereoscopic image in a three-dimensional space, and a predetermined axis on the stereoscopic image is used as a reference. We propose an information processing method including performing display control for maintaining a state in which a straight line connecting the predetermined axis and the user's viewpoint is substantially perpendicular to the stereoscopic image.

According to the present disclosure, with reference to display control in which a computer arranges a two-dimensional image obtained by photographing a real object in a three-dimensional space as a stereoscopic image, and a predetermined axis on the stereoscopic image as a reference. We propose a program for functioning as a control unit that performs display control for maintaining a state in which a straight line connecting the predetermined axis and the user's viewpoint is substantially perpendicular to the stereoscopic image.

As described above, according to the present disclosure, it is possible to make a two-dimensional image appear as a pseudo three-dimensional image in a three-dimensional space.

It should be noted that the above effects are not necessarily limited, and together with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.

It is a figure explaining the outline of the information processing system by one Embodiment of this disclosure. It is a figure which shows the movement of the user viewpoint in the virtual space by this embodiment. It is a figure explaining the distortion which occurs when the stereoscopic image by a stereo image is viewed from an angle. It is a figure which shows an example of the whole structure of the information processing system by this embodiment. It is a block diagram which shows an example of the structure of the information processing terminal by this embodiment. It is a figure explaining the rotation control of a stereoscopic image by 1st Example of this Embodiment. It is a figure which shows an example of the appearance when the stereoscopic image by the 1st Example of this Embodiment is rotation-controlled. It is a figure explaining the case where the stereoscopic image becomes an unnatural posture when it is controlled so that it always faces the user's viewpoint in virtual space. It is a figure explaining the definition of the rotation axis of the stereoscopic image by 1st Example of this Embodiment. It is a figure which shows an example of the warning which is displayed when viewing from the direction restricted by the 1st Embodiment of this embodiment. It is a figure explaining the stereoscopic image and 3D model by 2nd Example of this Embodiment. It is a figure which shows an example of the decorative display of the 3D model with respect to the stereoscopic image by the 2nd Example of this Embodiment. It is a figure explaining the distortion of the stereoscopic image and the deviation of the decoration position of a 3D model when the user viewpoint in virtual space moves. It is a figure explaining the deviation of the decoration position of a 3D model in the case where only the stereoscopic image is rotated and controlled according to the movement of the user's viewpoint in the virtual space. It is a figure explaining the rotation control of the stereoscopic image and the 3D model according to the movement of the user's viewpoint in virtual space by the 2nd Embodiment of this Embodiment. It is a figure explaining the movement interlocked with the stereoscopic image of the decorated 3D model by the 2nd Example of this Embodiment. It is a flowchart which shows an example of the flow of operation processing by this Embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

In addition, the explanation shall be given in the following order.
1. 1. Outline of the information processing system according to the embodiment of the present disclosure 2. Configuration of information processing terminal 3. Stereoscopic display control example 3-1. First Example (3-1-1. Rotation Control)
(3-1-2. Rotation constraint)
(3-1-3. Warning)
3-2. Second Example 4. Operation processing 5. Others 6. summary

<< 1. Outline of information processing system according to one embodiment of the present disclosure >>
FIG. 1 is a diagram illustrating an outline of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the information processing system according to the present embodiment is an information processing terminal realized by a display device such as a non-transmissive HMD that is attached to the user's head and covers the user's field of view with a display unit 140, for example. 10. Presents an image of the virtual space to the user. By configuring the HMD to block the outside world when it is attached to the head, it is possible to increase the virtual reality during viewing.

Further, in the virtual space, for example, virtual objects 50a and 50b which are generated three-dimensional images (so-called 3D models) are localized as a background (localization virtual object). The virtual objects 50a and 50b have shape data including depth information in the virtual space, and can be displayed without breaking the shape even when viewed from an arbitrary viewpoint. That is, the virtual objects 50a and 50b localized in the virtual space change their motion parallax in response to changes in the user's viewpoint and posture.

Further, in the information processing system according to the present disclosure, the stereoscopic image 40 is arranged in the virtual space based on the information obtained by imaging a person or the like (real object) in the real space.

(background)
Here, when a more realistic 3D model of a person in the real world is used as the viewing target, in order to make the person etc. into a 3D model from the live-action video, in a special shooting studio where a large number of cameras are installed. I need to shoot. In addition, the image quality of the 3D model generated in this way may be significantly deteriorated as compared with the image quality of a person or the like who is normally viewing on a television or the like.

As a stereoscopic vision technique, a stereo image technique is known. A stereo image is composed of two 2D images (left viewpoint image and right viewpoint image) taken by a stereo camera. By showing the left viewpoint image only to the left eye and the right viewpoint image only to the right eye, the stereo image (stereo image) 2D images) can be viewed stereoscopically. Therefore, by arranging the stereo image obtained by shooting a real object (for example, a person) as a stereoscopic image in the virtual space, the image quality can be lowered and the cost of shooting can be reduced without making a 3D model from the live-action image. It is possible to stereoscopically view a person in the real world in a virtual space without hanging it.

However, when the virtual space is viewed only from a specific viewpoint (front), the stereoscopic image presented by the stereo image does not collapse, but as shown in FIG. 2, the user can view the viewpoint in the virtual space. When the stereoscopic image 40 is viewed from an angle, as shown in FIG. 3, the stereoscopic image 40 does not look strange when viewed from the front (see the left in FIG. 3), and when viewed from an angle, The stereoscopic image 40, which is a dimensional image, is distorted (see the right side of FIG. 3).

Therefore, in the information processing system according to the present disclosure, it is possible to make a two-dimensional image appear as a pseudo three-dimensional image in a three-dimensional space by maintaining a state in which the stereoscopic image faces the user's viewpoint. .. Specifically, in the information processing system according to the present disclosure, a straight line connecting the predetermined axis and the user's viewpoint is substantially perpendicular to the surface of the stereoscopic image with reference to a predetermined axis passing through the stereoscopic image. By rotating the orientation of the stereoscopic image so as to be, it is possible to face the user's viewpoint.

In the virtual space, there is a spatially localized 3D model (localized virtual object) such as a background, and the motion parallax is changed in response to a change in the user's viewpoint or posture (the display coordinates of the localized virtual object change). The user who simultaneously visually recognizes such a 3D model and the stereoscopic image does not, even if the stereoscopic image is rotationally controlled so as to face the user's viewpoint when the viewpoint is changed. Is difficult to recognize as a two-dimensional image, and it is easy to mistake it as a three-dimensional image.

As described above, in the information processing system according to the present disclosure, a two-dimensional image (specifically,) generated by a simple device / equipment / method without performing 3D modeling from a live image and without incurring a cost at the time of shooting. Is a stereo image), it is possible to present a real-world person or the like in a three-dimensional space without degrading the image quality.

In addition, by maintaining the state in which the stereoscopic image faces the user's line of sight, the stereoscopic image, which is a two-dimensional image, is simulated without causing distortion of the stereoscopic image even when the user's viewpoint moves. It is possible to make it look like a three-dimensional image. That is, in the video experience in the virtual space, it is possible to give an appropriate video change (pseudo three-dimensionalization) according to the change in the user's viewpoint, and it is possible to improve the video experience value of the user.

The stereoscopic image may be generated based on a two-dimensional image obtained by photographing a real object in advance, or may be generated based on a two-dimensional image obtained by photographing a real object in real time. ..

Subsequently, the overall configuration of the information processing system according to the present embodiment will be described with reference to FIG. FIG. 4 is a diagram showing an example of the overall configuration of the information processing system according to the present embodiment.

As shown in FIG. 4, the information processing system according to the present embodiment includes an information processing terminal 10 that generates an image of a virtual space and controls display on the display unit 140, and a server 2 that stores content information of the virtual space. It may be a configuration. The information processing terminal 10 and the server 2 can communicate with each other via the network 3 to transmit and receive data.

The information processing terminal 10 acquires the content of the virtual space from the server 2, generates a free viewpoint image from the content according to the movement of the user, the controller operation, and the like, and provides the content to the display unit 140. The movement of the user is a change in the position or posture of the user (including the direction of the head), and the information processing terminal 10 uses the detected position or posture of the user to determine the user's viewpoint position (three-dimensional position) in the real world. ) Is recognized, and the user viewpoint position in the virtual space (hereinafter, also referred to as the user viewpoint in the virtual space) is calculated based on the recognized user viewpoint position.

The content of the virtual space provided to the user is spherical content, free-viewpoint content, game content, or the like. The free-viewpoint content generates an image when the virtual camera is placed at an arbitrary position, and enables the image to be viewed from any viewpoint. Also, as used herein, the term "virtual space" (or "VR space") can be perceived by a real user through one or more display devices and / or one or more users. Refers to a representation of a real or fictitious environment with rules of interaction simulated by one or more processors that can interact through an interface. The term "user interface" refers to the actual device on which the user can send input or receive output to and from the virtual world.

In the virtual space, the user may be represented by an avatar, or the world of the virtual space may be displayed from the viewpoint of the avatar without displaying the avatar on the display. In the present specification, the user's (or avatar's) viewpoint (user's viewpoint in the virtual space) in the virtual space can be regarded as the field of view of the virtual camera. The "virtual camera" refers to a viewpoint in a virtual space, and is used in a calculation for drawing a three-dimensional virtual space as a two-dimensional image on a display (display device). The information processing system according to the present embodiment may further include a remote controller or a game controller for communicating the user's intention to the system.

Further, as described above, the information processing terminal 10 is realized by, for example, an HMD worn on the user's head. The HMD may have an image display unit for each of the left and right eyes, and may be configured to be able to control vision and hearing by using headphones in combination. The HMD can also project different images to the left and right eyes, and can present a 3D image by displaying an image having parallax with respect to the left and right eyes. Further, the information processing terminal 10 may be a device that presents a virtual space image on a non-wearable display such as a television device, a smartphone, or a tablet terminal.

The information processing system according to the embodiment of the present disclosure has been described above. Subsequently, a specific configuration of the information processing terminal 10 included in the information processing system according to the present embodiment will be described with reference to the drawings.

<< 2. Configuration of information processing terminal 10 >>
FIG. 5 is a block diagram showing an example of the configuration of the information processing terminal 10 according to the present embodiment. As shown in FIG. 5, the information processing terminal 10 includes a control unit 100, a communication unit 110, an operation input unit 120, a sensor unit 130, a display unit 140, a speaker 150, and a storage unit 160.

The control unit 100 functions as an arithmetic processing unit and a control device, and controls the overall operation in the information processing terminal 10 according to various programs. The control unit 100 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. Further, the control unit 100 may include a ROM (Read Only Memory) for storing programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) for temporarily storing parameters and the like that change as appropriate.

The control unit 100 according to the present embodiment also functions as a user viewpoint recognition unit 101, a virtual space viewpoint calculation unit 102, an image generation unit 103, and a display control unit 104.

The user viewpoint recognition unit 101 recognizes the user viewpoint position (three-dimensional position) in the real world from the position and posture (including the orientation of the head) of the user sensed by the sensor unit 130 or the like. Next, the virtual space viewpoint calculation unit 102 calculates the user viewpoint in the virtual space based on the user viewpoint in the real world recognized by the user viewpoint recognition unit 101. Next, the image generation unit 103 generates a free viewpoint image of the virtual space corresponding to the user viewpoint in the virtual space calculated by the virtual space viewpoint calculation unit 102, and the display control unit 104 generates the free viewpoint image of the generated virtual space. The free viewpoint image is displayed on the display unit 140. By continuously performing these processes, when the user moves back and forth and left and right or changes the viewpoint in the real world, the same amount of movement (or the corresponding movement amount at a predetermined ratio) moves back and forth and left and right in the virtual space. You can move and change the viewpoint, and you can enjoy a highly immersive VR experience.

The user's position and posture are detected by, for example, the sensor unit 130 provided in the information processing terminal 10. The user viewpoint recognition unit 101 tracks the movement of the user's head (head tracking), follows the movement of the line of sight (eye tracking), and positions and postures based on the sensing data sensed by the sensor unit 130. By following (position tracking), the user's viewpoint position in the real space can be continuously recognized.

More specifically, for example, in head tracking, the posture information of the head is calculated based on the detection result of 9 axes detected by the sensor unit 130 of the information processing terminal 10 mounted on the user's head. Can be done. Further, the eye tracking is performed on an image of the user's eye detected by the sensor unit 130 of the information processing terminal 10 mounted on the user's head (for example, an infrared image obtained by emitting infrared rays to the eyes and capturing the reflection thereof). Based on this, it can be performed by calculating the line of sight (line-of-sight direction) of the user. For position tracking, there are an outside-in method that uses an external sensor, an inside-out method that uses a sensor mounted on the object to be measured (HMD), and a hybrid method that is a combination of these. There is.

In the outside-in method, for example, the positions of a plurality of LED lights (markers) provided in the HMD are imaged and acquired by a camera installed outside, and the position and posture are determined based on the change in the acquired position information of the LED lights. calculate. The user viewpoint recognition unit 101 can also calculate the user posture including the head posture of the user based on the change in the position information of the LED light and the motion sensor data provided in the HMD. On the other hand, in the inside-out method, the position and posture of the user are calculated based on the information acquired by the motion sensor, the depth sensor, and the outward camera provided in the HMD. For example, the outside world is imaged by an outward camera provided in the HMD, and the user's position information is acquired by comparing with a spatial map generated in advance by SLAM (Simultaneous Localization and Mapping) technology, or information on a motion sensor or a depth sensor. At the same time, it is possible to calculate the posture of the user.

Further, the image generation unit 103 according to the present embodiment can generate an image in which a two-dimensional image (for example, a stereoscopic image of a person) obtained by photographing a real object is arranged in a three-dimensional space as a stereoscopic image. As a result, it is possible to arrange a live-action person or the like in a virtual space without incurring a shooting cost and without degrading the image quality. Since the live-action stereo image does not blend with the background of the three-dimensional space if the background is left, the stereo image with the background removed may be used. Background removal is possible, for example, by shooting with a clean back.

Further, the display control unit 104 makes the two-dimensional image face the user's viewpoint in the virtual space so that the two-dimensional image to be stereoscopically viewed does not feel uncomfortable when the user's viewpoint in the virtual space changes. It is possible to control the rotation. The display control unit 104 considers the image for the right eye and the image for the left eye to be stereoscopically viewed as one object, and rotates the object so as to face the viewpoint. As a result, it is possible to more reliably make the two-dimensional image appear as a pseudo three-dimensional image in the three-dimensional space, and it is possible to improve the video experience value of the user.

The details of the stereoscopic display control of the two-dimensional image obtained by photographing the real object according to the present embodiment will be described later with reference to the drawings.

(Communication unit 110)
The communication unit 110 connects to the network 3 by wire or wirelessly, and transmits / receives data to / from the server 2 on the network. The communication unit 110 is, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution)), or 3G (third generation mobile body). Communication connection with network 3 by communication method)) or the like.

(Operation input unit 120)
The operation input unit 120 receives an operation instruction by the user and outputs the operation content to the control unit 100. The operation input unit 120 may be, for example, a touch sensor, a pressure sensor, or a proximity sensor. Alternatively, the operation input unit 120 may have a physical configuration such as a button, a switch, and a lever.

(Sensor unit 130)
The sensor unit 130 has a function of sensing the real space such as the situation of the user or the surroundings. Specifically, the sensor unit 130 includes a position information acquisition unit 131, a camera 132 (inward / outward camera), a microphone (microphone) 133, an acceleration sensor 134, and a biological sensor 135 (pulse, heartbeat, sweating, blood pressure, etc.). Includes body temperature, respiration, myoelectric value, brain wave), etc. The specific example of the sensor unit 130 is not limited to this, and for example, the sensor unit 130 has a sensor capable of detecting a total of 9 axes of a 3-axis gyro sensor, a 3-axis acceleration sensor, and a 3-axis geomagnetic sensor. You may. Further, the sensor unit 130 is an inward camera such as an infrared sensor (infrared light emitting unit and an infrared camera) provided around the display unit 140 located in front of the user's eyes as a line-of-sight detection sensor for detecting the user's line of sight. , A myoelectric sensor that detects the movement of muscles around the user's eye, a brain wave sensor, or the like may be provided.

(Display unit 140)
When the information processing terminal 10 is configured as an HMD, for example, the display unit 140 includes left and right screens fixed to the left and right eyes of the user, and displays an image for the left eye and an image for the right eye. The screen of the display unit 140 is composed of, for example, a display panel such as a liquid crystal display (LCD) or an organic EL ((Electro Luminescence) display), or a laser scanning display such as a direct retinal drawing display. The display unit 140 may include an imaging optical system that magnifies and projects the display screen and forms an enlarged imaginary image having a predetermined angle of view on the pupil of the user.

(Speaker 150)
For example, when the information processing terminal 10 is configured as an HMD, the speaker 150 is configured as headphones worn on the user's head and reproduces an audio signal. The speaker 150 is not limited to the headphone type, and may be configured as an earphone or a bone conduction speaker.

(Storage unit 160)
The storage unit 160 is realized by a ROM (Read Only Memory) that stores programs and arithmetic parameters used for processing of the control unit 100, and a RAM (Random Access Memory) that temporarily stores parameters and the like that change as appropriate.

Although the configuration of the information processing terminal 10 has been specifically described above, the configuration of the information processing terminal 10 according to the present disclosure is not limited to the example shown in FIG. For example, the information processing terminal 10 may be realized by a plurality of devices. Specifically, the configuration includes a display device realized by an HMD or the like (corresponding to at least the display unit 140) and an information processing device realized by a smartphone, a tablet terminal, a PC or the like (corresponding to at least the control unit 100). There may be. Further, the tracking process by the information processing terminal 10 described above may be performed by an external device.

Further, each function of the control unit 100 of the information processing terminal 10 may be realized by a server provided on the network, or a dedicated terminal, a smartphone, a tablet terminal, or a PC arranged in the same space as the user. It may be realized by such as.

<< 3. Stereoscopic display control example >>
<3-1. First Example>
The information processing terminal 10 according to the present embodiment maintains a state in which the stereoscopic image based on the two-dimensional image acquired by photographing the real object faces the user's viewpoint in the virtual space, so that the user in the virtual space Even if the viewpoint changes, the stereoscopic image can be shown without being broken, and the two-dimensional image can be shown as a pseudo three-dimensional image in the three-dimensional space. Hereinafter, the stereoscopic display control of the two-dimensional image according to the present embodiment will be described in detail with reference to FIGS. 6 to 10.

(3-1-1. Rotation control)
FIG. 6 is a diagram illustrating rotation control of the stereoscopic image 40 according to the first embodiment of the present embodiment. As shown in FIG. 6, the information processing terminal 10 stereoscopically uses a predetermined axis 42 passing through the stereoscopic image 40 as a reference so as to maintain a state in which the stereoscopic image 40 faces the user's viewpoint in the virtual space. The visual image 40 is rotated. More specifically, in the information processing terminal 10, a straight line 45 connecting a predetermined axis 42 passing through the stereoscopic image 40 and the user's viewpoint (user's viewpoint in the virtual space) is formed on the stereoscopic image 40 (plane). The stereoscopic image 40 is rotated with respect to a predetermined axis 42 so as to be substantially vertical to the stereoscopic image 40. In the example shown in FIG. 6, the information processing terminal 10 rotates the stereoscopic image 40 with respect to a predetermined axis 42 passing through the stereoscopic image 40, that is, rotates the stereoscopic image 40 in the yaw direction (left and right).

FIG. 7 shows an example of how the stereoscopic image 40 looks when the rotation is controlled. For example, when the viewpoint is moved to the right with respect to the stereoscopic image 40 from the viewpoint when the stereoscopic image 40 shown on the left of FIG. 7 is viewed from the front, the stereoscopic image 40 is rotated so as to face the user's viewpoint. Therefore, as shown on the right side of FIG. 7, the image can be viewed without discomfort without the distortion shown on the right side of FIG.

(3-1-2. Rotation constraint)
Here, if the rotation control of the stereoscopic image 40 is performed so as to always face the user's viewpoint in the virtual space, the posture may be unnatural depending on the stereoscopic image 40 (subject).

FIG. 8 is a diagram illustrating a case where the stereoscopic image 40 has an unnatural posture when controlled so as to always face the user's viewpoint in the virtual space. For example, when the user's viewpoint in the virtual space is moved upward with respect to the stereoscopic image 40 from the viewpoint of viewing the stereoscopic image 40 from the front as shown on the left of FIG. 8 (that is, above the stereoscopic image 40). (When the angle looks down from) will be explained. In this case, when the stereoscopic image 40 is rotated so as to face the user's viewpoint (specifically, it is rotated (up and down) in the Pitch direction with reference to the horizontal axis passing through the center of the stereoscopic image 40. ), As shown on the right side of FIG. 8, the stereoscopic image 40 has an unnatural posture as if it were lying on the ground.

Therefore, in the information processing system according to the present embodiment, it is possible to avoid an unnatural posture by setting a rotation constraint for each stereoscopic image and performing rotation control within the rotation constraint. And. Rotational constraints include constraints on the rotatable axis (rotational direction) and the rotatable angle (rotation amount).

Here, the definition of the rotation axis used in the present specification will be described with reference to FIG. FIG. 9 is a diagram illustrating a definition of a rotation axis of the stereoscopic image 40 according to the first embodiment of the present embodiment. As shown in FIG. 9, left-right rotation with respect to an axis vertically passing through the center of the stereoscopic image 40 (that is, an axis in the vertical direction in the virtual space) is called Yaw rotation, and the center of the stereoscopic image 40 is horizontal. The vertical rotation with respect to the axis passing through (that is, the horizontal axis in the virtual space) is referred to as Pitch rotation. Further, rotation based on an axis (that is, an axis in the front-rear direction in the virtual space) that passes through the center of the stereoscopic image 40 perpendicularly to the plane of the stereoscopic image 40 is referred to as Roll rotation.

The rotation constraint may be set for each stereoscopic image. Further, the rotation constraint may include information on the angle at which the rotation can be performed for each rotation axis. For example, when the real object is a person, it is possible to avoid an unnatural posture by limiting the rotation to the Yaw rotation. An example of the parameters of the rotation constraint information in this case is shown below.
-Rotation constraint parameter Yaw: 0 ° to 360 °
Pitch: 0 °
Roll: 0 °

The rotation constraint is assumed to be possessed by the stereoscopic image as a parameter, but the present embodiment is not limited to this, and may be embedded as a fixed value in the application.

Further, in the above-described example, 0 ° to 360 ° is set to be rotatable in Yaw rotation, but the present embodiment is not limited to this, and for example, the user views a person (stereoscopic image 40) from directly behind. Even in this case, if the person can be seen in the front, it may be uncomfortable to see the back side, so Yaw: 0 ° to 180 ° or the like may be set. Further, in this case, if there is another camera in the vicinity of 180 °, the image may be switched to the image taken by the other camera (for example, the stereo image of the person taken from the side).

Further, the rotation angle set by the rotation constraint may be an angle with reference to the viewing start point, or may be an angle with reference to the front in the global coordinates of the virtual space.

(3-1-3. Warning)
By controlling the rotation within the rotation constraint, it is possible to prevent the stereoscopic image 40 from being in an unnatural posture as described above, but when the stereoscopic image 40 is viewed from the restricted direction, the image of the stereoscopic image 40 is displayed. May be greatly distorted. In this case, the information processing terminal 10 may issue a warning (warning using voice, display, vibration, etc.) such as indicating the recommended viewing direction to the user so that comfortable viewing can be realized.

FIG. 10 is a diagram showing an example of a warning displayed when viewing from a restricted direction according to the first embodiment of the present embodiment. For example, as shown on the left side of FIG. 10, when the user's viewpoint in the virtual space moves upward with respect to the stereoscopic image 40 (that is, when the angle is such that it looks down from above), the stereoscopic image 40 is touched due to rotation restrictions. As shown on the right side of FIG. 10, the stereoscopic image 40 appears to be greatly distorted because it cannot be rotated. In this case, the information processing terminal 10 may display a warning display 142 notifying the recommended viewing direction, such as "Please watch from the front as much as possible". The information processing terminal 10 may give a warning to the stereoscopic image 40 when the stereoscopic image 40 is viewed at an angle of a certain value or more in a direction in which rotation is restricted.

<3-2. Second Example>
Subsequently, as a second embodiment of the present embodiment, a case where a 3D model (a virtual object having three-dimensional shape data) is used to decorate a stereoscopic image that performs rotation control as described above is used. This will be described with reference to FIGS. 11 to 16.

FIG. 11 is a diagram illustrating a stereoscopic image 40 and a 3D model 60 according to a second embodiment of the present embodiment. As shown in FIG. 11, the stereoscopic image 40 is a stereo image composed of two two-dimensional images, and the stereoscopic image is realized by presenting these separately to the left and right eyes. On the other hand, the 3D model 60 is a three-dimensional object having three-dimensional shape data including depth information in the virtual space, and the shape does not collapse even when viewed from an arbitrary viewpoint. In this embodiment, it is possible to realize an expression in which the stereoscopic image 40 which is such two-dimensional data and the 3D model 60 which is three-dimensional data are combined, and it is possible to further improve the viewing experience of the user.

FIG. 12 is a diagram showing an example of a decorative display of the 3D model 60 with respect to the stereoscopic image 40 according to the present embodiment. As shown in FIG. 12, by arranging the 3D model 60 (present image) at the position of the hand of the stereoscopic image 40 (person) on the front side of the stereoscopic image 40, the person holds the present. It is possible to realize a decorative display to be shown. The 3D model 60 is a virtual object that can be interacted with in the virtual space. Further, the information on the position of the hand in the real space of the stereoscopic image 40 may be recognized by image analysis, or the controller held by the subject person in the hand at the time of shooting may be detected and recognized. In addition, information on the position of the hand (three-dimensional position information) recognized at the time of shooting or the like may be recorded in this way.

Here, when the user's viewpoint in the virtual space moves, as shown in FIG. 13, distortion occurs in the stereoscopic image 40 which is two-dimensional data, and since the 3D model is three-dimensional data, the distortion of the image is caused. The shape does not collapse even if the image is viewed from an arbitrary position without occurring, but the decorative position with respect to the stereoscopic image 40 shifts (shifts from the position of the person's hand).

On the other hand, when the rotation of only the stereoscopic image 40 (so as to face the user's viewpoint in the virtual space) is controlled according to the movement of the user's viewpoint in the virtual space, the distortion of the image of the stereoscopic image 40 is as shown in FIG. Although it is resolved, there is still a shift in the decorative position of the 3D model 60.

Therefore, in this embodiment, as shown on the left side of FIG. 15, the surface of the stereoscopic image 40 faces the user viewpoint in the virtual space with reference to the predetermined axis 42 according to the movement of the user viewpoint in the virtual space. In addition to controlling the rotation of the stereoscopic image 40, the 3D model 60 (with the stereoscopic image 40) has the same amount of rotation (rotation direction, rotation angle) with reference to the same rotation axis (that is, a predetermined axis 42). Rotate.

The information processing terminal 10 revolves the 3D model 60 with reference to a predetermined axis 42, which is the rotation axis of the stereoscopic image 40, so that when the stereoscopic image 40 and the 3D model 60 are rotated, the 3D model 60 The position relative to the stereoscopic image 40 can be maintained. Revolving the 3D model 60 means that the 3D model 60 is orbited around a predetermined axis 42 around a predetermined axis 42 (rotated along a circular orbit around the predetermined axis). That is, the coordinate position of the 3D model 60 in the virtual space changes. On the other hand, the stereoscopic image 40 rotates about a predetermined axis 42 passing through the stereoscopic image 40 (that is, the coordinate position in the virtual space does not change and rotates about the predetermined axis 42 on the spot).

As a result, as shown on the right side of FIG. 15, the distortion of the image of the stereoscopic image 40 can be eliminated, and the deviation of the decorative position of the 3D model 60 can also be eliminated.

Also in the second embodiment, when the rotation constraint is set on the stereoscopic image 40, the information processing terminal 10 can perform rotation control within the rotation constraint and give a warning if necessary.

Further, the display position of the 3D model 60 is not fixed, but may be moved in conjunction with the stereoscopic image 40. FIG. 16 is a diagram for explaining the movement of the decorated 3D model 60 according to the present embodiment in conjunction with the stereoscopic image 40. As shown in the upper part of FIG. 16, for example, when the 3D model 60 (for example, a present image) is displayed according to the position of the hand of the person in the stereoscopic image 40, the display position of the 3D model 60 is changed to the movement of the hand position. It is possible to follow. As described above, the position information of the hand of the person in the stereoscopic image 40 can be recognized and recorded at the time of shooting, for example, so that the 3D model 60 can be arranged according to the position of the hand at the time of reproduction. It becomes.

Further, as shown in the lower part of FIG. 16, it is possible to express that the person in the stereoscopic image 40 hands the present (3D model 60) to the user. The information on the position of the hand also includes depth information (depth information), and when the present (3D model 60) approaches the user, it is possible to control the display so that the 3D model 60 looks large.

Even when the display position of the 3D model 60 is moved in conjunction with the stereoscopic image 40, when the rotation control is performed so that the stereoscopic image 40 faces the user's viewpoint in response to a change in the user's viewpoint in the virtual space, 3D As explained with reference to FIG. 15, the model 60 can also be expressed without deviating from the position of the hand by rotating the model 60 with the same rotation axis and the same amount of rotation as the stereoscopic image 40. Become.

<< 4. Operation processing >>
Subsequently, the operation processing of the information processing system according to the present embodiment will be specifically described with reference to FIG. FIG. 17 is a flowchart showing an example of the flow of the display control process according to the present embodiment. Here, as an example, the decoration of the 3D model 60 described in the second embodiment, the rotation constraint described in the first embodiment, and the processing related to the warning as necessary will be described.

As shown in FIG. 17, first, the information processing terminal 10 calculates the viewpoint position in the virtual space (user viewpoint in the virtual space) based on the viewpoint position in the real world (step S103). The user viewpoint in the virtual space may be calculated by sensing the direction and line of sight of the user's head in the real world, or may be calculated based on the controller operated by the user.

Next, the information processing terminal 10 rotates the stereoscopic image 40 (based on the two-dimensional image obtained by photographing the real object) so as to face the viewpoint position in the virtual space (step S106). As a result, even if the user's viewpoint in the virtual space changes, it is possible to prevent the image of the stereoscopic image 40, which is two-dimensional data, from being distorted. Further, at this time, the information processing terminal 10 may be rotated within the rotation constraint set in the stereoscopic image 40 (only in the rotatable direction and angle).

Next, when the 3D model 60 is decoratively displayed on the stereoscopic image 40, the information processing terminal 10 rotates the 3D model 60 by the same amount of rotation (angle) on the same rotation axis as the stereoscopic image 40 (step S109). .. As a result, the display position of the 3D model 60 can be prevented from shifting even if the stereoscopic image 40 is rotated.

Next, when the user's viewpoint in the virtual space with respect to the stereoscopic image 40 becomes an angle of a certain value or more in the rotation constraint direction (step S113 / Yes), the information processing terminal 10 indicates a recommended viewing direction, and the like. (Step S115).

Then, the information processing terminal 10 draws all the objects according to the viewpoint position in the virtual space (step S118).

An example of the operation processing according to the present embodiment has been described above. The operation process shown in FIG. 17 is an example, and the present disclosure is not limited to the example shown in FIG. For example, the present disclosure is not limited to the order of the steps shown in FIG. At least one of the steps may be processed in parallel or in reverse order. For example, the process of step S115 and the process of step S118 may be processed in parallel, or may be processed in the reverse order.

Further, not all the processes shown in FIG. 17 need to be executed. Further, not all the processes shown in FIG. 17 need to be performed by a single device.

<< 5. Others >>
The information processing system according to this embodiment will be supplemented below.

When the stereoscopic image 40 is rotated and controlled, the information processing terminal 10 may move the sound source position of the corresponding audio information in accordance with the rotation control of the stereoscopic image 40. For example, the information processing terminal 10 may perform sound image localization processing so that the voice information of the person in the stereoscopic image 40 can always be heard from the front of the stereoscopic image 40 even when the stereoscopic image 40 is rotated. Further, the information processing terminal 10 may change the audio information corresponding to the stereoscopic image 40 when the stereoscopic image 40 is shielded by the virtual object. For example, when the 3D model 60 (present image) is decoratively displayed on the person in the stereoscopic image 40, when the person in the stereoscopic image 40 hides his mouth with a present, the information processing terminal 10 uses the stereoscopic image. The voices of 40 people may be changed into a clogged voice.

Further, the information processing terminal 10 changes (re-uses, etc.) the lighting effect (shadow position, size, density, etc.) of the stereoscopic image 40 based on the illumination position in the virtual space according to the rotation angle of the stereoscopic image 40. Lighting may be added). The change in the lighting effect of the stereoscopic image 40 may be generated (or photographed) in advance.

In addition, when the viewing position is set to a special position (for example, when the subject wraps around the back side of the stereoscopic image 40), the subject of the stereoscopic image 40 cannot face in that direction. The information processing terminal 10 may not control the rotation of the stereoscopic image 40 and may display a warning such as "not visible from this direction". This makes it possible to reduce a sense of discomfort such as the stereoscopic image 40 facing an impossible direction. Information such as the direction in which the subject of the stereoscopic image 40 cannot be directed may be set as a rotation constraint.

<< 6. Summary >>
As described above, in the information processing system according to the embodiment of the present disclosure, it is possible to make a two-dimensional image appear as a pseudo three-dimensional image in a three-dimensional space.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.

For example, it is possible to create a computer program for exerting the functions of the information processing terminal 10 or the server 2 on the hardware such as the CPU, ROM, and RAM built in the information processing terminal 10 or the server 2 described above. .. Also provided is a computer-readable storage medium that stores the computer program.

In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The present technology can also have the following configurations.
(1)
Display control that arranges a two-dimensional image obtained by shooting a real object as a stereoscopic image in a three-dimensional space,
Display control that maintains a state in which a straight line connecting the predetermined axis and the user's viewpoint with respect to the predetermined axis on the stereoscopic image is substantially perpendicular to the stereoscopic image.
An information processing device provided with a control unit for performing the above.
(2)
The control unit
The above (1), wherein the stereoscopic image is controlled to be rotated by the predetermined axis so that the straight line connecting the predetermined axis and the user's viewpoint is substantially perpendicular to the stereoscopic image. Information processing device.
(3)
The control unit
The information processing apparatus according to (2) above, which controls rotation within a rotation constraint corresponding to the two-dimensional image.
(4)
The rotation constraint is a constraint on the rotatable axis.
The control unit rotates the stereoscopic image on a axis set to be rotatable among the predetermined axes on the stereoscopic image, and the straight line connecting the predetermined axis and the user's viewpoint is the stereoscopic image. The information processing apparatus according to (3) above, which maintains a state of being substantially perpendicular to a visual image.
(5)
The rotation constraint is a constraint on the rotatable angle.
The information processing device according to (3) or (4), wherein the control unit controls rotation within the rotatable angle when rotating the stereoscopic image on a predetermined axis on the stereoscopic image.
(6)
The control unit
When the straight line connecting the predetermined axis and the user's viewpoint cannot be maintained in a state of being substantially perpendicular to the stereoscopic image due to the rotation constraint, a warning is given, according to the above (3) to (5). The information processing apparatus according to any one of the above items.
(7)
The control unit
When the straight line connecting the predetermined axis and the user viewpoint has an angle of a certain value or more in the direction in which the rotation is restricted with respect to the stereoscopic image, the recommended viewpoint direction is notified. , The information processing apparatus according to (6) above.
(8)
The stereoscopic image is a stereo image of a person, and is a stereoscopic image.
The information processing apparatus according to (4), wherein the rotatable axis is an axis that is perpendicular to the three-dimensional space among predetermined axes on the stereoscopic image.
(9)
The control unit
The information processing device according to any one of (2) to (8) above, which controls the display of a virtual object which is a three-dimensional image according to the position of the stereoscopic image.
(10)
The control unit
The information processing apparatus according to (9), wherein the display coordinates of the virtual object in the three-dimensional space change so as to follow a predetermined position of the stereoscopic image.
(11)
The control unit
(9) or ( The information processing apparatus according to 10).
(12)
The control unit
A localization virtual object of a three-dimensional image localized in the three-dimensional space is displayed, and the object is displayed.
When the user viewpoint changes, the localization virtual object from the user viewpoint is generated and displayed without changing the display coordinates of the localization virtual object in the three-dimensional space. The information processing apparatus according to any one of ().
(13)
The information processing device includes a display unit.
The control unit
Based on the information acquired from the real space, the user's viewpoint in the three-dimensional space is calculated.
The information processing device according to any one of (2) to (12), wherein the generated image of the three-dimensional space of the user's viewpoint is displayed on the display unit.
(14)
The processor,
Display control that arranges a two-dimensional image obtained by shooting a real object as a stereoscopic image in a three-dimensional space,
Display control that maintains a state in which a straight line connecting the predetermined axis and the user's viewpoint with respect to the predetermined axis on the stereoscopic image is substantially perpendicular to the stereoscopic image.
Information processing methods, including doing.
(15)
Computer,
Display control that arranges a two-dimensional image obtained by shooting a real object as a stereoscopic image in a three-dimensional space,
Display control that maintains a state in which a straight line connecting the predetermined axis and the user's viewpoint with respect to the predetermined axis on the stereoscopic image is substantially perpendicular to the stereoscopic image.
A program to function as a control unit that performs

2 Server 3 Network 10 Information processing terminal 100 Control unit 101 User viewpoint recognition unit 102 Virtual space viewpoint calculation unit 103 Image generation unit 104 Display control unit 110 Communication unit 120 Operation input unit 130 Sensor unit 131 Position information acquisition unit 132 Camera 133 Microphone ( Microphone)
134 Accelerometer 135 Biological sensor 140 Display unit 142 Warning display 150 Speaker 160 Storage unit 40 Stereoscopic image 42 Predetermined axis 50a Virtual object (localization virtual object)
50b virtual object (localized virtual object)
60 3D model (virtual object)

Claims (15)

Display control that arranges a two-dimensional image obtained by shooting a real object as a stereoscopic image in a three-dimensional space,
Display control that maintains a state in which a straight line connecting the predetermined axis and the user's viewpoint with respect to the predetermined axis on the stereoscopic image is substantially perpendicular to the stereoscopic image.
An information processing device provided with a control unit for performing the above. The control unit
The information according to claim 1, wherein the stereoscopic image is controlled to be rotated by the predetermined axis so that the straight line connecting the predetermined axis and the user's viewpoint is substantially perpendicular to the stereoscopic image. Processing equipment. The control unit
The information processing apparatus according to claim 2, wherein rotation control is performed within the rotation constraint corresponding to the two-dimensional image. The rotation constraint is a constraint on the rotatable axis.
The control unit rotates the stereoscopic image on a axis set to be rotatable among the predetermined axes on the stereoscopic image, and the straight line connecting the predetermined axis and the user's viewpoint is the stereoscopic image. The information processing apparatus according to claim 3, which maintains a state of being substantially perpendicular to the visual image. The rotation constraint is a constraint on the rotatable angle.
The information processing device according to claim 3, wherein the control unit controls rotation within the rotatable angle when rotating the stereoscopic image on a predetermined axis on the stereoscopic image. The control unit
The information processing according to claim 3, wherein a warning is given when the straight line connecting the predetermined axis and the user's viewpoint cannot be maintained in a state of being substantially perpendicular to the stereoscopic image due to the rotation constraint. Device. The control unit
When the straight line connecting the predetermined axis and the user viewpoint has an angle of a certain value or more in the direction in which the rotation is restricted with respect to the stereoscopic image, the recommended viewpoint direction is notified. , The information processing apparatus according to claim 6. The stereoscopic image is a stereo image of a person, and is a stereoscopic image.
The information processing apparatus according to claim 4, wherein the rotatable axis is an axis that is perpendicular to the three-dimensional space among predetermined axes on the stereoscopic image. The control unit
The information processing apparatus according to claim 2, wherein the information processing device controls to display a virtual object which is a three-dimensional image according to the position of the stereoscopic image. The control unit
The information processing apparatus according to claim 9, wherein the display coordinates of the virtual object in the three-dimensional space change so as to follow a predetermined portion of the stereoscopic image. The control unit
The ninth aspect of claim 9, wherein when the stereoscopic image is rotated about the predetermined axis, the virtual object is rotated with the same rotation amount in the same direction as the rotation of the stereoscopic image with respect to the predetermined axis. Information processing device. The control unit
A localization virtual object of a three-dimensional image localized in the three-dimensional space is displayed, and the object is displayed.
The information according to claim 2, wherein when the user viewpoint changes, the localization virtual object from the user viewpoint is generated and displayed without changing the display coordinates of the localization virtual object in the three-dimensional space. Processing equipment. The information processing device includes a display unit.
The control unit
Based on the information acquired from the real space, the user's viewpoint in the three-dimensional space is calculated.
The information processing device according to claim 2, wherein the generated image of the three-dimensional space of the user's viewpoint is displayed on the display unit. The processor,
Display control that arranges a two-dimensional image obtained by shooting a real object as a stereoscopic image in a three-dimensional space,
Display control that maintains a state in which a straight line connecting the predetermined axis and the user's viewpoint with respect to the predetermined axis on the stereoscopic image is substantially perpendicular to the stereoscopic image.
Information processing methods, including doing. Computer,
Display control that arranges a two-dimensional image obtained by shooting a real object as a stereoscopic image in a three-dimensional space,
Display control that maintains a state in which a straight line connecting the predetermined axis and the user's viewpoint with respect to the predetermined axis on the stereoscopic image is substantially perpendicular to the stereoscopic image.
A program to function as a control unit that performs

Images