JP6649010B2 - Information processing device - Google Patents

Description

  The present invention relates to an information processing apparatus for displaying a stereoscopic video on a video display device, a video display system, a control method for the video display device, and a program.

  2. Description of the Related Art An image display device that realizes display of a stereoscopic image using parallax by allowing a user to view a left-eye image and a right-eye image that is different from the left-eye image for the left eye of a user is known.

  When the image display apparatus described above displays an image prepared for stereoscopic viewing in an inclined state, the position of the same object in the image in the height direction is shifted between the left-eye image and the right-eye image. Would. Then, vertical parallax is generated, so that the user cannot correctly recognize the depth of the object, and may feel uncomfortable.

  The present invention has been made in consideration of the above circumstances, and one of its objects is to provide information capable of reducing a user's uncomfortable feeling when an image in a tilted state is displayed on a video display device. An object of the present invention is to provide a processing device, a video display system, a control method for a video display device, and a program.

  An information processing apparatus according to the present invention is an information processing apparatus connected to a video display apparatus that displays a stereoscopic video, and obtains two captured images representing scenes viewed from two shooting positions in a shooting space. Unit, a display image drawing unit that draws two display images based on the obtained two captured images, and displays the three-dimensional image on the video display device, and specifies a gazing point of the user in the display image. A gaze point specifying unit, wherein the display image drawing unit draws the two display images representing a state in which the scenery represented by the two captured images is tilted, at the position of the specified gaze point. The two display images are drawn so as to reduce a shift in a height direction between corresponding points corresponding to each other in the two captured images.

  Also, the video display system according to the present invention is a video display system including a video display device for displaying a stereoscopic video and an information processing device connected to the video display device, wherein the information processing device An acquisition unit configured to acquire two captured images representing scenery viewed from two imaging positions in a space, a display image rendering unit configured to render two display images based on the acquired two captured images, and the display image A fixation point specifying unit that specifies a fixation point of the user in, the video display device displays two display images drawn by the display image drawing unit as a stereoscopic image, and the display image drawing unit includes: When rendering the two display images representing the state in which the scenes represented by the two captured images are tilted, at the position of the specified point of regard, the corresponding points in the two captured images corresponding to each other Is to reduce the direction of displacement, characterized in that drawing the two display images.

  Further, the control method for a video display device according to the present invention is a control method for a video display device that displays a stereoscopic video, and acquires two captured images representing a scene viewed from two capturing positions in a capturing space. An acquiring step, a display image rendering step of rendering two display images based on the acquired two captured images, and displaying the two display images as a stereoscopic image on the video display device, and specifying a point of interest of the user in the display image The display image drawing step, when drawing the two display images representing a state where the scenes represented by the two captured images are tilted, the position of the specified fixation point Wherein the two display images are drawn so as to reduce a shift in a height direction between corresponding points corresponding to each other in the two captured images.

  Further, the program according to the present invention includes a computer connected to an image display device that displays a stereoscopic image, an acquisition unit that acquires two captured images representing scenes viewed from two imaging positions in an imaging space, Display image drawing means for drawing two display images based on the two photographed images thus obtained and displaying the display image as a stereoscopic image on the video display device, and fixation point specifying means for specifying a fixation point of the user in the display image Wherein the display image drawing means draws the two display images representing a state in which the scenery represented by the two captured images is tilted, At the position, the two display images are drawn so as to reduce a shift in a height direction between corresponding points corresponding to each other in the two captured images.This program may be stored in a computer-readable non-transitory information storage medium and provided.

1 is a configuration block diagram illustrating a configuration of a video display system including an information processing device according to an embodiment of the present invention. FIG. 2 is a functional block diagram illustrating functions of the information processing apparatus according to the embodiment. It is a figure which shows the positional relationship of the imaging | photography position at the time of imaging | photographing a captured image. FIG. 4 is a diagram illustrating a positional relationship between a viewpoint camera and a virtual screen. FIG. 4 is a diagram illustrating a pasting target area on a virtual screen. It is a figure showing a field-of-view range in the state where a viewpoint camera was inclined. FIG. 7 is a diagram illustrating a displacement of a subject in a height direction in a display image in a state where a viewpoint camera is tilted. FIG. 6 is a diagram illustrating an example of a flow of a process executed by the information processing apparatus according to the embodiment. FIG. 6 is a diagram illustrating a positional relationship between a viewpoint camera and a virtual screen when a display image is drawn based on a captured image including a scene of the entire sky.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a configuration block diagram illustrating a configuration of a video display system 1 including an information processing device 10 according to an embodiment of the present invention. As shown in FIG. 1, the video display system 1 includes an information processing device 10, an operation device 20, a camera 25, a relay device 30, and a video display device 40.

  The information processing device 10 is a device that supplies an image to be displayed by the image display device 40, and may be, for example, a home game machine, a portable game machine, a personal computer, a smartphone, a tablet, or the like. As shown in FIG. 1, the information processing device 10 includes a control unit 11, a storage unit 12, and an interface unit 13.

  The control unit 11 includes at least one processor such as a CPU, and executes a program stored in the storage unit 12 to execute various types of information processing. Note that a specific example of the processing executed by the control unit 11 in the present embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores a program executed by the control unit 11 and data processed by the program.

  The interface unit 13 is an interface for data communication between the operation device 20, the camera 25, and the relay device 30. The information processing device 10 is connected to each of the operation device 20, the camera 25, and the relay device 30 via the interface unit 13 in a wired or wireless manner. As a specific example, the interface unit 13 may include a multimedia interface such as HDMI (High-Definition Multimedia Interface: registered trademark) in order to transmit the video and audio supplied by the information processing device 10 to the relay device 30. Further, a data communication interface such as a USB may be included in order to receive various kinds of information from the video display device 40 via the relay device 30 and to transmit control signals and the like. Further, the interface unit 13 includes a data communication interface such as Bluetooth (registered trademark) or USB in order to receive a signal indicating the content of a user's operation input to the operation device 20. Further, the interface unit 13 includes a data communication interface such as a USB in order to receive data of an image captured by the camera 25.

  The operation device 20 is a controller or the like of a home game machine, and is used by a user to perform various instruction operations on the information processing device 10. The content of the user's operation input to the operation device 20 is transmitted to the information processing apparatus 10 either by wire or wirelessly. The operation device 20 may include an operation button, a touch panel, and the like disposed on the surface of the housing of the information processing apparatus 10. The camera 25 is installed in front of the user using the video display system 1 and captures a video including the video display device 40 worn by the user.

  The relay device 30 is connected to the video display device 40 either by wire or wirelessly, receives the video data supplied from the information processing device 10, and outputs a video signal corresponding to the received data to the video display device 40. Output to At this time, the relay device 30 may perform processing for correcting distortion generated by the optical system of the video display device 40 on the supplied video data as needed, and output the corrected video signal. Good. Note that the video signal supplied from the relay device 30 to the video display device 40 includes two videos: a left-eye video and a right-eye video. The relay device 30 relays various information transmitted and received between the information processing device 10 and the video display device 40, such as audio data and control signals, in addition to the video data.

  The video display device 40 displays a video according to the video signal input from the relay device 30, and allows the user to browse. In the present embodiment, it is assumed that the video display device 40 is a video display device that is worn on the user's head and used for browsing video with both eyes. That is, the image display device 40 displays an image in front of each of the right eye and the left eye of the user. The video display device 40 displays a stereoscopic video using binocular parallax. As shown in FIG. 2, the video display device 40 includes a video display element 41, an optical element 42, a light emitting element 43, a motion sensor 44, and a communication interface 45.

  The image display element 41 is an organic EL display panel, a liquid crystal display panel, or the like, and displays an image according to an image signal supplied from the relay device 30. In the present embodiment, the image display element 41 displays two images, a left-eye image and a right-eye image. Note that the video display element 41 may be a single display element that displays a left-eye video and a right-eye video side by side, or may be configured by two display elements that display each video independently. In addition, a known smartphone or the like may be used as the image display element 41. Note that the image display element 41 displays different images as a left-eye image and a right-eye image in order to realize stereoscopic viewing.

  The optical element 42 is a hologram, a prism, a half mirror, or the like, is arranged in front of the user's eyes, transmits or refracts light of an image displayed by the image display element 41, and is incident on the left and right eyes of the user. Let it. Specifically, the image for the left eye displayed by the image display element 41 enters the user's left eye via the optical element 42, and the image for the right eye enters the user's right eye via the optical element 42. This allows the user to view the left-eye video with the left eye and the right-eye video with the right eye with the video display device 40 mounted on the head. In the present embodiment, the video display device 40 is a non-transmissive video display device that does not allow the user to visually recognize the outside world.

  The light emitting element 43 is an LED or the like, is arranged on the front of the video display device 40, and emits light in response to an instruction from the information processing device 10. By photographing the light of the light emitting element 43 with the camera 25, the information processing device 10 can specify the position of the video display device 40 within the field of view of the camera 25. That is, the information processing device 10 can track changes in the position and the inclination of the video display device 40 using the light of the light emitting element 43. Note that a plurality of light emitting elements 43 may be arranged.

  The motion sensor 44 measures various information related to the position, the direction, and the movement of the video display device 40. For example, the motion sensor 44 may include an acceleration sensor, a gyroscope, a geomagnetic sensor, or the like. The measurement result of the motion sensor 44 is transmitted to the information processing device 10 via the relay device 30. The information processing device 10 can use the measurement result of the motion sensor 44 in order to specify a change in the motion or the direction of the video display device 40. Specifically, the information processing device 10 can detect the inclination or the parallel movement of the video display device 40 with respect to the vertical direction by using the measurement result of the acceleration sensor. In addition, by using the measurement result of the gyroscope or the geomagnetic sensor, the rotational movement of the image display device 40 can be detected.

  The communication interface 45 is an interface for performing data communication with the relay device 30. For example, when the image display device 40 transmits and receives data to and from the relay device 30 by wireless communication such as a wireless LAN or Bluetooth, the communication interface 45 includes a communication antenna and a communication module. Further, the communication interface 45 may include a communication interface such as HDMI or USB for performing wired data communication with the relay device 30.

  Next, functions realized by the information processing apparatus 10 will be described with reference to FIG. As illustrated in FIG. 2, the information processing apparatus 10 functionally includes a captured image acquisition unit 51, a display image drawing unit 52, a viewpoint control unit 53, and a gazing point identification unit 54. These functions are realized by the control unit 11 executing a program stored in the storage unit 12. This program may be provided to the information processing apparatus 10 via a communication network such as the Internet, or may be provided by being stored in a computer-readable information storage medium such as an optical disk.

  The photographed image acquiring unit 51 acquires two photographed images representing the scenery viewed from two photographing positions in the photographing space. Here, it is specifically assumed that the photographing space is a real space, and the photographed image is obtained by photographing an actual scene with a camera. Further, the two photographing positions are positions arranged right and left along the horizontal direction in the real space. By photographing a subject in the photographing space from the camera installed at each of the two photographing positions, two photographed images can be obtained. Hereinafter, the cameras arranged at the two photographing positions are referred to as a left photographing camera CL and a right photographing camera CR. Further, a photographed image obtained by photographing the photographing space with the left photographing camera CL is described as a left photographed image PL, and a photographed image obtained by photographing the photographing space with the right photographing camera CR is described as a right photographed image PR. The middle point between the two shooting positions is referred to as a reference shooting position Rc. It is assumed that the two photographing positions are separated from each other by a distance dc.

  FIG. 3 is a diagram showing a state in which a photographed image is photographed by two photographing cameras, and shows a state in which the two photographing cameras are viewed from the zenith direction. The broken line in the figure indicates the horizontal angle of view of the photographing camera, and a subject existing within the photographing range surrounded by the broken line is included in the photographed image. In the example of this figure, the optical axes (photographing directions) of the two cameras are parallel to each other, and the photographing directions are orthogonal to the vertical plane including the two photographing positions. In addition, as a specific example, it is assumed that the captured image includes the subject S, and the distance from the two capturing positions to the subject S along the capturing direction is dx.

  Hereinafter, a point where the same position in the photographing space appears in two photographed images is referred to as a corresponding point. By photographing from two photographing positions arranged along the horizontal direction, the corresponding points in the two photographed images are displayed at positions shifted from each other in the horizontal direction. Here, since the shooting directions of the two shooting cameras are assumed to be the same, the two corresponding points are arranged closer to each other in each shot image as the distance from the shooting position increases, and the distance from the shooting position decreases. The closer to each other, the farther apart they are. Due to the shift of the corresponding point in the left-right direction, stereoscopic viewing by binocular parallax is realized.

  The display image rendering unit 52 renders two display images based on the two captured images acquired by the captured image acquisition unit 51, and outputs the rendered display image to the relay device 30 so that the video display device 40 To be displayed. Specifically, the display image drawing unit 52 draws the left-eye display image DL based on the left captured image PL, and draws the right-eye display image DR based on the right captured image PR. Then, by displaying the two display images on the video display device 40, the left-view display image DL is viewed by the user's left eye and the right-view display image DR is viewed by the right eye.

  In the present embodiment, the display image drawing unit 52 draws each of the left-eye display image DL and the right-eye display image DR as follows. That is, the display image drawing unit 52 arranges the flat virtual screen VS in the virtual space, and positions the virtual screen VS corresponding to the left and right eyes of the user at a position where at least a part of the virtual screen VS is included in the visual field. Place two viewpoint cameras. Hereinafter, these two viewpoint cameras are referred to as a left viewpoint camera VL and a right viewpoint camera VR. The left viewpoint camera VL and the right viewpoint camera VR are arranged in the virtual space side by side at a predetermined distance from each other like the left and right eyes of a person. Hereinafter, the midpoint between the arrangement positions of the two viewpoint cameras is referred to as a reference viewpoint position Rv. The distance between the two viewpoint cameras is dv. Here, it is assumed that the distance dv matches the distance dc between the two photographing positions in the real space. The optical axes (viewing directions) of the two viewpoint cameras are parallel to each other as in the case of the photographing camera.

  FIG. 4 illustrates a virtual screen VS and a virtual space in which two viewpoint cameras are arranged, as viewed from above. The broken line in the figure indicates the horizontal angle of view of the two viewpoint cameras. In the figure, the virtual screen VS is arranged at a position separated from the reference viewpoint position Rv by a distance ds along the line of sight. Also, here, the two viewpoint cameras are arranged in a plane parallel to the virtual screen VS, and the viewing direction is orthogonal to the surface of the virtual screen VS.

  Under the conditions described above, the display image drawing unit 52 pastes the left captured image PL as a texture on the surface of the virtual screen VS, and displays the left captured image PL as viewed from the left viewpoint camera VL on the left eye. Is drawn as the display image DL for use. Similarly, an image showing the surface of the virtual screen VS on which the right captured image PR is pasted as a texture as viewed from the right viewpoint camera VR is drawn as a right-eye display image DR. Then, the rendered left-eye display image DL and right-eye display image DR are displayed on the video display device 40, and the user browses with the left and right eyes. Thus, the user can three-dimensionally recognize the scene viewed from the two shooting positions.

  Note that the position and size of each of the two captured images when pasting them on the virtual screen VS are determined according to the position and angle of view of the camera that captured each captured image. Specifically, it is assumed that the two photographing cameras are arranged in the virtual space while maintaining the positional relationship at the time of photographing such that the reference photographing position Rc matches the reference viewpoint position Rv in the virtual space. At this time, the two photographing cameras are arranged in the horizontal direction as in photographing, and are arranged such that the vertical plane including the two photographing cameras is parallel to the virtual screen VS. In this case, an area occupied on the virtual screen VS by an imaging range determined according to the angle of view of each imaging camera is an area where a captured image is pasted. Hereinafter, the area on the virtual screen VS where the left captured image PL is pasted is referred to as a pasting target area AL, and the area where the right captured image PR is pasted is referred to as a pasting target area AR. The pasting target area AL and the pasting target area AR are areas shifted from each other by a distance dc along the horizontal direction on the virtual screen VS. The size of each pasting target area is determined according to the angle of view of the photographing camera and the distance ds from the reference viewpoint position Rv to the virtual screen VS. Each captured image is enlarged or reduced in accordance with the size of the paste target area, and is pasted on the corresponding paste target area. FIG. 5 is a diagram showing the pasting target areas AL and AR on the virtual screen VS, and shows a state in which the virtual screen VS is viewed from behind two viewpoint cameras. Note that the broken lines in the figure indicate the visual field ranges of the two viewpoint cameras.

  The viewpoint controller 53 controls the positions of two viewpoint cameras arranged in the virtual space. In the present embodiment, the viewpoint control unit 53 executes rotation control for rotating the two viewpoint cameras according to a change in the inclination of the video display device 40. Specifically, the viewpoint control unit 53 acquires the measurement result of the motion sensor 44 or the image captured by the camera 25 at regular time intervals, and analyzes the acquired image to specify the inclination of the video display device 40 at the time of acquisition. Here, the inclination of the video display device 40 is defined by a roll angle representing the amount of rotation of the video display device 40 with the front-back direction as a rotation axis. Since the video display device 40 is used while being fixed to the user's head, the tilt of the video display device 40 directly corresponds to the tilt of the user's face. Here, the direction that coincides with the front of the user's face when the video display device 40 is worn is defined as the front direction (front direction) of the video display device 40. Further, the value of the roll angle may represent the amount of rotation from this reference state with the state in which the left and right eyes are horizontal without the user tilting the face as a reference state.

  The viewpoint control unit 53 rotates the two viewpoint cameras in the vertical plane in conjunction with the change in the specified tilt of the video display device 40. FIG. 6 is a view showing a state in which the viewpoint camera is rotated, and shows a state in which two viewpoint cameras arranged in the virtual space are viewed from the back direction, similarly to FIG. The broken lines in the figure indicate the visual field ranges of the two viewpoint cameras. As shown in the drawing, the two viewpoint cameras rotate around the reference viewpoint position Rv as a rotation center and the viewing line direction of the viewpoint camera as a rotation axis. The rotation direction and the rotation amount φ may match the specified roll angle of the video display device 40. As shown in FIG. 6, even if the viewpoint camera rotates, the positions and directions of the pasting target areas AR and AL do not change. Therefore, the captured images PR and PL included in the visual field range of the viewpoint camera are tilted in conjunction with the rotation of the viewpoint camera. The display image drawing unit 52 draws, as a display image, a state of viewing the virtual screen VS on which the corresponding captured image is pasted from each viewpoint camera rotated in this way. This allows the user to view the scenery represented by the two captured images in a state where the scene is tilted according to a change in the inclination of his / her face.

  Note that the viewpoint control unit 53 may specify not only the inclination of the video display device 40 but also the direction in which the front surface of the video display device 40 is facing. For example, the direction of the video display device 40 is defined by a yaw angle indicating a rotation amount with the vertical direction as a rotation axis, and a pitch angle indicating a rotation amount with the horizontal direction as a rotation axis. The values of the yaw angle and the pitch angle may each represent the amount of rotation with respect to the reference direction. Here, the reference direction may be a direction determined when the use of the video display device 40 is started. Further, the viewpoint control unit 53 specifies not only the inclination and the direction of the video display device 40 but also the direction and the amount of translation when the video display device 40 is translated and its position is changed. Good. Then, the viewpoint control unit 53 may change the line-of-sight directions and the positions of the two viewpoint cameras arranged in the virtual space according to the change in the direction and the position of the video display device 40. This allows the viewpoint camera to change the position and orientation of the viewpoint camera in conjunction with the movement of the user's head when the line-of-sight camera has only a limited area in the captured image as the field of view, and to set an arbitrary position in the captured image. Can be included in the field of view of the viewpoint camera.

  However, when the control for tilting the two viewpoint cameras is performed as described above, a vertical parallax occurs in the subject in the captured image, and it may be difficult for the user to recognize the subject three-dimensionally. This will be described below. As described above, in the left photographed image PL and the right photographed image PR, the corresponding points corresponding to the same position in the photographing space are shown at positions shifted in the left-right direction. Therefore, when each photographed image is viewed from an inclined viewpoint, the positions of the two corresponding points are shifted in the height direction. FIG. 7 is a schematic diagram for explaining the displacement in the height direction, and is a left-eye display image DL and a right-eye display image drawn based on the left photographed image PL and the right photographed image PR in which the subject S is captured. DR is shown. The upper part shows a display image before tilting the viewpoint camera (state in which the viewpoint cameras are arranged in a horizontal direction), and the lower part shows a display image in a state where the viewpoint camera is tilted. As shown in the upper part, between the left-eye display image DL and the right-eye display image DR, the position of the subject S in the height direction matches, but the position in the left-right direction is shifted. The user can three-dimensionally recognize the subject S by the shift in the left-right direction. On the other hand, when the virtual screen VS is viewed from the rotated viewpoint camera, the height direction of the subject S is shifted between the left-eye display image DL and the right-eye display image DR as shown in the lower part of FIG. It can be seen that a shift occurs in the position of. When such a display image is displayed, vertical parallax occurs, and it is difficult for the user to correctly recognize the depth of the subject S based on binocular parallax.

  Thus, the information processing apparatus 10 according to the present embodiment draws a display image so that the user can correctly recognize a subject appearing at a position of interest in the captured image as a stereoscopic video. Hereinafter, a position in the display image to which the user pays attention is referred to as a point of interest, and a position corresponding to the point of interest in the captured image is referred to as a point of interest. In addition, a subject appearing at a target position in a captured image is referred to as a target subject. By executing the drawing processing described below, the vertical parallax as described above may occur for subjects other than the subject of interest (especially subjects whose distance from the shooting position is significantly different from the subject of interest). The depth of the subject can be correctly recognized. This can reduce the discomfort felt by the user while viewing the stereoscopic video. Hereinafter, a specific example of such a drawing process will be described.

  In order to realize the above-described drawing process, the gazing point specifying unit 54 specifies the gazing point of the user when the display image drawn by the display image drawing unit 52 is displayed on the video display device 40. Specifically, for example, the gazing point specifying unit 54 specifies, as the gazing point, the position where the user is actually looking at the line of sight in the displayed image being displayed. In this case, it is assumed that the camera is mounted at a position in the video display device 40 where the left and right eyes of the user can be imaged. The gazing point identification unit 54 analyzes the image of the user's eyes captured by the camera, and identifies the line of sight of each of the left and right eyes of the user. Then, the position in the display image to which the specified line of sight is directed is specified as the point of gaze.

  Alternatively, the gazing point specifying unit 54 may specify the center position of the display image as the gazing point, assuming that the user is gazing at the center of the display image. In this case, the position of the gazing point in the display image is fixed and does not change, but when the direction or the position of the viewpoint camera changes, the attention position in the captured image changes accordingly. .

  Further, the gazing point specifying unit 54 may specify the gazing point according to the content of the operation input to the operation device 20 by the user. In this case, for example, the gazing point identifying unit 54 displays a marker indicating the gazing point on the video display device 40 so as to be superimposed on the display image, and moves the marker in accordance with an operation input indicating a user's direction. Then, the position in the display image overlapping the marker is specified as the user's point of gaze.

  When the gazing point identifying unit 54 identifies the gazing point, the display image drawing unit 52 reduces the height-direction deviation of the corresponding points in the two captured images at the identified gazing point position. Draw two display images. Accordingly, the vertical parallax is reduced at least at the position of the gazing point, and it becomes easier for the user to three-dimensionally recognize the subject of interest at that position. Hereinafter, a specific example of a drawing method for reducing such vertical parallax will be described.

  First, the display image drawing unit 52 specifies a position of interest in the captured image corresponding to the point of gaze. Specifically, the point of interest in the left photographed image PL or the right photographed image PR pasted on the virtual screen VS is specified by converting the point of gaze in the display image into position coordinates on the virtual screen VS. The subject shown in the noted position is the noted subject.

  Next, the display image drawing unit 52 specifies the distance dn along the shooting direction from the shooting position in the shooting space to the subject of interest. For example, the display image drawing unit 52 converts the amount of parallax into the distance dn to the subject of interest by measuring the amount of shift (parallax) in the left-right direction of the target position in the two captured images. Alternatively, the display image drawing unit 52 may specify the distance dn to the subject of interest using a distance image (depth map) prepared in advance together with the captured image. In this case, the distance image is generated using a distance sensor or the like at the time of capturing the captured image. This distance image is an image including the same shooting range as the shot image, and is an image in which the pixel value of each pixel in the image is a numerical value indicating the distance to the subject located at the position. If such a distance image is prepared in advance, the display image drawing unit 52 can easily specify the distance dn by referring to the distance image.

  In addition, when specifying the distance dn, the display image drawing unit 52 does not use only one point of interest corresponding to the gazing point to specify the distance to the object of interest corresponding to the one point. The distance dn may be determined by specifying the distance to the subject within a predetermined range including the target position in. The predetermined range in this case may be, for example, a rectangular area of a predetermined size centered on the target position. In this case, the display image drawing unit 52 specifies, for each of the plurality of pixels included in a predetermined range of the captured image, the distance to the subject shown in the pixel. Then, the representative value of the distance for each specified pixel is calculated as the distance dn to the target subject to be obtained. The representative value in this case may be, for example, an average value or a minimum value. By adopting the minimum value of the distances to the subject shown in each pixel as the distance dn, the subject located closest to the user in the range of interest of the user can be set as the subject of interest. According to such control, it is possible to prevent the value of the distance dn specified by the display image drawing unit 52 from frequently changing.

  Subsequently, the display image drawing unit 52 arranges the virtual screen VS at a position in the virtual space determined according to the specified value of the distance dn. Specifically, the virtual screen VS is arranged at a position separated from the reference viewpoint position Rv by the same distance as the distance dn. By arranging the virtual screen VS at the position determined in this way and pasting each photographed image on its surface, the amount of parallax between the two photographed images of the subject of interest matches the distance dc, and the left photographed image PL And the target object in the right photographed image PR are pasted at the same position on the virtual screen VS. Accordingly, when the captured images on the virtual screen VS are viewed from each viewpoint camera in a state where the two viewpoint cameras are tilted, the position in the height direction of the subject of interest becomes the same between the two display images. Therefore, the user can three-dimensionally recognize the subject of interest without causing vertical parallax.

  As a specific example, it is assumed that when photographing the photographing space shown in FIG. In this case, the virtual screen VS is arranged such that the distance dx from the reference shooting position Rc to the subject S matches the distance ds from the reference viewpoint position Rv to the virtual screen VS. Thereby, when the photographed image pasted on the virtual screen VS is viewed from each of the two viewpoint cameras arranged in the virtual space in the same positional relation as the photographing camera, the subject S is positioned at the same position on the virtual screen VS. Will look like Therefore, even if the viewpoint camera is tilted, no vertical parallax occurs between the subjects S in the two display images, and the user can correctly recognize the depth of the subjects S.

  Here, it is assumed that the distance dc between the two photographing positions and the distance dv between the two viewpoint cameras match, but even if the distance dc and the distance dv are different, the distance dc from the reference viewpoint position Rc is assumed to be virtual. The distance ds to the screen VS can be determined according to the distance dn from the reference shooting position Rc to the subject of interest regardless of the distance dv. That is, by matching the distance ds at which the virtual screen VS is arranged with the distance dn from the reference shooting position Rc to the target object, the target object is pasted at the same position on the virtual screen VS. Control can be performed such that vertical parallax does not occur in the target object when viewed.

  In the above description, after detecting the point of interest at a certain point in time, the virtual screen VS is immediately arranged at a position determined according to the distance dn to the target subject corresponding to the point of interest. However, according to such control, the position of the virtual screen VS greatly moves back and forth with a change in the point of regard. Therefore, the arrangement position of the virtual screen VS may be determined based on a value obtained by performing a low-pass filter process on a numerical sequence of the distance dn determined according to the point of gaze for each unit time. As a result, the position of the virtual screen VS can be prevented from suddenly changing.

  Hereinafter, an example of a flow of a drawing process in which the information processing apparatus 10 according to the present embodiment draws a display image will be described with reference to a flowchart of FIG. The process shown in this figure is repeatedly executed at predetermined time intervals while the display image is displayed on the video display device 40.

  First, the viewpoint control unit 53 acquires information on the posture (inclination, direction, and position) of the video display device 40 at the time of execution of the processing (STEP 1), and two viewpoint cameras in the virtual space according to the acquired information. Is updated (STEP 2).

  Next, the gazing point specifying unit 54 specifies the gazing point of the user at the time of execution of the process (STEP 3). Then, the display image drawing unit 52 determines the arrangement position of the virtual screen VS in the virtual space, using the distance dn to the subject at the position of interest in the captured image corresponding to the specified point of gaze ( (STEP 4).

  After that, the display image drawing unit 52 sees the captured image pasted on the virtual screen VS arranged at the position determined in STEP 4 from each of the two viewpoint cameras arranged in the position determined in STEP 2 A display image showing the situation is drawn (STEP 5). Then, the rendered two display images are output to the relay device 30 to be displayed on the video display device 40 (STEP 6).

  By repeatedly executing such processing at predetermined time intervals, the information processing apparatus 10 can link the user's gaze point and head movement in real time, However, the stereoscopic video can be displayed in such a manner that the subject of interest can be recognized three-dimensionally.

  In the above description, the photographed image is an image including a scene within a certain photographing range. However, for example, a panoramic image obtained by photographing a wider range with an omnidirectional camera or the like may be used as the photographed image. Good. As a specific example, a case where a panoramic image including the entire sky view is used as a captured image will be described below. Here, a panoramic image including an entire sky view will be described as an example, but the same processing can be applied to a case where a captured image obtained by capturing a partial range is used.

  Also in this example, two shot images are generated by shooting the entire sky view from each of the two shooting positions arranged in the horizontal direction. For example, such a photographed image is generated by arranging two photographing cameras side by side and performing photographing while rotating 360 degrees around the midpoint between the two photographing cameras. These captured images may be recorded in a data format such as equirectangular projection.

  The display image drawing unit 52 of the information processing device 10 draws two display images based on such a captured image. Specifically, in this example, the inner surface of the sphere becomes the virtual screen VS. The display image drawing unit 52 arranges the spherical virtual screen VS in the virtual space, and arranges two viewpoint cameras arranged side by side at the center of the sphere. The display image drawing unit 52 pastes a texture, which is generated based on the left photographed image PL, representing the view seen from the photographing position on the inner surface of the sphere, and looks at the state viewed from the left viewpoint camera VL for the left eye. Draw as a display image DL. Similarly, a texture representing a scene generated based on the right photographed image PR is pasted on the inner surface of the sphere, and a state viewed from the right viewpoint camera VR is drawn as a right-eye display image DR. By displaying these two display images on the video display device 40, the user can view the scenery in the captured image as a stereoscopic image, as in the example of the flat virtual screen VS described above.

  Further, the viewpoint control unit 53 controls the position of the viewpoint camera according to the change in the inclination, direction, and position of the video display device 40. In particular, the viewpoint control unit 53 changes the line-of-sight direction of the line-of-sight camera in accordance with the change in the direction of the video display device 40, so that the user can view the landscape in any direction around the entire sky.

  Further, the display image drawing unit 52 performs two-dimensional recognition so that the target subject in the target position corresponding to the target point can be three-dimensionally recognized in accordance with the change in the user's target point specified by the target point specifying unit 54. Draw two display images. Specifically, similarly to the above-described example of the flat virtual screen VS, the distance dn from the reference shooting position Rc to the target object and the distance ds from the reference viewpoint position Rv to the virtual screen VS match. , A virtual screen VS is arranged. Here, since the virtual screen VS is a spherical surface and the reference viewpoint position Rv is located at the center of the sphere, the distance ds from the reference viewpoint position Rv to the virtual screen VS is the radius of the sphere. Therefore, the display image drawing unit 52 changes the size of the sphere according to the distance dn. FIG. 9 is a diagram showing a positional relationship between the two viewpoint cameras and the spherical virtual screen VS in this example, and shows a state viewed from above.

  Note that the display image drawing unit 52 changes not only the size of the virtual screen VS but also the position where the texture is attached to the virtual screen VS based on the two captured images according to the point of gaze. More specifically, the positions at which the two textures are pasted are shifted so that the pasting position of the subject of interest on the virtual screen VS matches between the two textures. Thereby, the subject of interest can be arranged in the two display images with the amount of parallax corresponding to the distance dn, and vertical parallax can be prevented from occurring in the subject of interest even when the viewpoint camera is tilted.

  According to the information processing apparatus 10 according to the present embodiment described above, when a state where a captured image is tilted is displayed as a display image, the vertical parallax of the position of interest of the user is reduced, and the image is captured at that position. It is possible to display a stereoscopic image that allows the user to stereoscopically recognize an object of interest. Therefore, it is possible to reduce a sense of discomfort when the user views the stereoscopic video.

  The embodiments of the present invention are not limited to those described above. For example, in the above description, the photographing space is a real space, and the photographed image is obtained by photographing a real scene. However, the photographed image is not limited thereto, and the photographed image may be obtained from a photographing position arranged in the virtual space. It may be an image showing the appearance of the virtual space.

  In the above description, the video display device 40 is a video display device used by being worn on the head of the user. However, the present invention is not limited to this. It may be of a type that is installed and used in In this case, the gazing point specifying unit 54 may specify the gazing point of the user by photographing the user's eyes using a camera arranged in front of the user. Further, the viewpoint control unit 53 may move the viewpoint camera according to an operation input to the operation device 20 by the user, instead of changing the inclination and the direction of the viewpoint camera according to the movement of the user's head.

  In the above description, the captured image is a still image, but the captured image may be a moving image that changes with time. Further, in this case, the subject is located relatively close to the shooting position in the shooting space and moves in the shooting range (that is, a subject such as an animal or a vehicle that moves relatively to another background). ) May be executed by assuming that the user pays attention to the moving subject as the subject of interest.

  REFERENCE SIGNS LIST 1 video display system, 10 information processing device, 11 control unit, 12 storage unit, 13 interface unit, 20 operation device, 25 camera, 30 relay device, 40 video display device, 41 video display element, 42 optical element, 43 light emitting element , 44 motion sensor, 45 communication interface, 51 captured image acquisition unit, 52 display image drawing unit, 53 viewpoint control unit, 54 gaze point identification unit.

Claims (8)

An information processing device connected to a video display device that displays a stereoscopic video,
An acquisition unit that acquires two captured images representing the scenery viewed from the two imaging positions in the imaging space,
The two scenes shown by the two captured images acquired are pasted on a virtual screen arranged in a virtual space, and the virtual screen is viewed from two viewpoint cameras arranged side by side in the virtual space. A display image drawing unit that draws a display image and causes the video display device to display the display image as a stereoscopic image;
A gazing point specifying unit that specifies a gazing point of the user in the display image,
Including
The display image drawing unit, when drawing the two display images representing a state in which the scenery represented by the two captured images is tilted, to the target subject in the position corresponding to the specified gazing point as the distance along the imaging direction from the midpoint of the two shooting position, the distance from the midpoint of the previous SL two view camera to the virtual screen along the direction of the two-view camera matches An information processing apparatus, comprising: arranging the virtual screen to draw the two display images so as to reduce a shift in the height direction of the subject of interest in the two display images. The information processing apparatus according to claim 1,
The video display device is a display device used by the user wearing on the head,
The information processing apparatus further includes a viewpoint control unit that rotates the two viewpoint cameras according to the inclination of the video display device,
The information processing apparatus according to claim 1, wherein the display image drawing unit draws the two display images representing a state in which the scenery represented by the two captured images is tilted according to the rotation of the two viewpoint cameras . The information processing apparatus according to claim 1, wherein
The information processing apparatus, wherein the display image drawing unit calculates a distance from the shooting position to the subject of interest based on a parallax amount in the two captured images of the subject of interest. The information processing apparatus according to claim 1, wherein
The display image drawing unit specifies, in the photographed image, a subject closest to the distance from the photographing position as a subject of interest, among subjects appearing within a predetermined range corresponding to the specified point of regard. An information processing apparatus characterized in that: The information processing apparatus according to any one of claims 1 to 4,
The information processing apparatus according to claim 1, wherein the gazing point identifying unit identifies the gazing point by detecting a gaze direction of the user. A video display system that includes a video display device that displays a stereoscopic video and an information processing device connected to the video display device,
The information processing device,
An acquisition unit that acquires two captured images representing the scenery viewed from the two imaging positions in the imaging space,
The two scenes shown by the two captured images acquired are pasted on a virtual screen arranged in a virtual space, and the virtual screen is viewed from two viewpoint cameras arranged side by side in the virtual space. A display image drawing unit for drawing a display image,
A gazing point specifying unit that specifies a gazing point of the user in the display image,
Including
The image display device displays two display images drawn by the display image drawing unit as a stereoscopic image,
The display image drawing unit, when drawing the two display images representing a state in which the scenery represented by the two captured images is tilted, to the target subject in the position corresponding to the specified gazing point as the distance along the imaging direction from the midpoint of the two shooting position, the distance from the midpoint of the previous SL two view camera to the virtual screen along the direction of the two-view camera matches A video display system, comprising: arranging the virtual screen to draw the two display images so as to reduce a shift in the height direction of the subject of interest in the two display images. A control method of a video display device that displays a stereoscopic video,
An acquisition step of acquiring two photographed images representing the scenery viewed from two photographing positions in the photographing space,
Two views showing the appearance of the scenes represented by the two captured images acquired are pasted on a virtual screen arranged in a virtual space, and the virtual screen is viewed from two viewpoint cameras arranged side by side in the virtual space. A display image drawing step of drawing a display image and displaying the display image as a stereoscopic image on the video display device;
A gazing point specifying step of specifying a gazing point of the user in the display image,
Including
In the display image drawing step, when drawing the two display images representing a state in which the scenes represented by the two captured images are tilted, the display image drawing step is performed until the target subject in the position corresponding to the specified gazing point. as the distance along the imaging direction from the midpoint of the two shooting position, the distance from the midpoint of the previous SL two view camera to the virtual screen along the direction of the two-view camera matches A method of controlling a video display device, comprising: arranging the virtual screen to draw the two display images so as to reduce the height difference of the subject of interest in the two display images. . A computer connected to a video display device that displays stereoscopic video,
Acquisition means for acquiring two captured images representing the scenery viewed from the two imaging positions in the imaging space,
The two scenes shown by the two captured images acquired are pasted on a virtual screen arranged in a virtual space, and the virtual screen is viewed from two viewpoint cameras arranged side by side in the virtual space. A display image drawing unit for drawing a display image and displaying the display image as a stereoscopic image on the video display device; and
Gazing point specifying means for specifying the gazing point of the user in the display image;
A program to function as
The display image drawing means, when drawing the two display images representing a state in which the scenery represented by the two captured images is tilted, to a target subject in a position corresponding to the specified gazing point. as the distance along the imaging direction from the midpoint of the two shooting position, the distance from the midpoint of the previous SL two view camera to the virtual screen along the direction of the two-view camera matches A non-transitory computer-readable storage medium storing a program, wherein, by arranging the virtual screen, the two display images are drawn so as to reduce a shift in the height direction of the subject of interest in the two display images.

Images