JP6378794B1 - Image processing apparatus, image processing program, and image processing method - Google Patents

Abstract

When a stereoscopic image generated by a geometry pipeline stereoscopic image generation process is displayed on a user terminal, a left-eye compatible camera and a right that are generated when the user terminal faces in the left-right direction from the front direction. Suppresses the reduction in parallax with the eye-compatible camera.
A viewpoint setting unit 56 determines a left-eye viewpoint for a left-eye corresponding stereo model 74L in the world coordinate system and a right for a right-eye corresponding stereo model 74R in the world coordinate system based on the orientation of the user terminal 16. Set the eye viewpoint. Thus, for each of the left eye viewpoint and the right eye viewpoint, the line-of-sight direction is the Z-axis negative direction, the direction of the upper vector is the Y-axis positive direction, and the direction direction constituting the right-handed system with respect to the Z-axis and Y-axis is X A view coordinate system with a positive axis is defined. The viewpoint correction unit 58 moves the left eye viewpoint in the _XVL axis negative direction in the left eye view coordinate system, and moves the right eye viewpoint in the X _VR axis positive direction in the right eye view coordinate system.
[Selection] Figure 5

Description

  The present invention relates to an image processing apparatus, an image processing program, and an image processing method.

  Since the left and right eyes of a human are at different positions (a predetermined distance apart), the images obtained with the left and right eyes are slightly different. Due to the difference between the images obtained by the left and right eyes, a human can obtain a stereoscopic effect. The difference between the images obtained by the left and right eyes is called binocular parallax.

  Conventionally, a system that simulates binocular parallax and provides an image (hereinafter referred to as “stereoscopic image”) that allows a viewer to obtain a stereoscopic effect has been proposed (for example, Patent Document 1). Specifically, a system has been proposed in which a left-eye camera and a right-eye camera are installed at a predetermined distance, a left-eye camera is used to capture a left-eye camera, and a right-eye camera is used to capture a right-eye camera. Has been. The predetermined distance between the two cameras is determined based on the distance between the human eyes. Since the left-eye compatible camera and the right-eye compatible camera are installed at a predetermined distance, a parallax occurs between the left-eye compatible camera and the right-eye compatible camera. As a result, the left-eye compatible image and the right-eye compatible image Then, the image is slightly different depending on the parallax. Then, the stereoscopic image is generated by combining the left eye corresponding image and the right eye corresponding image. Various modes have been proposed for stereoscopic images. For example, left-eye corresponding image and right-eye corresponding image are displayed alternately, left-eye corresponding image and right-eye corresponding image are alternately arranged every other horizontal line, or left-eye compatible For example, the image and the right eye corresponding image are arranged in parallel. For such a stereoscopic image, the user can obtain a stereoscopic effect by viewing the left eye corresponding image with the left eye and viewing the right eye corresponding image with the right eye.

Japanese Patent Laid-Open No. 2002-232913

  A system is conceivable in which a stereoscopic image generated based on the left-eye corresponding image and the right-eye corresponding image is displayed on a user terminal used by a viewer (user). As the user terminal, for example, a tablet terminal or an HMD (Head Mounted Display) can be considered. According to such a system, for example, a live image of an artist is shot with a left-eye compatible camera and a right-eye compatible camera, and a live image that is a stereoscopic image based on the obtained left-eye compatible image and right-eye compatible image is used. By displaying it on the person terminal, the user can view the live video of the artist while obtaining a stereoscopic effect.

  In such a system, the following processing may be performed so that the user can feel VR (Virtual Reality) more. First, each image captured by the left-eye compatible camera and the right-eye compatible camera is once projected onto a three-dimensional model defined in the world coordinate system. Accordingly, two stereoscopic image models corresponding to the left eye corresponding image and the right eye corresponding image are generated. In addition, in the world coordinate system, the left eye viewpoint for the stereoscopic image model corresponding to the left eye is set, and the right eye viewpoint for the stereoscopic image model corresponding to the right eye is set. Here, the line-of-sight direction from each viewpoint to each stereoscopic image model is set according to the orientation of the user terminal. Then, each stereoscopic image model is two-dimensionally projected based on each set viewpoint and line-of-sight direction, thereby obtaining a processed left-eye corresponding image and right-eye corresponding image. A stereoscopic image is generated based on the processed left eye corresponding image and the processed right eye corresponding image. In the present specification, a series of processes as described above is referred to as “geometry pipeline stereoscopic image generation process”.

  According to the geometry pipeline stereoscopic image generation process, the stereoscopic image displayed on the user terminal can be changed according to the orientation of the user terminal. For example, when the user terminal is facing the front, the front area of each stereoscopic image model (that is, the image area captured by the incident light from the front direction of the left-eye compatible camera and the right-eye compatible camera) is displayed. When the user turns the user terminal to the right side, the right region of each stereoscopic image model (that is, the image region captured by the incident light from the right direction of the left-eye compatible camera and the right-eye compatible camera) is displayed. Moreover, the user can view these images while feeling a three-dimensional effect. As a result, the user can feel as if he / she is at the place where the image was captured (for example, a live venue).

  As described above, the user can obtain a stereoscopic effect from the stereoscopic image because there is a parallax between the left-eye compatible camera and the right-eye compatible camera. However, in the geometry pipeline stereoscopic image generation process, when the user points the user terminal in the left-right direction and displays the left and right regions of each stereoscopic image model, the left-eye compatible camera, the right-eye compatible camera, There arises a problem that the parallax between the two becomes small. Thereby, in the left and right side regions, there arises a problem that the stereoscopic effect that the user can obtain decreases. The problem will be described with reference to FIG.

  FIG. 17 shows a left-eye correspondence lens and a right-eye correspondence lens that are installed at a predetermined distance d apart in the xy plane that is a horizontal plane. As described above, d is set based on the distance between the human left eye and right eye. As shown in FIG. 17, when viewing the front from each lens, the line-of-sight direction from each lens is the front direction (parallel to the x-axis). At this time, the parallax between the left-eye corresponding lens and the right-eye corresponding lens can be said to be an interval between both lenses, that is, d. On the other hand, the user terminal is directed in the left-right direction, and the left and right regions of each stereoscopic image model (this is also the left and right regions of the imaging regions of the left-eye camera and the right-eye camera) are displayed on the user terminal. Considering the case, even in that case, the direction of the left-eye corresponding lens and the right-eye corresponding lens is constant (front direction), so the line-of-sight direction from each lens has an angle with respect to the front direction. It will be. For example, as shown in FIG. 17, when viewing the right direction from each lens, the viewing direction from each lens is an angle (90-φ) from the front direction (x-axis direction) and an angle φ from the true lateral direction (y-axis direction). It becomes the direction with Accordingly, as shown in FIG. 17, the parallax between the left-eye corresponding lens and the right-eye corresponding lens becomes d ′ smaller than the distance d between both the lenses. Specifically, d ′ = dsinφ. Thus, when the user turns the user terminal to the left and right and looks at the left and right regions of each stereoscopic image model, the parallax between the left-eye corresponding lens and the right-eye corresponding lens becomes small.

  An object of the present invention is to support a left eye that occurs when a user terminal faces in the left-right direction from the front direction when displaying a stereoscopic image generated by the geometry pipeline stereoscopic image generation process on the user terminal. The object is to suppress a decrease in parallax between the camera and the right-eye camera.

The present invention processes images from a left-eye compatible camera and a right-eye compatible camera installed so as to face the same direction at a predetermined distance, and generates an image to be displayed on a user terminal used by the user. An image processing apparatus, wherein a left-eye corresponding image captured by the left-eye compatible camera and a right-eye corresponding image captured by the right-eye compatible camera are each projected and converted into a three-dimensional model in a world coordinate system Based on the modeling unit that builds the left-eye corresponding stereo model and the right-eye corresponding stereo model, and the orientation of the user terminal, in the world coordinate system, the position of the left-eye viewpoint with respect to the left-eye corresponding stereo model, the left eye Set the direction of the left eye gaze direction that is the viewpoint direction from the viewpoint and the direction of the upper left eye vector that indicates the upward direction of the left eye viewpoint, and based on the direction of the user terminal, the world coordinate system The right eye viewpoint position relative to the right eye corresponding stereo model, the right eye gaze direction that is the viewpoint direction from the right eye viewpoint, and the direction of the right eye upper vector indicating the upward direction of the right eye viewpoint are set. When the orientation of the viewpoint setting unit and the user terminal is directed from the front direction determined by calibration in the left-right direction, the direction opposite to the left eye gaze direction is set as the _ZVL axis positive direction, and the left the orientation of the eye upward vector and Y _VL-axis positive direction, the Z _VL-axis and Y _VL X _VL-axis negative direction in the left-eye view coordinate system is orientation X _VL-axis positive direction constituting a right-hand system relative axis The left eye viewpoint is moved, the direction opposite to the right eye line-of-sight direction is the Z _VR axis positive direction, the right eye upper vector direction is the Y _VR axis positive direction, and the Z _VR axis and the Y _VR axis are For the right hand system By causing orientation moves the right eye viewpoint the X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction, and the viewpoint correction unit for correcting the position of the left eye viewpoint and the right eye viewpoint, Based on the corrected position of the left eye viewpoint, the left eye gaze direction, and the direction of the left eye upper vector, a processed left eye correspondence image is generated by two-dimensionally projecting the left eye correspondence stereo model. Based on the corrected position of the right eye viewpoint, the right eye gaze direction, and the direction of the right eye upper vector, the processed right eye corresponding image is generated by two-dimensionally projecting the right eye corresponding stereo model An image processing apparatus comprising: a projection conversion unit configured to perform the projection conversion.

  Preferably, the viewpoint correction unit is a correction that is half of the difference between the predetermined distance and the parallax between the left-eye compatible camera and the right-eye compatible camera calculated according to the orientation of the user terminal. The positions of the left eye viewpoint and the right eye viewpoint are moved by a distance corresponding to the distance.

  Preferably, the modeling unit does not draw an image of the imaging target of the left-eye corresponding camera in a rendering area in which the image of the imaging target of the left-eye corresponding camera is drawn in the left-eye corresponding stereo model. A drawing in which a drawing image is drawn in the right-eye corresponding model in the right-eye corresponding stereo model by performing a process of gradually transmitting the drawing image toward the non-drawing area in the border area on the non-drawing area side. A process of gradually transmitting the drawn image toward the non-drawing area in a boundary area on the non-drawing area side where the image of the imaging target of the right-eye compatible camera is not drawn among the areas is characterized in that To do.

Further, the present invention processes images from a left-eye compatible camera and a right-eye compatible camera that are installed so as to face the same direction at a predetermined distance, and displays an image to be displayed on a user terminal used by the user. The computer for generating the left eye corresponding image captured by the left eye corresponding camera and the right eye corresponding image captured by the right eye corresponding camera is projected and converted into a three-dimensional model, respectively, and the left eye in the world coordinate system Based on the modeling unit for constructing the corresponding stereo model and the stereo model corresponding to the right eye, and the orientation of the user terminal, in the world coordinate system, the position of the left eye viewpoint relative to the left eye corresponding stereo model, from the left eye viewpoint Set the direction of the left eye line-of-sight direction, and the direction of the upper left eye vector indicating the upper direction of the left eye viewpoint, and based on the direction of the user terminal, the world coordinate system The right eye viewpoint position relative to the right eye corresponding stereo model, the right eye gaze direction that is the viewpoint direction from the right eye viewpoint, and the direction of the right eye upper vector indicating the upward direction of the right eye viewpoint are set. When the orientation of the viewpoint setting unit and the user terminal is directed from the front direction determined by calibration in the left-right direction, the direction opposite to the left eye gaze direction is set as the _ZVL axis positive direction, and the left the orientation of the eye upward vector and Y _VL-axis positive direction, the Z _VL-axis and Y _VL X _VL-axis negative direction in the left-eye view coordinate system is orientation X _VL-axis positive direction constituting a right-hand system relative axis The left eye viewpoint is moved, the direction opposite to the right eye line-of-sight direction is the Z _VR axis positive direction, the right eye upper vector direction is the Y _VR axis positive direction, and the Z _VR axis and the Y _VR axis are For the right hand system By causing orientation moves the right eye viewpoint the X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction, and the viewpoint correction unit for correcting the position of the left eye viewpoint and the right eye viewpoint, Based on the corrected position of the left eye viewpoint, the left eye gaze direction, and the direction of the left eye upper vector, a processed left eye correspondence image is generated by two-dimensionally projecting the left eye correspondence stereo model. Based on the corrected position of the right eye viewpoint, the right eye gaze direction, and the direction of the right eye upper vector, the processed right eye corresponding image is generated by two-dimensionally projecting the right eye corresponding stereo model An image processing program that functions as a projection conversion unit.

Further, the present invention processes images from a left-eye compatible camera and a right-eye compatible camera that are installed so as to face the same direction at a predetermined distance, and displays an image to be displayed on a user terminal used by the user. An image processing method to be generated, wherein a left-eye corresponding image captured by the left-eye compatible camera and a right-eye corresponding image captured by the right-eye compatible camera are each projected and converted into a three-dimensional model, and world coordinates A modeling step of constructing a left-eye corresponding stereo model and a right-eye corresponding stereo model in the system, and the position of the left-eye viewpoint with respect to the left-eye corresponding stereo model in the world coordinate system, based on the orientation of the user terminal, The direction of the left eye gaze direction that is the viewpoint direction from the left eye viewpoint and the direction of the upper left eye vector that indicates the upward direction of the left eye viewpoint are set, and the work direction is determined based on the orientation of the user terminal. In a coordinate system, the position of the right eye viewpoint relative to the right eye corresponding stereo model, the right eye gaze direction that is the viewpoint direction from the right eye viewpoint, and the direction of the right eye upper vector indicating the upward direction of the right eye viewpoint A viewpoint setting step, and when the orientation of the user terminal is directed from the front direction determined by calibration to the left-right direction, the direction opposite to the left-eye line-of-sight direction is defined as the _ZVL axis positive direction. the orientation of the left eye upward vector and Y _VL-axis positive direction, the Z _VL-axis and X _VL axis in the left-eye view coordinate system is orientation X _VL-axis positive direction constituting a right-handed with respect to Y _VL axis The left eye viewpoint is moved in the negative direction, the direction opposite to the right eye line-of-sight direction is the Z _VR axis positive direction, the right eye upper vector direction is the Y _VR axis positive direction, and the Z _VR axis and Y versus the _VR-axis Constituting the right-hand system Te orientation by moving the right eye viewpoint the X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction, it corrects the position of the left eye viewpoint and the right eye viewpoint The left-eye corresponding stereo model and two-dimensional projection of the left-eye corresponding stereo model based on the corrected left-eye viewpoint position, the left-eye gaze direction, and the left-eye upper vector direction. Generated an eye-corresponding image and processed by two-dimensionally projecting the right-eye corresponding stereo model based on the corrected position of the right-eye viewpoint, the right-eye gaze direction, and the direction of the right-eye upper vector A projection conversion step of generating a right- eye corresponding image.

  According to the present invention, when the stereoscopic image generated by the geometry pipeline stereoscopic image generation process is displayed on the user terminal, the left-eye response that occurs when the user terminal faces the left-right direction from the front direction is generated. Reduction in parallax between the camera and the right-eye camera can be suppressed.

1 is a schematic configuration diagram of a stereoscopic image providing system according to an embodiment. It is the structure schematic of a left eye corresponding camera and a right eye corresponding camera. It is a figure which shows the space | interval of the fisheye lens of a left-eye corresponding camera, and the fisheye lens of a right-eye corresponding camera. 1 is a schematic configuration diagram of an image distribution server. It is a structure schematic diagram of a user terminal. It is a figure which shows a mode that a stereoscopic vision image is displayed on a user terminal. It is a conceptual diagram which shows the example of a spherical solid model. It is a figure which shows the correspondence of the incident angle to a fisheye lens, and the position in a fisheye image. It is a conceptual diagram which shows the mode of the projection process of the image corresponding to the left eye to a spherical solid model. It is a conceptual diagram which shows the processing content with respect to the drawing area | region of the left-eye corresponding | compatible stereo model. It is a conceptual diagram which shows the left eye viewpoint set in the world coordinate system. It is a figure which shows a mode that a user terminal is pointed in the left-right direction. It is a figure which shows parallax correction distance (DELTA) d when a user terminal is faced in the left-right direction. It is a conceptual diagram which shows a mode that the position of a left eye viewpoint is correct | amended. It is a conceptual diagram which shows a mode that the position of a right eye viewpoint is correct | amended. It is a flowchart which shows the flow of a process of the user terminal which concerns on this embodiment. It is a figure which shows a mode that the parallax between both lenses reduces, when a user terminal is faced in the left-right direction.

  Hereinafter, embodiments of the present invention will be described.

  FIG. 1 shows a schematic configuration diagram of a stereoscopic image providing system 10 according to the present embodiment. The stereoscopic image providing system 10 includes a left-eye compatible camera 12L, a right-eye compatible camera 12R, an image distribution server 14, and a user terminal 16 as a user terminal. The left-eye compatible camera 12L, the right-eye compatible camera 12R, and the image distribution server 14, and the image distribution server 14 and the user terminal 16 are connected to be communicable with each other via a communication line 18. The communication line 18 is configured by, for example, the Internet, a local area network (LAN), or near field communication.

  In the stereoscopic image providing system 10, after the left-eye corresponding image and the right-eye corresponding image (including moving images) captured by the left-eye corresponding camera 12L and the right-eye compatible camera 12R are temporarily stored in the image distribution server 14, the user terminal In response to a request from 16, the image distribution server 14 transmits a left-eye corresponding image and a right-eye corresponding image to the user terminal 16. Then, a stereoscopic image based on the left-eye corresponding image and the right-eye corresponding image is displayed on the display unit of the user terminal 16. As a result, a user as a user can browse a stereoscopic image. Although details will be described later, the stereoscopic image provided to the user in the stereoscopic image providing system 10 is generated by the geometry pipeline stereoscopic image generation process. Note that the content (content) of the stereoscopic image provided by the stereoscopic image providing system 10 is, for example, an artist or idol live video, but is not limited thereto.

  FIG. 2A shows a schematic configuration diagram of the left-eye camera 12L.

  The left-eye compatible camera 12L includes a fisheye lens 20L, that is, the left-eye compatible camera 12L is a fisheye camera. The fish-eye lens 20L has a hemispherical convex lens, and the convex lens allows the imaging target to be captured with a spherical surface instead of a flat surface. Since a fisheye lens generally has a wide angle of view, the imaging region can be made relatively wide even when the orientation is fixed by one lens. The fisheye lens 20L in the present embodiment has an angle of view of 230 °. Various projection methods are known for the fisheye lens, but the fisheye lens 20L in the present embodiment employs an equisolid angle projection. However, the fish-eye lens 20L may employ another projection method (for example, orthographic projection, equidistant projection, or three-dimensional projection).

  The image sensor 22L is constituted by, for example, a CCD (Charge Coupled Device) image sensor. The image sensor 22L has a plurality of light receiving elements arranged two-dimensionally. The plurality of light receiving elements correspond to the respective pixels of the generated left eye corresponding image. The plurality of light receiving elements included in the imaging element 22L receive the light incident on the fisheye lens 20L and convert the received light into an electrical signal. The light receiving element that receives the light is determined according to the incident angle of the light to the fisheye lens 20L. In the present embodiment, each light receiving element of the image sensor 22L receives light and converts it into pixel values having R (red), G (green), and B (blue) values according to the light. To do. When the above-described processing is performed by the image sensor 22L, a left-eye corresponding image that is a two-dimensional image is formed.

  The communication unit 24L is composed of, for example, a network adapter. The communication unit 24L is a so-called communication interface and has a function of transmitting a left-eye compatible image to the image distribution server 14 via the communication line 18. The communication unit 24L may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  FIG. 2B shows a schematic configuration diagram of the right-eye compatible camera 12R. The right-eye compatible camera 12 </ b> R generates a right-eye compatible image and transmits it to the image distribution server 14. Since the configuration and function of each unit included in the right-eye compatible camera 12R are the same as those included in the left-eye compatible camera 12L, description thereof will be omitted.

FIG. 3 shows the positional relationship between the fisheye lens 20L included in the left-eye compatible camera 12L and the fisheye lens 20R included in the right-eye compatible camera 12R. FIG. 3 is a plan view, and the fish-eye lenses 20L and 20R are installed at a predetermined distance d apart on the X _R1 Y _R1 plane, which is the horizontal plane of the real space coordinate system. Specifically, the predetermined distance d is a distance between the center of the fisheye lens 20L and the center of the fisheye lens 20R. The predetermined distance d is determined based on the distance between the human left eye and right eye, and is set to 64 mm in this embodiment. The fisheye lenses 20L and 20R are installed so as to face the same direction. In the present embodiment, the fisheye lenses 20L and 20R are both oriented in the horizontal direction along the _XR1 axis. In this specification, a direction from the fish-eye lens 20L to X _R1 axial positive direction, and the direction from the fish-eye lens 20R to X _R1 axial positive direction is referred to as a "front direction". The _YR1 axis direction is referred to as “left-right direction”.

  As described above, since the fisheye lenses 20L and 20R have an angle of view of 230 °, the imaging region of the fisheye lens 20L is 230 ° in the left, right, top, bottom, and diagonal directions centered on the front direction of the fisheye lens 20L. The same applies to the fisheye lens 20R.

  FIG. 4 shows a schematic configuration diagram of the image distribution server 14. The image distribution server 14 may be any device as long as it is a computer that exhibits the functions described below.

  The communication unit 30 is composed of, for example, a network adapter. The communication unit 30 has a function of receiving a left-eye corresponding image from the left-eye compatible camera 12L and a right-eye corresponding image from the right-eye compatible camera 12R via the communication line 18. The communication unit 30 also has a function of receiving image distribution request information from the user terminal 16 via the communication line 18. Further, the communication unit 30 receives the left-eye corresponding image 34L and the right-eye corresponding image 34R received from the left-eye corresponding camera 12L and the right-eye corresponding camera 12R via the communication line 18 and stored in the storage unit 32. It has a function to transmit (distribute) to. At this time, it is preferable that the left-eye corresponding image 34L and the right-eye corresponding image 34R are simultaneously transmitted (distributed) as one image.

  The storage unit 32 includes a hard disk, a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The storage unit 32 stores a program for operating each unit of the image distribution server 14. Further, the storage unit 32 stores a left-eye correspondence image 34L and a right-eye correspondence image 34R received from the left-eye correspondence camera 12L and the right-eye correspondence camera 12R.

  The control unit 36 includes, for example, a CPU (Central Processing Unit) and the like, and controls the operation of each unit of the image distribution server 14 in accordance with a program stored in the storage unit 32. In addition, when receiving the distribution request information from the user terminal 16, the control unit 36 controls to transmit the left eye corresponding image 34L and the right eye corresponding image 34R related to the distribution request to the user terminal 16 that has transmitted the distribution request information. I do.

  FIG. 5 shows a schematic configuration diagram of the user terminal 16. The user terminal 16 according to the present embodiment is a tablet terminal such as a smartphone, but the user terminal 16 may be any device as long as it is a computer that exhibits the functions described below. For example, a head mounted display (HMD) may be used. In the user terminal 16, an application for viewing a stereoscopic image based on the left eye corresponding image 34L and the right eye corresponding image 34R distributed from the image distribution server 14 can be operated.

  In the present embodiment, the user terminal 16 processes the left eye corresponding image 34L and the right eye corresponding image 34R received from the image distribution server 14 to generate a stereoscopic image. That is, in this embodiment, the user terminal 16 functions as an image processing apparatus at the same time as a user terminal.

  The control unit 40 includes, for example, a CPU or a microcontroller, and controls the operation of each unit of the user terminal 16 according to a program stored in a storage unit 48 described later. In addition, when receiving an image distribution request transmission instruction from a user, the control unit 40 performs control to transmit distribution request information to the image distribution server 14.

  The communication unit 42 is composed of, for example, a network adapter. The communication unit 42 has a function of transmitting distribution request information to the image distribution server 14 via the communication line 18. In addition, the communication unit 42 has a function of receiving the left-eye corresponding image 34L and the right-eye corresponding image 34R from the image distribution server 14 via the communication line 18.

  The display unit 44 is composed of, for example, a liquid crystal panel. The display unit 44 displays a user interface screen for the user to perform various operations. The display unit 44 displays a stereoscopic image received from the image distribution server 14 and generated based on the left eye corresponding image 34L and the right eye corresponding image 34R. In the present embodiment, as shown in FIG. 6, the stereoscopic image is generated by arranging a processed left-eye corresponding image and a processed right-eye corresponding image processed by the image processing unit 52 described later in parallel. Is. The user attaches a tool such as a special lens for viewing the processed left-eye image with the left eye and the processed right-eye image with the right eye to the user terminal 16 and views the stereoscopic image. In this way, the user can view the stereoscopic image displayed on the display unit 44 while obtaining a stereoscopic effect.

  Returning to FIG. 5, the input unit 46 includes, for example, a touch panel or buttons. The input unit 46 is for inputting a user instruction to the user terminal 16.

  The storage unit 48 includes, for example, a ROM or a RAM. The storage unit 48 stores a program for operating each unit of the user terminal 16. In addition, the storage unit 48 stores an image processing program for exhibiting each function of the image processing unit 52 described later, and an application for viewing a stereoscopic image. In addition, the storage unit 48 stores various processing data.

The angular velocity sensor 50 includes, for example, a gyro sensor. The angular velocity sensor 50 measures the rotational speed (angular velocity) of the user terminal 16 around three axes in the real space coordinate system, that is, around the _XR2 axis, around the _YR2 axis, and around the _ZR2 axis. Further, the angular velocity sensor 50 calculates the angle around each axis of the user terminal 16, that is, the direction by integrating the measured angular velocity around each axis with the time from the calibration time point after performing calibration. be able to.

  The image processing unit 52 includes, for example, a GPU (Graphics Processing Unit). The image processing unit 52 performs a geometry pipeline stereoscopic image generation process in cooperation with the stereoscopic image generation program stored in the storage unit 48. As illustrated in FIG. 5, the image processing unit 52 includes functional blocks of a modeling unit 54, a viewpoint setting unit 56, a viewpoint correction unit 58, a projection conversion unit 60, and a stereoscopic image generation unit 62. Hereinafter, each functional block included in the image processing unit 52 will be described with reference to FIG.

  The modeling unit 54 projects and converts the left-eye corresponding image 34L received from the image distribution server 14 to the three-dimensional model (three-dimensional model) defined in the world coordinate system that is a virtual space coordinate system, and corresponds to the left eye. Build a 3D model. Similarly, the modeling unit 54 projects and converts the right-eye corresponding image 34R received from the image distribution server 14 on the three-dimensional model defined in the world coordinate system to construct a left-eye corresponding three-dimensional model.

In the present embodiment, a spherical solid model is used. FIG. 7 shows a spherical solid model 70 used in the present embodiment. In the present embodiment, the world coordinate system, X _W axis is orthogonal coordinate system with the origin as O _W, as defined in the Y _W-axis, and Z _W-axis. The scale of the world coordinate system may be set as appropriate, but in the present embodiment, the same scale as the real space coordinate system is set. Spherical three-dimensional model 70 is centered at the origin O _W, the surface thereof is formed of a plurality of triangular polygons 72. Specifically, orthogonal to each longitude lines and a plurality of longitudinal lines arranged in the longitudinal direction so as to connect the two poles of the spherical three-dimensional model 70 (maximum and minimum values of the Y _W coordinate), the surface of the sphere, latitudinal A plurality of latitude lines are defined, and intersection points between the plurality of longitude lines and the plurality of latitude lines are defined as vertices. Information indicating a plurality of vertex coordinates for a plurality of vertices is stored in advance in the storage unit 48 or the like, and can be referred to by the modeling unit 54. One triangular polygon 72 is defined by three adjacent vertices among a plurality of vertices defined on the spherical solid model 70 (surface thereof).

  The modeling unit 54 projects and converts the left-eye corresponding image 34 </ b> L into a spherical solid model 70, and performs a drawing process on each triangular polygon 72 included in the spherical solid model 70. Thereby, a spherical model corresponding to the left eye is constructed. Similarly, the modeling unit 54 projects and converts the right-eye corresponding image 34 </ b> R into the spherical solid model 70 and performs drawing processing on each triangular polygon 72 included in the spherical solid model 70. Thereby, a spherical right-eye corresponding three-dimensional model is constructed.

  Before describing the projection conversion by the modeling unit 54, the left-eye corresponding image 34L and the right-eye corresponding image 34R that are fish-eye images captured by the left-eye corresponding camera 12L and the right-eye corresponding camera 12R that are fish-eye cameras. explain.

FIG. 8 is a diagram showing a correspondence relationship between the incident angle to the fisheye lens and the position in the fisheye image. FIG. 8 is a model diagram, and the angle of view of the fisheye lens shown in FIG. 8 is 180 °. FIG. 8 shows an example in the case of equisolid angle projection. In FIG. 8, the fisheye lens viewed from the front and the fisheye image are superimposed on the left side, and the fisheye lens viewed from the side on the right side. The coordinates of each pixel of the fish-eye image shown in FIG. 8, the upper left corner of the fish-eye image as the origin O _P, and X _P axis coordinates indicating the position of the horizontal (lateral) direction, vertical (longitudinal) It is assumed to be expressed by _YP axis coordinates indicating the position in the direction.

In the equal solid angle projection, the incident angle θ to the fisheye lens and the image height y in the image sensor are given by the following equations.
y = 2fsin (θ / 2) (Formula 1)
Here, the incident angle θ is an angle of incident light with respect to the front direction of the fisheye lens, the image height y is a physical distance from the center of the image sensor, and f is a focal length.

For example, when the incident angle θ = 0 °, that is, incident light from the front direction of the fisheye lens, y = 0. This means that the light receiving element located at the center of the two-dimensionally arranged light receiving elements of the image pickup element receives incident light having an incident angle of 0 °. That is, the pixel in the corresponding fisheye image on the incident light angle of incidence 0 ° (from the front of the fish-eye lens), the center coordinates of the fish-eye image, i.e., the maximum _{X P} coordinates _{X PMAX,} the maximum _{Y P} coordinate _{Y PMAX} If it is, it becomes a pixel of coordinates (X _PMAX / 2, Y _PMAX / 2).

Next, when the incident angle θ is not 0 °, the image height y has a certain value. For example, when the incident angle θ = 30 °, the image height y = f. This indicates that, with respect to incident light having an incident angle of 30 °, the light receiving element positioned on the circumference of the radius y = f centering on the center of the imaging element receives light. Pixels in the fisheye image corresponding to incident light with an incident angle of 30 ° are pixels located on the circumference of a radius y ′ centered on the center coordinates of the fisheye image. Here, since y is a physical distance on the image sensor, the radius y ′ in the fisheye image is converted to y based on the size of the fisheye image (X _PMAX and Y _PMAX ). It is acquired over time.

In the present embodiment, since the projection method of the fisheye lenses 20L and 20R is an equisolid angle projection, the image height y is calculated by the equation 1.
When the projection method of the fisheye lenses 20L and 20R is an orthographic projection,
y = fsinθ (Expression 2)
When the projection method of the fisheye lenses 20L and 20R is equidistant projection,
y = fθ (Expression 3)
When the projection method of the fisheye lenses 20L and 20R is a three-dimensional projection,
y = 2 f tan (θ / 2) (4)
The image height y is determined for each.

  As described above, in the fish-eye image, an image is formed on the pixels on the concentric circumference according to the incident angle to the fish-eye lens, so that no image is formed at the four corners of the fish-eye image as shown in FIG. It is common. Hereinafter, in the fish-eye image, the four corner portions where the image is not formed are referred to as “non-image region”, and the other portions are referred to as “image region”.

FIG. 9 shows a state in which the modeling unit 54 projects and converts the left-eye corresponding image 34 </ b> L onto the spherical three-dimensional model 70. As shown in FIG. 9, conceptually, the modeling unit 54, the X _W Y _W plane parallel to the plane of the negative side in the Z _W axis than the spherical three-dimensional model 70, the upper Y _W-axis positive direction The left eye corresponding image 34L is arranged so that the spherical solid model 70 is wrapped in the left eye corresponding image 34L, and each pixel of the left eye corresponding image 34L and each triangular polygon 72 of the spherical solid model 70 are arranged. Then, each pixel of the left-eye corresponding image 34L is projected onto the corresponding triangular polygon 72. Thereby, a drawing process is performed on each triangular polygon 72 of the spherical solid model 70.

Specifically, modeling unit 54 includes a line connecting the O _W (center of the spherical three-dimensional model 70) triangles 72 and the world coordinate system of the origin of the drawing object, extending from the origin O _W to Z _W-axis negative direction based on the angle line forms (hereinafter referred to as "counter Z _W-axis angle") to select pixels of the left eye corresponding image 34L to be projected to the triangles 72 of the drawing target. Specifically, for a triangular polygon group whose Z _W axis angle is θ ₁ , among the pixels of the left-eye corresponding image 34L, a pixel group corresponding to the same incident angle θ ₁ as the corresponding Z _W axis angle. (Indicated by a one-dot chain line on the left-eye corresponding image 34L in FIG. 9) is projected. As described above, since the distance y ′ from the center coordinates of the left-eye corresponding image 34 is calculated based on the incident angle, the distance y ′ corresponds to the incident angle θ ₁ on the left-eye corresponding image 34L. A pixel group to be identified can be specified. When there are a plurality of pixels of the left-eye corresponding image 34L corresponding to one triangular polygon 72, a drawing process may be performed on the triangular polygon 72 based on the plurality of pixels. In that case, for example, drawing may be performed using representative values (for example, average values) of a plurality of pixel values of a plurality of pixels.

  In this way, the left-eye corresponding image 34L is projected onto the spherical three-dimensional model 70, whereby the left-eye corresponding image 34L that is a two-dimensional image is converted into a left-eye corresponding three-dimensional model.

  Similarly, the right-eye corresponding model 34R is constructed by projecting and converting the right-eye corresponding image 34R to the spherical three-dimensional model 70 for the right-eye corresponding image 34R.

Figure 10 is a cross-sectional view is shown in X _W Z _W plane of the left eye corresponding stereo model 74L which is constructed by projection processing in the modeling unit 54. As described above, the angle of view of the fish-eye lens 20L in the present embodiment is 230 °, and there are no image areas at the four corners of the fish-eye image. Therefore, as shown in FIG. triangular polygons groups located on the side (Z _W-axis positive direction) after 74L is a region left eye corresponding image 34L is not projected, and comprises a region free image area of the left eye corresponding image 34L is projected That is, it is a non-drawing area (that is, the image of the imaging target of the left eye corresponding camera 12L is not drawn). Specifically, triangular polygons unit located rearward of the central angle 230 ° with the middle negative direction of the Z _W axis represents the non-rendering area. In the present embodiment, the triangular polygon group included in the non-drawing area is black. On the other hand, the triangular polygon groups located in front of the central angle 230 ° with the middle negative direction of the Z _W-axis (Z _W-axis negative direction side), the image area of the left eye corresponding image 34L, i.e., the left eye corresponding camera This is a drawing area in which a 12L image to be imaged is projected and drawn.

Here, the modeling unit 54, out of the drawing area, to delete the drawing image drawn on a triangular polygon unit located rearward of the central angle 180 ° with the middle negative direction of the Z _W-axis. An area where the drawn image is deleted by the deletion process is referred to as a “drawn image deletion area” (see FIG. 10). This is because, as shown in FIG. 3, the fisheye lens 20L of the left-eye compatible camera 12L and the fisheye lens 20R of the right-eye compatible camera 12R are arranged side by side in the left-right direction. In the drawn image deletion area on the right side ( _XW axis positive direction side), the image of the fisheye lens 20R is drawn. Similarly, the drawing image deletion area to the right eye corresponding stereo model 74R of the left (X _W-axis negative direction), the image of the fish-eye lens 20L from being drawn. In the present embodiment, when a stereoscopic image is generated by two-dimensionally projecting a region including the drawing image deletion region (the details of the two-dimensional projection process will be described later), the fisheye lenses 20L and 20R can be seen by the user. The modeling unit 54 deletes the drawn image in the drawn image deletion area so as not to be stuttered. By drawing image is deleted in the drawing image deletion area, the front becomes drawing area than the central angle 180 ° with the middle negative direction of the Z _W-axis, the center angle 180 ° with the middle negative direction of the Z _W-axis The rear side is a non-drawing area.

  Further, the modeling unit 54 gradually transmits the drawn image toward the non-drawing area in the boundary area (the left and right ends of the drawing area) with the non-drawing area in the drawing area of the left-eye corresponding stereo model 74L. Perform transparency processing. An area where the transparent process is performed is referred to as a “transparent process area” (see FIG. 10). As described above, nothing is drawn in the non-drawing area, and in this embodiment, it is black. Therefore, when a stereoscopic image is generated by two-dimensionally projecting a region including the boundary between the drawing region and the non-drawing region, pixels corresponding to the non-drawing region suddenly appear in the stereoscopic image. End up. For example, a boundary line appears on a stereoscopic image, and there is an image corresponding to the captured image on one side of the boundary line, but a region suddenly filled with black pixels appears on the boundary line. Become. As a result, the user may feel uncomfortable.

  Therefore, even when the modeling unit 54 performs the transparency process on the transparency process area, the stereoscopic image is generated by two-dimensionally projecting the area including the boundary between the drawing area and the non-drawing area. In the stereoscopic image, since the pixels gradually pass from the pixel area corresponding to the drawing area to the pixel area corresponding to the non-drawing area, it is possible to reduce discomfort felt by the user.

  The modeling unit 54 also performs the drawing image deletion process for the drawing image deletion area and the transmission process for the transmission processing area for the right-eye corresponding stereo model 74R.

  In the world coordinate system, the viewpoint setting unit 56 performs settings relating to the left eye viewpoint for the left eye corresponding stereo model 74L constructed by the modeling unit 54 and settings relating to the right eye viewpoint for the right eye corresponding stereo model 74R constructed by the modeling unit 54. . Specifically, the position of the left eye viewpoint and the position of the right eye viewpoint, the left eye gaze direction that is the gaze direction from the left eye viewpoint, the right eye gaze direction that is the gaze direction from the right eye viewpoint, and the left eye viewpoint The direction of the upper left eye vector indicating the upward direction and the direction of the upper right eye vector indicating the upward direction of the right eye viewpoint are set.

The viewpoint setting unit 56 determines the positions of the left eye viewpoint and the right eye viewpoint, the left eye gaze direction and the right eye gaze direction, the direction of the left eye upper vector, and the direction of the right eye upper vector according to the orientation of the user terminal 16. decide. FIG. 11 shows a state in which the orientation of the user terminal 16 is changed to the right on the X _R2 Y _R2 plane that is a horizontal plane. In the present embodiment, the angular velocity sensor 50 performs calibration at the timing when an application for viewing a stereoscopic image is activated on the user terminal 16. The orientation of the user terminal 16 when the calibration is performed is set as the front direction. In FIG. 11, the _XR2 axis positive direction is the front direction in the horizontal direction, and the user terminal 16 facing the front direction is indicated by a broken line. 11 after calibrating, is changed the direction of the user terminal 16 by the user, an angle with respect to the front direction in the clockwise (right direction) (90-phi) °, namely Y _R axis direction (just beside direction) The state of being rotated in the direction of φ is shown. Of course, the orientation of the user terminal 16 is changed while the state where the display unit 44 faces the user is maintained.

FIG. 12 shows the left eye viewpoint set in the world coordinate system. 12, FIG. 10 Similarly, a cross-sectional view in _X W _{Z W} plane of the left eye corresponding stereo model 74L. In the present embodiment, the left eye viewpoint the origin O _W of the world coordinate _system, i.e., is set from the center point of the spherical left eye corresponding stereo model 74L to a position at a predetermined distance. As will be described later, the position of the left eye viewpoint can be changed according to the orientation of the user terminal 16, but the distance from the origin _OW to the left eye viewpoint remains unchanged. Therefore, the movement locus of the left eye viewpoint is on a spherical surface with the origin _OW as the center.

Left eye gaze direction from the left eye viewpoint is a direction from the position of the left eye viewpoint to the origin O _W. Therefore, when the position of the left eye viewpoint is determined, it can be said that the left eye gaze direction is determined accordingly. Further, in the present embodiment, for the left eye corresponding stereo model 74L, on which to match the upper direction of the Y _W-axis positive direction and the left-eye corresponding image 34L, the left eye corresponding image 34L spherical three-dimensional model 70 from being projected, if the left eye viewpoint is on X _W Z _W plane, the orientation of the upper vector of the left eye viewpoint becomes Y _W-axis positive direction. Note that the directional relationship between the left-eye viewing direction and the direction of the upper vector of the left-eye viewpoint is maintained constant (orthogonal) regardless of the position of the left-eye viewpoint.

In the left eye corresponding stereo model 74L built in the world coordinate system, since the Z _W-axis negative direction corresponds to the front direction, the viewpoint setting unit 56, the left-eye viewing direction for Z _W-axis negative direction in the world coordinate system The position of the left eye viewpoint is set so as to match the orientation of the user terminal 16 with respect to the front direction.

First, when the user terminal 16 is facing the front direction, the viewpoint setting unit 56, as the left-eye viewing direction is directed in the negative direction Z _W-axis, the left eye from the origin O _W to Z _W-axis positive direction position Set the viewpoint. In FIG. 12, the position is indicated by a point VPL _ini . Orientation of the left eye upper vector in this case is the Y _W-axis positive direction.

After that, as shown in FIG. 11, when the user terminal 16 is rotated (90−φ) ° from the front direction to the right direction on the X _R2 Y _R2 plane, the viewpoint setting unit 56 sets the left eye viewpoint to is moved on a moving path, a left eye viewing direction in the world coordinate system, the position of the left eye viewpoint as an angle formed between the line extending from the origin O _W to Z _W-axis negative direction is (90-φ) ° Set. In FIG. 12, the position of the left eye viewpoint set in this way is indicated by a point VPL ₁ . Incidentally, in this example, the left eye viewpoint, since only moves over X _W Z _W plane, the direction of the left eye upward vector is maintained in the Y _W-axis positive direction.

When the position of the left eye viewpoint is set by the viewpoint setting unit 56, a left eye view coordinate system with the left eye viewpoint as the origin is defined. In the left-eye view coordinate system in the present embodiment, the direction opposite to the left-eye line-of-sight direction is the _ZVL axis positive direction (that is, the left-eye line-of-sight direction is the _ZVL axis negative direction), and the direction of the left-eye upper vector is Y _VL The direction that constitutes the right-handed system with respect to the Z _VL axis and the Y _VL axis is set to the X _VL axis positive direction. Therefore, when the position of the left eye viewpoint is changed, the position of the origin of the left eye view coordinate system with respect to the world coordinate system and the directions of the X _VL axis Y _VL axis Z _VL axis are also changed.

  If the position of the left eye viewpoint is not corrected by the viewpoint correction unit 58 described later, the left eye corresponding stereo model 74L included in a predetermined viewing angle centered on the left eye gaze direction is described by the projection conversion unit 60 described later. , That is, a portion represented by a thick line on the left-eye corresponding stereo model 74L in FIG. 12 is two-dimensionally projected to generate a two-dimensional image corresponding to the left eye. That is, the two-dimensional image corresponding to the thick line portion in FIG. 12 is a portion that is displayed on the display unit 44 (visible to the user).

The right eye viewpoint for the right eye corresponding stereo model 74R is also set by the same processing as described above. Then, a right eye view coordinate system with the right eye viewpoint as the origin is defined. In the right-eye view coordinate system in the present embodiment, the direction opposite to the right-eye line-of-sight direction is the Z _VR axis positive direction (that is, the right-eye line-of-sight direction is the Z _VR axis negative direction), and the direction of the right eye upper vector is Y _VR. The direction that constitutes the right-hand system with respect to the Z _VR axis and the Y _VR axis is set as the positive direction of the X _VR axis.

  The viewpoint correction unit 58 moves the set position of the left-eye viewpoint on the left-eye view coordinate system based on the orientation of the user terminal 16 when the user terminal 16 is directed from the front direction to the left-right direction, A process of correcting the positions of the left eye viewpoint and the right eye viewpoint is performed by moving the set position of the right eye viewpoint on the right eye view coordinate system. Specifically, the positions of the left eye viewpoint and the right eye viewpoint are moved in a direction in which the parallax between the left eye corresponding camera 12L and the right eye corresponding camera 12R is expanded. Hereinafter, the details of the correction process will be described.

Similarly to the case shown in FIG. 17, also in this embodiment, when the user points the user terminal 16 in the left-right direction, the parallax between the left-eye compatible camera 12 </ b> L and the right-eye compatible camera 12 </ b> R becomes small. Occurs. The problem is shown again in FIG. Here again, consider the case where the user turns the user terminal 16 from the front direction to the right side (90-φ) °, that is, the direction from the right side direction to the angle φ. In this case, as described above, the two-dimensional image generated by the projective transformation unit 60 is composed of a Z _W-axis negative direction of the left eye corresponding stereo model 74L (front direction in the world coordinate system) and the image corresponding to the right . On the other hand, since the orientation of the fisheye lenses 20L and 20R remains facing the front direction, when viewed in real space, the line-of-sight direction from the fisheye lenses 20L and 20R is (90-φ) from the lens orientation (front direction) to the right side. ) It is in a state of facing. Therefore, the parallax between the left-eye compatible camera 12L and the right-eye compatible camera 12R is d ′, which is d ′ = dsinφ as described above. Accordingly, the parallax d ′ is the distance between the fish-eye lenses 20L and 20R, that is, the parallax d between the left-eye corresponding camera 12L and the right-eye corresponding camera 12R when the line-of-sight direction is the front direction from the fish-eye lenses 20L and 20R. Smaller than.

When the user terminal 16 is directed in the lateral direction, to suppress the reduction of the disparity between the left eye corresponding camera 12L and the right-eye corresponding camera 12R, the current of the current viewpoint LPL ₁ and fisheye lens 20R of the fish-eye lens 20L Each viewpoint LPR ₁ may be moved in the direction in which the parallax spreads. If the parallax is accurately maintained at d, the difference between the parallax d when the line-of-sight direction is the front and the current parallax d ′ is calculated, and the viewpoint of the fisheye lens 20L and the viewpoint of the fisheye lens 20R are half of the difference Δd It is only necessary to move in the direction in which the parallax spreads.

Here, since d ′ = dsinφ, the correction distance Δd is
Δd = {d (1-sinφ)} / 2 (Formula 5)
Given in.

Therefore, if, in a direction perpendicular to the viewpoint of the fisheye lens 20L to the viewing direction is moved by a distance Δd in the spreading direction parallax, and the position shown the viewpoint at the point LPL ₂ in FIG. 13, similarly, the fish-eye lens 20R Is moved in the direction orthogonal to the line-of-sight direction and in the direction in which the parallax spreads, and the viewpoint is moved to the position indicated by the point LPR ₂ in FIG. It is considered that the reduction in parallax that can occur when it is applied can be eliminated.

  In the present embodiment, the viewpoint correction unit 58 determines the positions of the left eye viewpoint and the right eye viewpoint set in the world coordinate system so that the movement of the viewpoint in the real space is realized in the world coordinate system. Move.

FIG. 14 shows how the position of the left eye viewpoint is corrected. FIG. 14 is a cross-sectional view of the left-eye corresponding stereo model 74L on the X _W Z _W plane, as in FIG. Consider the state in which the left eye viewpoint is set to the position of the point VPL ₁ by the viewpoint setting unit 56. The movement direction of the left eye viewpoint for expanding the parallax between the left eye corresponding camera 12L and the right eye corresponding camera 12R is directed from the left eye viewpoint toward the left eye gaze direction with the direction of the upper left eye vector facing upward. It is the left direction when it was. The direction opposite to the left-eye line-of-sight direction is the Z _VL axis positive direction, the direction of the left eye upper vector is the Y _VL axis positive direction, and the direction that forms the right-handed system with respect to the Z _VL axis and the Y _VL axis There can be said to be X _VL-axis negative direction in the left-eye view coordinate system is a X _VL-axis positive direction. In other words, when the left-eye viewpoint direction is the Z-axis positive direction and the left-eye upper vector direction is the Y-axis positive direction, the X-axis positive direction that forms the right-handed system with respect to the Z-axis and the Y-axis It can be said that it is a direction. Further, when moving the position of the left eye viewpoint, the left eye gaze direction and the direction of the left eye upper vector are maintained.

While maintaining the orientation of the left eye view direction and the left eye upward vector, to move the left eye viewpoint X _VL-axis negative direction in the left-eye view coordinate system shown in FIG. 13, the fish-eye lens 20L in the real coordinate system This corresponds to moving the viewpoint from the point LPL ₁ to the point LPL ₂ while maintaining the line-of-sight direction and the upward direction of the fish-eye lens 20L. Therefore, the parallax reduction between the left eye corresponding camera 12L and the right eye corresponding camera 12R can be suppressed by moving the left eye viewpoint in the negative _XVL axis direction.

FIG. 15 shows how the position of the right eye viewpoint is corrected. 15 shows a cross-sectional view of the right eye corresponding stereo model 74R of _X W _{Z W} plane. The movement direction of the right eye viewpoint for expanding the parallax between the left eye corresponding camera 12L and the right eye corresponding camera 12R is directed from the right eye viewpoint to the right eye gaze direction with the direction of the upper right eye vector facing upward. It is the right direction when it was. The direction opposite to the right-eye line-of-sight direction is the Z _VR axis positive direction, the direction of the right eye upper vector is the Y _VR axis positive direction, and the direction constituting the right hand system with respect to the Z _VR axis and the Y _VR axis There can be said to be X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction. In other words, when the right-eye viewpoint direction is the Z-axis positive direction and the direction of the right-eye upper vector is the Y-axis positive direction, the X-axis positive direction that forms the left-handed system with respect to the Z-axis and the Y-axis It can be said that it is a direction. Also, when moving the position of the right eye viewpoint, the right eye gaze direction and the direction of the right eye upper vector are maintained.

While maintaining the orientation of the right eye view direction and the right eye upward vector, to move the right-eye viewpoint X _VR-axis positive direction in the right-eye view coordinate system are shown in FIG. 13, the fish-eye lens 20R in the real coordinate system This corresponds to moving the viewpoint from the point LPR ₁ to the point LPR ₂ while maintaining the viewing direction and the upward direction of the fisheye lens 20R. Therefore, the parallax reduction between the left-eye compatible camera 12L and the right-eye compatible camera 12R can be suppressed by moving the right-eye viewpoint in the _XVR axis positive direction.

  Further, it is preferable that the parallax between the left-eye compatible camera 12L and the right-eye compatible camera 12R matches d by correcting the left-eye viewpoint and the right-eye viewpoint. In order to make the parallax coincide with d, the viewpoint correction unit 58 calculates a correction distance between the left eye viewpoint and the right eye viewpoint.

Here, as shown in FIG. 11, the user terminal 16 is tilted to the right (90-φ) ° from the front direction, and the position of the left eye viewpoint set at the position indicated by the point VPL _{1 in} FIG. 14 is corrected. Think about the case. The viewpoint correction unit 58 uses the distance d between the fisheye lens 20L and the fisheye lens 20R and the angle φ indicating the orientation of the user terminal 16 calculated by the angular velocity sensor 50 to calculate the correction distance Δd based on the above equation 5. calculate. The distance d may be measured in advance, and the upper part indicating the distance d may be stored in the storage unit 48.

  The viewpoint correction unit 58 moves the correction distance Δd calculated from the left eye viewpoint in the above-described direction. Similarly, the right eye viewpoint is moved in the above-described direction by the correction distance Δd. As a result, regardless of the orientation of the user terminal 16, the parallax between the fisheye lens 20L and the fisheye lens 20R can always be d. In the present embodiment, since the scales of the world coordinate system and the real space coordinate system are the same, the left eye viewpoint and the right eye viewpoint are moved by the correction distance Δd. When the scales of the systems are different, the left eye viewpoint and the right eye viewpoint may be moved by the corrected distance after applying the correction considering the scale of both coordinate systems to the calculated correction distance Δd.

The projection conversion unit 60 is set by the viewpoint setting unit 56 and is corrected by the viewpoint correction unit 58 as necessary, based on the left-eye viewpoint, the left-eye line-of-sight direction, and the left-eye upper vector orientation. By projecting the model 74L two-dimensionally, a processed left-eye corresponding image that is a two-dimensional image is generated. As shown in FIG. 14, it is a part of the left-eye corresponding three-dimensional model 74L included within a predetermined viewing angle range centering on the left-eye line-of-sight direction from the point VPL ₂ that is the position of the corrected left-eye viewpoint. The projection target area 80L is a target for two-dimensional projection. Specifically, the rendered image of the triangular polygon group included in the projection target area 80L is projected on the two-dimensional screen defined in the left-eye view coordinate system with the corrected left-eye viewpoint as a reference, and the processed left An eye corresponding image is generated. In the present embodiment, perspective projection conversion is used as a two-dimensional projection method, but other conversion methods may be used.

Similarly, the projection conversion unit 60 is set based on the right eye viewpoint, right eye gaze direction, and right eye upper vector direction set by the viewpoint setting unit 56 and corrected by the viewpoint correction unit 58 as necessary. By projecting the eye corresponding stereo model 74R two-dimensionally, a processed right eye corresponding image that is a two-dimensional image is generated. As shown in FIG. 15, it is a part of the right-eye corresponding stereo model 74R included in the range of a predetermined viewing angle centered on the right-eye line-of-sight direction from the point VPR ₂ that is the position of the corrected right-eye viewpoint. The projection target area 80R is a target for two-dimensional projection. The two-dimensional projection method may be the same as the projection method of the left-eye corresponding stereo model 74L.

  The stereoscopic image generation unit 62 generates a stereoscopic image based on the processed left eye corresponding image and the processed right eye corresponding image generated by the projection conversion unit 60. As shown in FIG. 6, the stereoscopic image in the present embodiment is an image in which a processed left-eye corresponding image and a processed right-eye corresponding image are arranged in parallel. Of course, the stereoscopic image may be in any other form as long as the processed left-eye image can be viewed with the user's left eye and the processed right-eye image can be viewed with the user's right eye. Good.

  In addition, the stereoscopic image generation unit 62 performs viewport conversion that changes the size of the generated stereoscopic image in accordance with the size of the display unit 44. Thereby, a stereoscopic image suitable for the size of the display unit 44 is generated.

  The generated stereoscopic image is displayed on the display unit 44 by the control unit 40.

  The configuration of the stereoscopic image providing system 10 according to the present embodiment is as described above. According to the stereoscopic image providing system 10, when the stereoscopic image is generated by the geometry pipeline stereoscopic image generation process, when the user terminal 16 is directed from the front direction to the left-right direction, the viewpoint correction unit 58 The positions of the left eye viewpoint and right eye viewpoint set in the world coordinate system are moved in a direction in which the parallax between the left eye corresponding camera 12 </ b> L and the right eye corresponding camera 12 </ b> R is expanded according to the orientation of the user terminal 16. Thereby, the reduction | decrease in the parallax between the left-eye corresponding | compatible camera 12L and the right-eye corresponding | compatible camera 12R which arises when the user terminal 16 is turned to the left-right direction is suppressed. As a result, even when the user terminal 16 is directed in the left-right direction, it is possible to provide a stereoscopic image that allows the user to feel a stereoscopic effect.

  Preferably, the left-eye viewpoint and the right-eye viewpoint are moved by a distance corresponding to the correction distance Δd calculated according to the orientation of the user terminal. Therefore, no matter which direction the user terminal 16 faces, The parallax between the eye corresponding camera 12L and the right eye corresponding camera 12R is maintained at d. As a result, it is possible to provide a stereoscopic image in which a stereoscopic effect can be more suitably felt, and when the stereoscopic image displayed on the display unit 44 changes momentarily according to the orientation of the user terminal 16. , The uncomfortable feeling given to the user can be reduced.

  In the above-described embodiment, the user terminal 16 has the image processing unit 52, that is, the modeling unit 54, the viewpoint setting unit 56, the viewpoint correction unit 58, the projection conversion unit 60, and the stereoscopic image generation unit 62. These components can also take the form of the image distribution server 14. In that case, the image distribution server 14 functions as a stereoscopic image generation device. When a stereoscopic image is generated in the image distribution server 14, the generated stereoscopic image is distributed to the user terminal 16. When this aspect is adopted, the measurement value of the angular velocity sensor 50 of the user terminal 16 is sequentially transmitted to the image distribution server 14.

  Hereinafter, the process flow of the user terminal 16 according to the present embodiment will be described with reference to the flowchart shown in FIG.

  In step S <b> 10, image distribution request information is transmitted from the user terminal 16 to the image distribution server 14 in accordance with a user instruction. The user terminal 16 receives the left-eye corresponding image 34L and the right-eye corresponding image 34R from the image distribution server 14 that has received it.

  In step S12, the modeling unit 54 projects and converts the received left-eye corresponding image 34L into a spherical three-dimensional model 70 defined in the world coordinate system, thereby constructing a left-eye corresponding three-dimensional model 74L. Similarly, the modeling unit 54 projects and converts the received right-eye corresponding image 34R into a spherical three-dimensional model 70 defined in the world coordinate system to construct a right-eye corresponding three-dimensional model 74R.

  In step S <b> 14, the angular velocity sensor 50 acquires the orientation of the user terminal 16.

  In step S16, the viewpoint setting unit 56, based on the orientation of the user terminal 16 acquired in step S14, in the world coordinate system, the left eye viewpoint, the left eye gaze direction, and the left eye upper direction with respect to the left eye corresponding stereo model 74L. Set the orientation of the vector. Similarly, the viewpoint setting unit 56, based on the orientation of the user terminal 16 acquired in step S14, in the world coordinate system, the right-eye viewpoint, right-eye gaze direction, and right-eye upward vector for the right-eye corresponding stereo model 74R. Set the orientation.

  In step S18, the viewpoint correction unit 58 calculates the correction distance Δd based on the orientation of the user terminal 16 acquired in step S14.

In step S20, the viewpoint correction unit 58 moves the left eye viewpoint and the right eye viewpoint on the respective view coordinate systems by a distance corresponding to the correction distance Δd calculated in step S18. As described above, the moving direction of the left-eye viewpoint is X _VL-axis negative direction of the left eye view coordinate system, the moving direction of the right eye viewpoint is X _VR-axis positive direction of the right-eye view coordinate system.

  In step S22, the projection conversion unit 60 performs two-dimensional projection of the left-eye corresponding stereo model 74L based on the position of the left eye viewpoint, the left eye gaze direction, and the direction of the left eye upper vector corrected in step S20. Thus, a processed left eye corresponding image is generated. Similarly, the projection conversion unit 60 performs two-dimensional projection of the right-eye corresponding stereo model 74R based on the right-eye viewpoint position, the right-eye gaze direction, and the right-eye upper vector direction corrected in step S20. Then, a processed right eye corresponding image is generated.

  In step S24, the stereoscopic image generation unit 62 generates a stereoscopic image by arranging the processed left eye corresponding image and the processed right eye corresponding image generated in step S22 in parallel.

  In step S26, the control unit 40 causes the display unit 44 to display the stereoscopic image generated in step S24.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is not limited to the said embodiment, A various change is possible unless it deviates from the meaning of this invention.

  10 stereoscopic image providing system, 12L left-eye camera, 12R right-eye camera, 14 image distribution server, 16 user terminal, 18 communication line, 20L, 20R fisheye lens, 22L, 22R image sensor, 24L, 24R, 30, 42 Communication unit, 32, 48 storage unit, 34L left eye correspondence image, 34R right eye correspondence image, 36, 40 control unit, 44 display unit, 46 input unit, 50 angular velocity sensor, 52 image processing unit, 54 modeling unit, 56 viewpoints Setting unit, 58 viewpoint correction unit, 60 projection conversion unit, 62 stereoscopic image generation unit.

Claims (5)

An image processing apparatus that processes images from a left-eye compatible camera and a right-eye compatible camera installed so as to face the same direction at a predetermined distance and generate an image to be displayed on a user terminal used by a user There,
The left-eye corresponding image captured by the left-eye compatible camera and the right-eye corresponding image captured by the right-eye compatible camera are each projected and converted into a three-dimensional model, and the left-eye corresponding three-dimensional model and right A modeling unit that builds an eye-compatible stereo model;
Based on the orientation of the user terminal, in the world coordinate system, the position of the left eye viewpoint with respect to the left eye corresponding stereo model, the left eye gaze direction that is the viewpoint direction from the left eye viewpoint, and the left eye viewpoint The direction of the upper left eye vector indicating the upward direction is set, and based on the orientation of the user terminal, in the world coordinate system, the position of the right eye viewpoint relative to the right eye corresponding stereo model, from the right eye viewpoint A viewpoint setting unit that sets a right eye gaze direction that is a viewpoint direction, and a direction of a right eye upper vector that indicates an upward direction of the right eye viewpoint;
When the orientation of the user terminal is directed from the front direction determined by calibration to the left-right direction, the direction opposite to the left-eye line-of-sight direction is defined as the _ZVL axis positive direction, and the orientation of the left-eye upper vector was a Y _VL-axis positive direction, the left-eye viewpoint to X _VL-axis negative direction in the left-eye view coordinate system is orientation X _VL-axis positive direction constituting a right-hand system to the Z _VL-axis and Y _VL axis The direction opposite to the right eye line-of-sight direction is the Z _VR axis positive direction, the direction of the right eye upper vector is the Y _VR axis positive direction, and the right-handed system is set to the Z _VR axis and the Y _VR axis. orientation constituting the by moving the right eye viewpoint the X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction, viewpoint correction unit for correcting the position of the left eye viewpoint and the right eye viewpoint When,
Based on the corrected position of the left eye viewpoint, the left eye gaze direction, and the direction of the left eye upper vector, a processed left eye correspondence image is generated by two-dimensionally projecting the left eye correspondence stereo model. Based on the corrected position of the right eye viewpoint, the right eye gaze direction, and the direction of the right eye upper vector, the processed right eye corresponding image is generated by two-dimensionally projecting the right eye corresponding stereo model A projection conversion unit,
An image processing apparatus comprising: The viewpoint correction unit responds to a correction distance that is half of the difference between the predetermined distance and the parallax between the left-eye compatible camera and the right-eye compatible camera calculated according to the orientation of the user terminal. Moving the position of the left eye viewpoint and the right eye viewpoint by a distance of
The image processing apparatus according to claim 1. The modeling unit is a non-rendering area in which an image of the imaging target of the left-eye camera is not drawn among rendering areas in which the imaging target of the left-eye camera is rendered in the left-eye corresponding stereo model In the drawing area where the image of the imaging target of the right-eye corresponding camera in the right-eye corresponding stereo model is drawn in the right-eye corresponding stereo model, in which the drawing image is gradually transmitted toward the non-drawing area in the boundary area on the side In the boundary region on the non-rendering region side where the image to be imaged by the right-eye compatible camera is not rendered, a process of gradually transmitting the rendered image toward the non-rendering region is performed.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A computer that processes images from a left-eye compatible camera and a right-eye compatible camera installed so as to face the same direction at a predetermined distance, and generates an image to be displayed on a user terminal used by a user;
The left-eye corresponding image captured by the left-eye compatible camera and the right-eye corresponding image captured by the right-eye compatible camera are each projected and converted into a three-dimensional model, and the left-eye corresponding three-dimensional model and right A modeling unit that builds an eye-compatible stereo model;
Based on the orientation of the user terminal, in the world coordinate system, the position of the left eye viewpoint with respect to the left eye corresponding stereo model, the left eye gaze direction that is the viewpoint direction from the left eye viewpoint, and the left eye viewpoint The direction of the upper left eye vector indicating the upward direction is set, and based on the orientation of the user terminal, in the world coordinate system, the position of the right eye viewpoint relative to the right eye corresponding stereo model, from the right eye viewpoint A viewpoint setting unit that sets a right eye gaze direction that is a viewpoint direction, and a direction of a right eye upper vector that indicates an upward direction of the right eye viewpoint;
When the orientation of the user terminal is directed from the front direction determined by calibration to the left-right direction, the direction opposite to the left-eye line-of-sight direction is defined as the _ZVL axis positive direction, and the orientation of the left-eye upper vector was a Y _VL-axis positive direction, the left-eye viewpoint to X _VL-axis negative direction in the left-eye view coordinate system is orientation X _VL-axis positive direction constituting a right-hand system to the Z _VL-axis and Y _VL axis The direction opposite to the right eye line-of-sight direction is the Z _VR axis positive direction, the direction of the right eye upper vector is the Y _VR axis positive direction, and the right-handed system is set to the Z _VR axis and the Y _VR axis. orientation constituting the by moving the right eye viewpoint the X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction, viewpoint correction unit for correcting the position of the left eye viewpoint and the right eye viewpoint When,
Based on the corrected position of the left eye viewpoint, the left eye gaze direction, and the direction of the left eye upper vector, a processed left eye correspondence image is generated by two-dimensionally projecting the left eye correspondence stereo model. Based on the corrected position of the right eye viewpoint, the right eye gaze direction, and the direction of the right eye upper vector, the processed right eye corresponding image is generated by two-dimensionally projecting the right eye corresponding stereo model A projection conversion unit,
An image processing program that functions as an image processing program. An image processing method for processing images from a left-eye compatible camera and a right-eye compatible camera installed so as to face the same direction at a predetermined distance and generate an image to be displayed on a user terminal used by the user There,
The left-eye corresponding image captured by the left-eye compatible camera and the right-eye corresponding image captured by the right-eye compatible camera are each projected and converted into a three-dimensional model, and the left-eye corresponding three-dimensional model and right Modeling steps to build an eye-compatible stereo model;
Based on the orientation of the user terminal, in the world coordinate system, the position of the left eye viewpoint with respect to the left eye corresponding stereo model, the left eye gaze direction that is the viewpoint direction from the left eye viewpoint, and the left eye viewpoint The direction of the upper left eye vector indicating the upward direction is set, and based on the orientation of the user terminal, in the world coordinate system, the position of the right eye viewpoint relative to the right eye corresponding stereo model, from the right eye viewpoint A viewpoint setting step of setting a direction of a right eye gaze direction that is a viewpoint direction, and a direction of a right eye upper vector indicating an upper direction of the right eye viewpoint;
When the orientation of the user terminal is directed from the front direction determined by calibration to the left-right direction, the direction opposite to the left-eye line-of-sight direction is defined as the _ZVL axis positive direction, and the orientation of the left-eye upper vector was a Y _VL-axis positive direction, the left-eye viewpoint to X _VL-axis negative direction in the left-eye view coordinate system is orientation X _VL-axis positive direction constituting a right-hand system to the Z _VL-axis and Y _VL axis The direction opposite to the right eye line-of-sight direction is the Z _VR axis positive direction, the direction of the right eye upper vector is the Y _VR axis positive direction, and the right-handed system is set to the Z _VR axis and the Y _VR axis. orientation constituting the by moving the right eye viewpoint the X _VR-axis positive direction in the right-eye view coordinate system is X _VR-axis positive direction, viewpoint correction step of correcting a position of the left eye viewpoint and the right eye viewpoint When,
Based on the corrected position of the left eye viewpoint, the left eye gaze direction, and the direction of the left eye upper vector, a processed left eye correspondence image is generated by two-dimensionally projecting the left eye correspondence stereo model. Based on the corrected position of the right eye viewpoint, the right eye gaze direction, and the direction of the right eye upper vector, the processed right eye corresponding image is generated by two-dimensionally projecting the right eye corresponding stereo model A projection transformation step,
An image processing method comprising: