JP2012018665A - Image display program, apparatus, system and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To show a virtual object combined with a real world image on a stereoscopic display device in a normal and stereoscopic manner.SOLUTION: On the basis of a relative position and a relative attitude of an outer imaging unit (left) against a marker calculated from a marker recognition result of a left real world image and a relative position and a relative attitude of an outer imaging unit (right) against the marker calculated from a marker recognition result of a right real world image, a space between a left virtual camera and a right virtual camera is determined. After the space between the left virtual camera and the right virtual camera is determined, positions and attitudes of the left virtual camera and the right virtual camera are determined on the basis of the determined space so that the position and attitude of the left virtual camera has an ideal relationship with the position and attitude of the right virtual camera.

Description

  The present invention relates to an image display program, an apparatus, a system, and a method, and in particular, an image display program for stereoscopically displaying a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing. The present invention relates to an apparatus, a system, and a method.

In recent years, AR (Augmented Reality) is displayed by composing and displaying a virtual object on an image of the real world, as if the virtual object actually exists in the real world.
Research into augmented reality technology is advancing.

  For example, in the stereoscopic display device disclosed in Patent Document 1, the right-eye camera and the left-eye camera with respect to the markers arranged in the real space from the images captured by the right-eye camera and the left-eye camera attached to the head-mounted display, respectively. Relative positions and orientations are obtained, and based on the results, a right-eye virtual object image and a left-eye virtual object image are generated. Then, the image of the right eye virtual object and the image of the left eye virtual object are respectively synthesized with the images captured by the right eye camera and the left eye camera, and these synthesized images are combined with the right eye LCD (liquid crystal screen). Each is displayed on the left-eye LCD.

  Non-Patent Document 1 discloses a method for calculating the relative position and relative orientation between a marker and a camera in the real world based on the position and orientation of the marker in an image captured by the camera.

JP 2008-146109 A

Hirokazu Kato, Mark Billinghurst, "Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System," iwar, pp.85, 2nd IEEE and ACM International Workshop on Augmented Reality, 1999

  However, when calculating the relative position and orientation between the marker and the camera in the real world based on the position and orientation of the marker in the image captured by the camera, the image captured by the camera is blurred, If the recognition accuracy is not perfect, an error is included in the calculation result. Therefore, in the stereoscopic display device disclosed in Patent Document 1, the relative position and orientation of the right eye camera with respect to the marker calculated based on the image captured by the right eye camera is not necessarily accurate. The relative position and orientation of the left-eye camera calculated with respect to the marker calculated based on the captured image is not necessarily accurate. Since the image of the virtual object for the right eye and the image of the virtual object for the left eye generated based on an inaccurate calculation result are inconsistent images that do not match each other, the user cannot normally stereoscopically view the virtual object. .

  Therefore, an object of the present invention is to provide an image display program, an apparatus, a system, and a method capable of displaying a virtual object so that it can be normally stereoscopically viewed.

  The present invention employs the following configuration in order to solve the above problems.

  The image display program of the present invention uses an output from a real camera for the right eye and a real camera for the left eye to display a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing. An image display program for stereoscopic display, which causes a computer to function as first position and orientation calculation means, virtual camera setting means, right virtual space image generation means, left virtual space image generation means, and display control means. The first position / orientation calculation means includes real world image data output from the real camera for the right eye and real world image data output from the real camera for the left eye. By recognizing a predetermined imaging object in the world image data, position and orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging object is calculated. The virtual camera setting unit generates a right virtual camera and a left eye image for generating a right eye image in a predetermined virtual space, using the position and orientation information calculated by the first position and orientation calculation unit. Determine the position and orientation for both of the left virtual cameras to do. The right virtual space image generation means generates a right virtual space image indicating the virtual space viewed from the right virtual camera. The left virtual space image generation means generates a left virtual space image indicating the virtual space viewed from the left virtual camera. The display control means combines the right virtual space image with the real world image data output from the real camera for the right eye, and the left virtual image with the real world image data output from the real camera for the left eye. The spatial images are synthesized and an image for stereoscopic viewing is output to the stereoscopic display device.

  The virtual camera setting unit does not use the recognition result of the predetermined imaging target in the real world image data output from the other real camera different from the one real camera in the first position and orientation calculation unit. In addition, the position and orientation of both the right virtual camera for generating an image for the right eye and the left virtual camera for generating an image for the left eye in a predetermined virtual space may be determined.

  According to the above configuration, even if the predetermined imaging target can be recognized only in one of the two real world image data output from the two real cameras, the one that can recognize the predetermined imaging target Based on the position and orientation information calculated based on the real world image data, the position and orientation of one of the two virtual cameras is determined, and based on the thus determined position and orientation of the one virtual camera. Thus, the position and posture of the other virtual camera are determined. Therefore, even when the predetermined imaging target can be recognized only in one of the two real world image data output from one of the two real cameras, the virtual object can be appropriately displayed in a stereoscopic manner. . Further, even when the predetermined imaging target can be recognized in both of the two real world image data output from the two real cameras, the predetermined imaging target is obtained from either one of the real world image data. Since the virtual object can be appropriately stereoscopically displayed simply by recognizing, the processing load on the computer can be reduced.

  As another preferred configuration example, the virtual camera setting means is configured so that a relative posture relationship between the right virtual camera and the left virtual camera is between the real camera for the right eye and the real camera for the left eye. You may determine the position and attitude | position of the said right virtual camera and the said left virtual camera so that it may become the same as the relationship of the design relative attitude | position.

According to the above configuration, a real world image output from one of the two real cameras due to an error in recognition accuracy of the predetermined imaging target, an error in accuracy in attaching the two real cameras, or the like. The position and orientation information calculated based on the data and the position and orientation information calculated based on the real world image data output from one of the other real cameras accurately correspond to the relative postures of the two real cameras. Even if it is not, since the relative posture between the right virtual camera and the left virtual camera can be set appropriately, the virtual object can be appropriately displayed in three dimensions.

  As another preferred configuration example, the virtual camera setting unit uses the position and orientation information calculated by the first position and orientation calculation unit, and uses the first position and orientation of the right virtual camera and the left virtual camera. Based on the attitude of the one virtual camera determined by the first virtual camera attitude determination unit and a first virtual camera attitude determination unit that determines the attitude of one virtual camera corresponding to the one real camera in the calculation unit Thus, the relative posture relationship between the right virtual camera and the left virtual camera is the same as the design relative posture relationship between the real camera for the right eye and the real camera for the left eye. In addition, a second camera posture determination unit that determines the posture of the other virtual camera may be included.

  As another preferred configuration example, the image display program causes the computer to further function as a virtual camera relative positional relationship determining unit that determines a relative positional relationship between the right virtual camera and the left virtual camera, and the virtual camera setting unit One of the right virtual camera and the left virtual camera corresponding to the one real camera in the first position and orientation calculation means using the position and orientation information calculated by the first position and orientation calculation means Determined by the virtual camera relative positional relationship determining means from the first virtual camera position determining unit for determining the position of the virtual camera and the position of the one virtual camera determined by the first virtual camera position determining unit A second virtual camera position determining unit that determines the position of the other virtual camera at a position separated by the relative position; It may be.

  The “relative positional relationship” may be a distance between the left virtual camera and the right virtual camera, or a relative position of the other virtual camera based on one virtual camera. Also good.

  The virtual camera relative positional relationship determining means is configured to detect the right virtual camera and the left virtual camera based on a result of recognizing a predetermined imaging target in the two real world image data output from the two real cameras. Relative position relationship between the right virtual camera and the left virtual camera based on the design relative position relationship between the real camera for the right eye and the real camera for the left eye. The relative positional relationship may be determined.

  According to the above configuration, a real world image output from one of the two real cameras due to an error in recognition accuracy of the predetermined imaging target, an error in accuracy in attaching the two real cameras, or the like. The position / orientation information calculated based on the data and the position / orientation information calculated based on the real world image data output from one of the other real cameras are accurately related to the relative positional relationship between the two real cameras. Even if it is not compatible, since the relative positional relationship between the right virtual camera and the left virtual camera can be set appropriately, the virtual object can be appropriately displayed in three dimensions.

As another preferred configuration example, the virtual camera setting unit uses the position and orientation information calculated by the first position and orientation calculation unit, and uses the first position and orientation of the right virtual camera and the left virtual camera. Using the first virtual camera position determining unit that determines the position of one virtual camera corresponding to the one real camera in the calculating unit and the position / orientation information calculated by the first position / orientation calculating unit, the right Of the virtual camera and the left virtual camera, a first virtual camera posture determining unit that determines the posture of the one virtual camera corresponding to the one real camera in the first position and posture calculation means, and the first virtual camera The one virtual camera determined by the first virtual camera posture determining unit as viewed from the position of the one virtual camera determined by the position determining unit. In a direction based on the orientation of the camera, it may include a second virtual camera position determination unit for determining the position of the other of the virtual camera.

  In addition, the second virtual camera position determination unit is the one determined by the first virtual camera posture determination unit when viewed from the position of the one virtual camera determined by the first virtual camera position determination unit. You may determine the position of the other virtual camera in the horizontal direction of the attitude | position of a virtual camera.

  As another preferred configuration example, the computer is further caused to function as a virtual inter-camera distance determining unit that determines a distance between the right virtual camera and the left virtual camera, and the second camera position determination unit includes the first camera position determination unit. The distance between the virtual cameras is determined in a direction based on the posture of the one virtual camera determined by the first virtual camera posture determination unit as seen from the position of the one virtual camera determined by the virtual camera position determination unit. The position of the other virtual camera may be determined at a position separated by the distance determined by the means.

  As another preferred configuration example, the virtual camera relative positional relationship determining means is based on a parallax between real world image data output from the real camera for the right eye and real world image data output from the real camera for the left eye. Then, the relative positional relationship between the right virtual camera and the left virtual camera may be determined.

  According to the above configuration example, since the relative positional relationship between the right virtual camera and the left virtual camera is determined based on the parallax between the two real world image data, the relative positional relationship between the two real cameras is not known. In some cases, the right virtual camera and the left virtual camera can be appropriately set even if the relative positional relationship between the two real cameras includes an error due to an error in the mounting accuracy of the two real cameras.

  As another preferred configuration example, the relative positional relationship determined by the relative positional relationship determining means may be an interval between the right virtual camera and the left virtual camera.

  As another preferable configuration example, the image display program may include the computer, the one real camera in the first position / orientation calculation unit of the two real world image data respectively output from the two real cameras. Is a position and orientation indicating a relative position and orientation between the other real camera and the imaging target by recognizing the predetermined imaging target in real world image data output from another different real camera The virtual camera relative position relationship determining means is further operated as a second position / orientation calculating means for calculating information, and the virtual camera relative position relationship determining means is a relative position between the one real camera calculated by the first position / orientation calculating means and the imaging target. Information on the relative position between the other real camera calculated by the second position and orientation calculation means and the imaging target, Used may calculate the distance between one of the real camera and the other real camera above.

  As another preferable configuration example, the first position / orientation calculation unit is configured to generate a coordinate value represented by a coordinate system with the predetermined imaging target as an origin based on real world image data output from the one real camera. Including first conversion matrix generation means for generating a first conversion matrix for converting into a coordinate value represented by a first imaging unit coordinate system with the one real camera as the origin. The means is configured to perform second imaging with the coordinate value represented by the coordinate system with the predetermined imaging target as the origin as the origin, with the other real camera as the origin, based on the real world image data output from the other real camera. You may include the 2nd conversion matrix production | generation means which produces | generates the 2nd conversion matrix for converting into the coordinate value represented by a partial coordinate system.

As another preferred configuration example, the virtual camera relative positional relationship determining means determines the relative positional relationship between the right virtual camera and the left virtual camera each time new real world image data is output from the two real cameras. You may determine the relative positional relationship of the said right virtual camera and the said left virtual camera based on the relative positional relationship calculation result for multiple times obtained by performing the process to calculate and performing the said process in multiple times.

  According to the above configuration example, since the influence of an error in recognition accuracy when recognizing the predetermined imaging target in real world image data is reduced, the right virtual camera determined by the virtual camera relative positional relationship determining unit And the reliability of the relative positional relationship of the left virtual camera is improved.

  As another preferred configuration example, the virtual camera relative positional relationship determination means is based on the plurality of relative positional relationship calculation results only when the multiple relative positional relationship calculation results are all within a predetermined range. Then, the relative positional relationship between the right virtual camera and the left virtual camera may be determined.

  According to the above configuration example, the reliability of the relative positional relationship between the right virtual camera and the left virtual camera determined by the virtual camera relative positional relationship determining unit is improved.

  As another preferred configuration example, the image display program causes the relative positional relationship to be determined after the computer determines the relative positional relationship between the right virtual camera and the left virtual camera by the virtual camera relative positional relationship determining unit. The virtual camera relative positional relationship is determined by further functioning as a depth change determination unit that determines whether or not the depth distance of the predetermined imaging target with respect to the two real cameras has changed beyond a predetermined range from the determined time point. The means may re-determine the relative positional relationship between the right virtual camera and the left virtual camera when the determination result of the depth change determination means is affirmative.

  The magnitude of the influence of the error in mounting accuracy of the two actual cameras changes depending on the depth distance from the two actual cameras to the predetermined imaging target. According to the configuration example, It is possible to appropriately correct the influence due to the error of the mounting accuracy of the actual camera as needed.

  As another preferable configuration example, the image display program causes the computer to calculate a depth distance of the predetermined imaging target with respect to the two real cameras based on real world image data including the predetermined imaging target. The depth change calculating means further functions as a depth distance calculating means for calculating, and the depth change calculating means calculates the depth distance when the relative positional relation between the right virtual camera and the left virtual camera is determined by the virtual camera relative positional relation determining means. By comparing the reference depth distance calculated by the means and the latest depth distance calculated by the depth distance calculation means, the depth distance of the predetermined imaging target with respect to the two actual cameras exceeds a predetermined range. It may be determined whether or not it has changed.

  As another preferable configuration example, the depth distance calculation unit calculates the depth distance of the predetermined imaging target with respect to the two real cameras each time new real world image data is output from the two real cameras. The reference depth distance may be calculated based on a plurality of depth distance calculation results obtained by executing the process and executing the process a plurality of times.

  According to the above configuration example, since the influence of the recognition accuracy error when recognizing the predetermined imaging target in real world image data is reduced, the two real cameras calculated by the depth distance calculating unit The reliability of the depth distance of a predetermined imaging target is improved.

  As another preferable configuration example, the depth distance calculation unit may calculate the reference depth distance based on the plurality of depth distance calculation results only when the plurality of depth distance calculation results are all within a predetermined range. May be calculated.

  According to the above configuration example, the reliability of the depth distance of the predetermined imaging target with respect to the two real cameras calculated by the depth distance calculation unit is improved.

  As another preferable configuration example, the image display program may include the computer and the one real camera in the first position / orientation calculation unit of the two real world image data output from the two real cameras. Shows the relative position and orientation between the other real camera and the predetermined imaging object by recognizing the predetermined imaging object in the real world image data output from another different real camera According to the position of the predetermined imaging object in at least one of the two real world image data output from the two real cameras, the second position and orientation calculation means for calculating the position and orientation information, respectively. It further functions as a real camera selection means for selecting one of the cameras, and the virtual camera setting means is controlled by the real camera selection means. When the selected real camera is the one real camera, the relative position and posture between the one real camera and the predetermined imaging target calculated by the first position / orientation calculation unit are calculated. The position and orientation information shown is used to determine the positions and orientations of the right virtual camera and the left virtual camera, and when the real camera selected by the real camera selection means is the other real camera, the second virtual camera The position of the right virtual camera and the left virtual camera is calculated using the position and orientation information indicating the relative position and orientation between the other real camera and the predetermined imaging target, calculated by the position and orientation calculation means. And the posture may be determined.

  According to the above configuration example, even if the predetermined imaging target is out of the shooting range of one real camera, the right virtual camera and the left virtual camera can be appropriately set, so that the stereoscopic display of the virtual object is continued. be able to.

  As another preferable configuration example, the real camera selection unit may be configured such that the position of the predetermined imaging target in the real world image data output from the real camera for the left eye of the two real cameras is the real world image data. In response to entering the right end area, the real camera for the left eye is switched to the real camera for the right eye, and the position of the predetermined imaging target in the real world image data output from the real camera for the right eye is In response to entering the left end area of the real world image data, the real camera for the right eye may be switched to the real camera for the left eye.

  As another preferable configuration example, the real world image data used by the first position and orientation calculation unit and the display control unit may be real world image data output in real time from the real camera.

  According to the above configuration example, it is possible to make the virtual object appear to exist in real time in the real world on the screen of the stereoscopic display device.

  As another preferable configuration example, the computer may be incorporated in an information processing apparatus including the two real cameras and the stereoscopic display device.

The image display device of the present invention uses a real camera for a right eye and an output from a real camera for a left eye to display a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing. An image display device that performs stereoscopic display, and includes first position and orientation calculation means, virtual camera setting means, right virtual space image generation means, left virtual space image generation means, and display control means. The first position / orientation calculation means includes real world image data output from the real camera for the right eye and real world image data output from the real camera for the left eye. By recognizing a predetermined imaging object in the world image data, position and orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging object is calculated.
The virtual camera setting unit generates a right virtual camera and a left eye image for generating a right eye image in a predetermined virtual space, using the position and orientation information calculated by the first position and orientation calculation unit. Determine the position and orientation for both of the left virtual cameras to do. The right virtual space image generation means generates a right virtual space image indicating the virtual space viewed from the right virtual camera. The left virtual space image generation means generates a left virtual space image indicating the virtual space viewed from the left virtual camera. The display control means combines the right virtual space image with the real world image data output from the real camera for the right eye, and the left virtual image with the real world image data output from the real camera for the left eye. The spatial images are synthesized and an image for stereoscopic viewing is output to the stereoscopic display device.

  The image display system of the present invention uses an output from a real camera for the right eye and a real camera for the left eye to display a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing. An image display system that performs stereoscopic display, and includes first position and orientation calculation means, virtual camera setting means, right virtual space image generation means, left virtual space image generation means, and display control means. The first position / orientation calculation means includes real world image data output from the real camera for the right eye and real world image data output from the real camera for the left eye. By recognizing a predetermined imaging object in the world image data, position and orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging object is calculated. The virtual camera setting unit generates a right virtual camera and a left eye image for generating a right eye image in a predetermined virtual space, using the position and orientation information calculated by the first position and orientation calculation unit. Determine the position and orientation for both of the left virtual cameras to do. The right virtual space image generation means generates a right virtual space image indicating the virtual space viewed from the right virtual camera. The left virtual space image generation means generates a left virtual space image indicating the virtual space viewed from the left virtual camera. The display control means combines the right virtual space image with the real world image data output from the real camera for the right eye, and the left virtual image with the real world image data output from the real camera for the left eye. The spatial images are synthesized and an image for stereoscopic viewing is output to the stereoscopic display device.

  The image display method of the present invention uses an output from a real camera for the right eye and a real camera for the left eye to display a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing. An image display method for stereoscopic display, comprising a first position and orientation calculation step, a virtual camera setting step, a right virtual space image generation step, a left virtual space image generation step, and a display control step. In the first position / orientation calculation step, the real world image data output from the real camera for the right eye and the real world image data output from the real camera for the left eye are output from one real camera. By recognizing a predetermined imaging object in the world image data, position and orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging object is calculated. In the virtual camera setting step, a right virtual camera and a left eye image for generating a right eye image in a predetermined virtual space are generated using the position and orientation information calculated in the first position and orientation calculation step. Determine the position and orientation for both of the left virtual cameras to do. In the right virtual space image generation step, a right virtual space image indicating the virtual space viewed from the right virtual camera is generated. In the left virtual space image generation step, a left virtual space image indicating the virtual space viewed from the left virtual camera is generated. In the display control step, the right virtual space image is synthesized with the real world image data output from the right eye real camera, and the left virtual image is output to the real world image data output from the left eye real camera. The spatial images are synthesized and an image for stereoscopic viewing is output to the stereoscopic display device.

  According to the present invention, a virtual object can be displayed so that it can be normally stereoscopically viewed.

Front view of game device 10 in the open state Side view of game device 10 in open state Left side view, front view, right side view, and rear view of game device 10 in the closed state A-A 'line sectional view of upper housing 21 shown in FIG. The figure which shows a mode that the slider 25a of the 3D adjustment switch 25 exists in the lowest point (3rd position). The figure which shows a mode that the slider 25a of 3D adjustment switch 25 exists in a position (1st position) above a lowest point. The figure which shows a mode that the slider 25a of the 3D adjustment switch 25 exists in the highest point (2nd position). Block diagram showing the internal configuration of the game apparatus 10 The figure which shows an example of the stereo image displayed on the screen of upper LCD22 The figure which shows another example of the stereo image displayed on the screen of upper LCD22. The figure which shows the marker 61 The figure which shows another example of the stereo image displayed on the screen of upper LCD22. The figure which shows the memory map of the main memory 32 of the game device 10 The figure which shows an example of the various variables stored in the main memory 32 Flow chart showing the flow of marker processing Flow chart showing the flow of main processing Flow chart showing details of update process Flow chart showing details of virtual camera interval determination processing Flowchart showing details of view matrix generation processing Flow chart showing details of main real world image setting process Flow chart showing details of display mode switching processing A diagram showing an example of a left real world image and a right real world image The figure which shows the position and attitude | position of the left virtual camera 63L calculated according to the result of a marker recognition process The figure which shows the position and attitude | position of the right virtual camera 63R calculated according to the result of a marker recognition process The figure which shows the cut-out range of the left real world image based on the stereoscopic vision zero distance The figure which shows the cut-out range of the right real world image based on the stereoscopic zero distance The figure which shows the positional relationship of the virtual object 62 and the left virtual camera 63L. The figure which shows the production | generation method of the image for left eyes The figure which shows the calculation method of the coordinate of the right virtual camera 63R represented by the left virtual camera coordinate system Diagram showing ew and ed The figure which shows the calculation method of the coordinate of the left virtual camera 63L and the right virtual camera 63R represented by a marker coordinate system. The figure which shows the position of the right virtual camera 63R determined based on the position and attitude | position of the left virtual camera 63L The figure which shows the position of the left virtual camera 63L determined based on the position and attitude | position of the right virtual camera 63R. | V. Figure showing z | Diagram for explaining the reason for recalculating EyeWidth Diagram for explaining the reason for recalculating EyeWidth Diagram for explaining the reason for recalculating EyeWidth Diagram for explaining the reason for recalculating EyeWidth Diagram for explaining left projection matrix Illustration for explaining the right projection matrix The figure which shows the judgment method of whether to switch a main real world image from a left real world image to a right real world image The figure which shows the judgment method of whether to switch a main real world image from a right real world image to a left real world image

(Configuration of game device)
Hereinafter, a game device according to an embodiment of the present invention will be described. 1 to 3 are plan views showing the appearance of the game apparatus 10. The game apparatus 10 is a portable game apparatus, and is configured to be foldable as shown in FIGS. 1 and 2 show the game apparatus 10 in an open state (open state), and FIG. 3 shows the game apparatus 10 in a closed state (closed state). FIG. 1 is a front view of the game apparatus 10 in the open state, and FIG. 2 is a right side view of the game apparatus 10 in the open state. The game apparatus 10 can capture an image with an imaging unit, display the captured image on a screen, and store data of the captured image. In addition, the game device 10 can be stored in a replaceable memory card or can execute a game program received from a server or another game device, and can be an image captured by a virtual camera set in a virtual space. An image generated by computer graphics processing can be displayed on the screen.

  First, the external configuration of the game apparatus 10 will be described with reference to FIGS. As shown in FIGS. 1 to 3, the game apparatus 10 includes a lower housing 11 and an upper housing 21. The lower housing 11 and the upper housing 21 are connected so as to be openable and closable (foldable). In the present embodiment, each of the housings 11 and 21 has a horizontally long rectangular plate shape, and is connected so as to be rotatable at the long side portions of each other.

  As shown in FIGS. 1 and 2, the upper long side portion of the lower housing 11 is provided with a protruding portion 11 </ b> A that protrudes in a direction perpendicular to the inner surface (main surface) 11 </ b> B of the lower housing 11. . In addition, the lower long side portion of the upper housing 21 is provided with a protruding portion 21A that protrudes from the lower side surface of the upper housing 21 in a direction perpendicular to the lower side surface. By connecting the protrusion 11A of the lower housing 11 and the protrusion 21A of the upper housing 21, the lower housing 11 and the upper housing 21 are foldably connected.

(Description of lower housing)
First, the configuration of the lower housing 11 will be described. As shown in FIGS. 1 to 3, the lower housing 11 includes a lower LCD (Liquid Crystal Display) 12, a touch panel 13, operation buttons 14A to 14L (FIGS. 1 and 3), an analog stick. 15, LED16A-16B, the insertion port 17, and the hole 18 for microphones are provided. Details of these will be described below.

As shown in FIG. 1, the lower LCD 12 is housed in the lower housing 11. The lower LCD 12 has a horizontally long shape, and is arranged such that the long side direction coincides with the long side direction of the lower housing 11. The lower LCD 12 is disposed at the center of the lower housing 11. The lower LCD 12 is provided on the inner surface (main surface) of the lower housing 11, and the screen of the lower LCD 12 is exposed from an opening provided in the lower housing 11. When the game apparatus 10 is not used, it is possible to prevent the screen of the lower LCD 12 from becoming dirty or damaged by keeping the game apparatus 10 closed. The number of pixels of the lower LCD 12 may be, for example, 256 dots × 192 dots (horizontal × vertical). Unlike the upper LCD 22 described later, the lower LCD 12 is a display device that displays an image in a planar manner (not stereoscopically viewable). In this embodiment, an LCD is used as the display device. For example, an EL (Electro Luminescence) is used.
Any other display device such as a display device using ce (electroluminescence) may be used. Further, as the lower LCD 12, a display device having an arbitrary resolution can be used.

  As shown in FIG. 1, the game apparatus 10 includes a touch panel 13 as an input device. The touch panel 13 is mounted on the screen of the lower LCD 12. In the present embodiment, the touch panel 13 is a resistive film type touch panel. However, the touch panel is not limited to the resistive film type, and any type of touch panel such as a capacitance type can be used. In the present embodiment, the touch panel 13 having the same resolution (detection accuracy) as that of the lower LCD 12 is used. However, the resolution of the touch panel 13 and the resolution of the lower LCD 12 do not necessarily match. An insertion port 17 (dotted line shown in FIGS. 1 and 3D) is provided on the upper side surface of the lower housing 11. The insertion slot 17 can accommodate a touch pen 28 used for performing an operation on the touch panel 13. In addition, although the input with respect to the touchscreen 13 is normally performed using the touch pen 28, it is also possible to input with respect to the touchscreen 13 not only with the touch pen 28 but with a user's finger | toe.

  Each operation button 14A-14L is an input device for performing a predetermined input. As shown in FIG. 1, on the inner surface (main surface) of the lower housing 11, among the operation buttons 14A to 14L, a cross button 14A (direction input button 14A), a button 14B, a button 14C, a button 14D, A button 14E, a power button 14F, a select button 14J, a HOME button 14K, and a start button 14L are provided. The cross button 14 </ b> A has a cross shape, and has buttons for instructing up, down, left, and right directions. The button 14B, the button 14C, the button 14D, and the button 14E are arranged in a cross shape. Functions according to a program executed by the game apparatus 10 are appropriately assigned to the buttons 14A to 14E, the select button 14J, the HOME button 14K, and the start button 14L. For example, the cross button 14A is used for a selection operation or the like, and the operation buttons 14B to 14E are used for a determination operation or a cancel operation, for example. The power button 14F is used to turn on / off the power of the game apparatus 10.

  The analog stick 15 is a device that indicates a direction, and is provided in an upper area of the left area from the lower LCD 12 of the inner surface of the lower housing 11. As shown in FIG. 1, since the cross button 14A is provided in the lower region on the left side of the lower LCD 12, the analog stick 15 is provided above the cross button 14A. In addition, the analog stick 15 and the cross button 14A are designed to be operable with the thumb of the left hand holding the lower housing. Further, by providing the analog stick 15 in the upper region, the analog stick 15 is arranged where the thumb of the left hand holding the lower housing 11 is naturally positioned, and the cross button 14A has the thumb of the left hand slightly below. Arranged at shifted positions. The analog stick 15 is configured such that its key top slides parallel to the inner surface of the lower housing 11. The analog stick 15 functions according to a program executed by the game apparatus 10. For example, when a game in which a predetermined object appears in the three-dimensional virtual space is executed by the game apparatus 10, the analog stick 15 functions as an input device for moving the predetermined object in the three-dimensional virtual space. In this case, the predetermined object is moved in the direction in which the key top of the analog stick 15 slides. In addition, as the analog stick 15, an analog stick that allows analog input by being tilted by a predetermined amount in any direction of up / down / left / right and oblique directions may be used.

The four buttons 14B, 14C, 14D, and 14E arranged in a cross shape are arranged where the thumb of the right hand that holds the lower housing 11 is naturally positioned. Also, these four buttons and the analog stick 15 are arranged symmetrically with the lower LCD 12 in between. Thereby, depending on the game program, for example,
It is also possible for a left-handed person to input a direction using these four buttons.

  A microphone hole 18 is provided on the inner surface of the lower housing 11. A microphone (see FIG. 6), which will be described later, is provided below the microphone hole 18, and the microphone detects sound outside the game apparatus 10.

  3A is a left side view of the game apparatus 10 in the closed state, FIG. 3B is a front view of the game apparatus 10 in the closed state, and FIG. 3C is a view of the game apparatus 10 in the closed state. FIG. 3D is a right side view, and FIG. 3D is a rear view of the game apparatus 10 in the closed state. As shown in FIGS. 3B and 3D, an L button 14 </ b> G and an R button 14 </ b> H are provided on the upper side surface of the lower housing 11. The L button 14 </ b> G is provided at the left end portion of the upper surface of the lower housing 11, and the R button 14 </ b> H is provided at the right end portion of the upper surface of the lower housing 11. The L button 14G and the R button 14H can function as, for example, a shutter button (shooting instruction button) of the imaging unit. Further, as shown in FIG. 3A, a volume button 14 </ b> I is provided on the left side surface of the lower housing 11. The volume button 14I is used to adjust the volume of a speaker provided in the game apparatus 10.

  As shown in FIG. 3A, an openable / closable cover portion 11 </ b> C is provided on the left side surface of the lower housing 11. A connector (not shown) for electrically connecting the game apparatus 10 and the data storage external memory 45 is provided inside the cover portion 11C. The data storage external memory 45 is detachably attached to the connector. The data storage external memory 45 is used, for example, for storing (saving) data of an image captured by the game apparatus 10. The connector and its cover portion 11 </ b> C may be provided on the right side surface of the lower housing 11.

  As shown in FIG. 3D, an insertion port 11D for inserting the game apparatus 10 and an external memory 44 in which a game program is recorded is provided on the upper side surface of the lower housing 11, and the insertion port Inside the 11D, a connector (not shown) for electrically detachably connecting to the external memory 44 is provided. When the external memory 44 is connected to the game apparatus 10, a predetermined game program is executed. The connector and its insertion port 11D may be provided on the other side surface of the lower housing 11 (for example, the right side surface).

  Further, as shown in FIG. 1 and FIG. 3C, on the lower side surface of the lower housing 11, the first LED 16A for notifying the user of the power ON / OFF state of the game apparatus 10 and the right side surface of the lower housing 11 Is provided with a second LED 16B for notifying the user of the wireless communication establishment status of the game apparatus 10. The game apparatus 10 can perform wireless communication with other devices, and the first LED 16B lights up when wireless communication is established. The game apparatus 10 is, for example, IEEE 802.11. It has a function of connecting to a wireless LAN by a method compliant with the b / g standard. A wireless switch 19 for enabling / disabling this wireless communication function is provided on the right side surface of the lower housing 11 (see FIG. 3C).

  Although not shown, the lower housing 11 stores a rechargeable battery that serves as a power source for the game apparatus 10, and the terminal is provided via a terminal provided on a side surface (for example, the upper side surface) of the lower housing 11. The battery can be charged.

(Description of upper housing)
Next, the configuration of the upper housing 21 will be described. As shown in FIGS. 1 to 3, the upper housing 21 includes an upper LCD (Liquid Crystal Display) 22, an outer imaging unit 23 (an outer imaging unit (left) 23a and an outer imaging unit (right) 23b). , An inner imaging unit 24, a 3D adjustment switch 25, and a 3D indicator 26 are provided. Details of these will be described below.

  As shown in FIG. 1, the upper LCD 22 is accommodated in the upper housing 21. The upper LCD 22 has a horizontally long shape and is arranged such that the long side direction coincides with the long side direction of the upper housing 21. The upper LCD 22 is disposed in the center of the upper housing 21. The area of the screen of the upper LCD 22 is set larger than the area of the screen of the lower LCD 12. Further, the screen of the upper LCD 22 is set to be horizontally longer than the screen of the lower LCD 12. That is, the ratio of the horizontal width in the aspect ratio of the screen of the upper LCD 22 is set larger than the ratio of the horizontal width in the aspect ratio of the screen of the lower LCD 12.

  The screen of the upper LCD 22 is provided on the inner surface (main surface) 21 </ b> B of the upper housing 21, and the screen of the upper LCD 22 is exposed from an opening provided in the upper housing 21. As shown in FIG. 2, the inner surface of the upper housing 21 is covered with a transparent screen cover 27. The screen cover 27 protects the screen of the upper LCD 22 and is integrated with the upper LCD 22 and the inner surface of the upper housing 21, thereby providing a sense of unity. The number of pixels of the upper LCD 22 may be, for example, 640 dots × 200 dots (horizontal × vertical). In the present embodiment, the upper LCD 22 is a liquid crystal display device. However, for example, a display device using EL (Electro Luminescence) may be used. In addition, a display device having an arbitrary resolution can be used as the upper LCD 22.

  The upper LCD 22 is a display device capable of displaying a stereoscopically visible image. In the present embodiment, the left-eye image and the right-eye image are displayed using substantially the same display area. Specifically, the display device uses a method in which a left-eye image and a right-eye image are alternately displayed in a horizontal direction in a predetermined unit (for example, one column at a time). Alternatively, a display device of a type in which the left-eye image and the right-eye image are alternately displayed may be used. In this embodiment, the display device is capable of autostereoscopic viewing. A lenticular method or a parallax barrier method (parallax barrier method) is used so that the left-eye image and the right-eye image that are alternately displayed in the horizontal direction appear to be decomposed into the left eye and the right eye, respectively. In the present embodiment, the upper LCD 22 is a parallax barrier type. The upper LCD 22 uses the right-eye image and the left-eye image to display an image (stereoscopic image) that can be stereoscopically viewed with the naked eye. In other words, the upper LCD 22 displays a stereoscopic image (stereoscopically viewable) having a stereoscopic effect for the user by using the parallax barrier to make the left eye image visible to the user's left eye and the right eye image to the user's right eye. be able to. Further, the upper LCD 22 can invalidate the parallax barrier. When the parallax barrier is invalidated, the upper LCD 22 can display an image in a planar manner (in the sense opposite to the above-described stereoscopic view, the planar LCD is planar). (This is a display mode in which the same displayed image can be seen by both the right eye and the left eye). As described above, the upper LCD 22 is a display device capable of switching between a stereoscopic display mode for displaying a stereoscopically viewable image and a planar display mode for displaying an image in a planar manner (displaying a planar view image). . This display mode switching is performed by a 3D adjustment switch 25 described later.

The outer imaging unit 23 is a general term for two imaging units (23 a and 23 b) provided on the outer surface (back surface opposite to the main surface on which the upper LCD 22 is provided) 21 D of the upper housing 21. The imaging directions of the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are both normal directions of the outer surface 21D. Further, all of these imaging units are designed in a direction 180 degrees opposite to the normal direction of the display surface (inner side surface) of the upper LCD 22. That is, the imaging direction of the outer imaging unit (left) 23a and the imaging direction of the outer imaging unit (right) 23b are parallel. The outer imaging unit (left) 23a and the outer imaging unit (right) 23b can be used as a stereo camera by a program executed by the game apparatus 10. Further, depending on the program, it is possible to use one of the two outer imaging units (23a and 23b) alone and use the outer imaging unit 23 as a non-stereo camera. Further, depending on the program, it is possible to perform imaging with an expanded imaging range by combining or complementarily using images captured by the two outer imaging units (23a and 23b). In the present embodiment, the outer imaging unit 23 includes two imaging units, an outer imaging unit (left) 23a and an outer imaging unit (right) 23b. The outer imaging unit (left) 23a and the outer imaging unit (right) 23b each include an imaging element (for example, a CCD image sensor or a CMOS image sensor) having a predetermined common resolution and a lens. The lens may have a zoom mechanism.

  As indicated by the broken line in FIG. 1 and the solid line in FIG. 3B, the outer imaging unit (left) 23 a and the outer imaging unit (right) 23 b that constitute the outer imaging unit 23 are arranged in the horizontal direction of the screen of the upper LCD 22. Arranged in parallel. That is, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are arranged so that a straight line connecting the two imaging units is parallel to the horizontal direction of the screen of the upper LCD 22. 1 indicate the outer imaging unit (left) 23a and the outer imaging unit (right) 23b existing on the outer surface opposite to the inner surface of the upper housing 21, respectively. As shown in FIG. 1, when the user views the screen of the upper LCD 22 from the front, the outer imaging unit (left) 23a is positioned on the left side and the outer imaging unit (right) 23b is positioned on the right side. When a program that causes the outer imaging unit 23 to function as a stereo camera is executed, the outer imaging unit (left) 23a captures an image for the left eye that is visually recognized by the user's left eye, and the outer imaging unit (right) 23b A right-eye image that is visually recognized by the user's right eye is captured. The interval between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b is set to be approximately the interval between both eyes of the human, and may be set in a range of 30 mm to 70 mm, for example. In addition, the space | interval of the outer side imaging part (left) 23a and the outer side imaging part (right) 23b is not restricted to this range.

  In the present embodiment, the outer imaging unit (left) 23a and the outer imaging unit (right) 23 are fixed to the housing, and the imaging direction cannot be changed.

  Further, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are respectively arranged at positions symmetrical from the center with respect to the left-right direction of the upper LCD 22 (upper housing 21). That is, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are respectively arranged at symmetrical positions with respect to a line dividing the upper LCD 22 into two equal parts. In addition, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are on the upper side of the upper housing 21 in a state where the upper housing 21 is opened, and on the back side of the upper side of the screen of the upper LCD 22. Placed in. That is, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are the outer surfaces of the upper housing 21, and when the upper LCD 22 is projected on the outer surface, the upper image 22 is higher than the upper end of the projected upper LCD 22 screen. Placed in.

  As described above, the two image pickup units (23a and 23b) of the outer image pickup unit 23 are arranged at symmetrical positions from the center with respect to the left-right direction of the upper LCD 22, so that when the user views the upper LCD 22 from the front, the outer image pickup is performed. The imaging direction of the unit 23 can be matched with the user's line-of-sight direction. Further, since the outer imaging unit 23 is disposed at a position on the back side above the upper end of the screen of the upper LCD 22, the outer imaging unit 23 and the upper LCD 22 do not interfere inside the upper housing 21. Accordingly, it is possible to make the upper housing 21 thinner than in the case where the outer imaging unit 23 is disposed on the back side of the screen of the upper LCD 22.

  The inner imaging unit 24 is an imaging unit that is provided on the inner side surface (main surface) 21B of the upper housing 21 and has an inward normal direction of the inner side surface as an imaging direction. The inner imaging unit 24 includes an imaging element (for example, a CCD image sensor or a CMOS image sensor) having a predetermined resolution, and a lens. The lens may have a zoom mechanism.

As shown in FIG. 1, the inner imaging unit 24 is disposed above the upper end of the upper LCD 22 and in the left-right direction of the upper housing 21 when the upper housing 21 is opened. Is arranged at the center position (on the line dividing the upper housing 21 (the screen of the upper LCD 22) into left and right halves). Specifically, as shown in FIG. 1 and FIG. 3B, the inner imaging unit 24 is an inner surface of the upper housing 21, and the left and right imaging units (outer imaging unit (left ) 23a and the outer imaging unit (right) 23b) are arranged at positions on the back side in the middle. That is, when the left and right imaging units of the outer imaging unit 23 provided on the outer surface of the upper housing 21 are projected onto the inner surface of the upper housing 21, the inner imaging unit 24 is provided in the middle of the projected left and right imaging units. It is done. A broken line 24 shown in FIG. 3B represents the inner imaging unit 24 existing on the inner surface of the upper housing 21.

  Thus, the inner imaging unit 24 images in the opposite direction to the outer imaging unit 23. The inner imaging unit 24 is provided on the inner surface of the upper housing 21 and on the back side of the intermediate position between the left and right imaging units of the outer imaging unit 23. Thereby, when the user views the upper LCD 22 from the front, the inner imaging unit 24 can capture the user's face from the front. Further, since the left and right imaging units of the outer imaging unit 23 and the inner imaging unit 24 do not interfere inside the upper housing 21, the upper housing 21 can be configured to be thin.

  The 3D adjustment switch 25 is a slide switch, and is a switch used to switch the display mode of the upper LCD 22 as described above. The 3D adjustment switch 25 is used to adjust the stereoscopic effect of a stereoscopically viewable image (stereoscopic image) displayed on the upper LCD 22. As shown in FIGS. 1 to 3, the 3D adjustment switch 25 is provided on the inner side surface and the right side end of the upper housing 21, and when the user views the upper LCD 22 from the front, the 3D adjustment switch 25 is visually recognized. It is provided at a position where it can be made. Moreover, the operation part of 3D adjustment switch 25 protrudes in both the inner surface and the right side surface, and can be visually recognized and operated from both. All switches other than the 3D adjustment switch 25 are provided in the lower housing 11.

  FIG. 4 is a cross-sectional view taken along line A-A ′ of the upper housing 21 shown in FIG. 1. As shown in FIG. 4, a recess 21C is formed at the right end of the inner surface of the upper housing 21, and a 3D adjustment switch 25 is provided in the recess 21C. As shown in FIGS. 1 and 2, the 3D adjustment switch 25 is disposed so as to be visible from the front surface and the right side surface of the upper housing 21. The slider 25a of the 3D adjustment switch 25 can be slid to an arbitrary position in a predetermined direction (vertical direction), and the display mode of the upper LCD 22 is set according to the position of the slider 25a.

  5A to 5C are diagrams illustrating a state in which the slider 25a of the 3D adjustment switch 25 slides. FIG. 5A is a diagram illustrating a state in which the slider 25a of the 3D adjustment switch 25 is present at the lowest point (third position). FIG. 5B is a diagram illustrating a state in which the slider 25a of the 3D adjustment switch 25 is present at a position above the lowest point (first position). FIG. 5C is a diagram illustrating a state in which the slider 25a of the 3D adjustment switch 25 exists at the uppermost point (second position).

As shown in FIG. 5A, when the slider 25a of the 3D adjustment switch 25 exists at the lowest point position (third position), the upper LCD 22 is set to the flat display mode, and a flat image is displayed on the screen of the upper LCD 22. (Alternatively, the upper LCD 22 may remain in the stereoscopic display mode, and the left-eye image and the right-eye image may be displayed as a flat image by making them the same image). On the other hand, when the slider 25a exists between the position shown in FIG. 5B (position above the lowest point (first position)) and the position shown in FIG. 5C (position of the highest point (second position)). The upper LCD 22 is set to the stereoscopic display mode. In this case, a stereoscopically viewable image is displayed on the screen of the upper LCD 22. When the slider 25a exists between the first position and the second position, the appearance of the stereoscopic image is adjusted according to the position of the slider 25a. Specifically, the shift amount of the horizontal position of the right-eye image and the left-eye image is adjusted according to the position of the slider 25a. The slider 25a of the 3D adjustment switch 25 is configured to be fixed at the third position, and is configured to be slidable to an arbitrary position in the vertical direction between the first position and the second position. Yes. For example, in the third position, the slider 25a is fixed by a convex portion (not shown) protruding laterally as shown in FIG. 5A from the side surface forming the 3D adjustment switch 25, and a predetermined force or more is applied upward. Otherwise, it is configured not to slide upward from the third position. When the slider 25a exists from the third position to the first position, the appearance of the stereoscopic image is not adjusted, but this is so-called play. In another example, play may be eliminated and the third position and the first position may be the same position. Further, the third position may be between the first position and the second position. In that case, the amount of shift in the lateral position of the right-eye image and the left-eye image is adjusted depending on whether the slider is moved from the third position to the first position or in the second direction. The direction to do is reversed.

  The 3D indicator 26 indicates whether or not the upper LCD 22 is in the stereoscopic display mode. The 3D indicator 26 is an LED, and lights up when the stereoscopic display mode of the upper LCD 22 is valid. The 3D indicator 26 is displayed when the upper LCD 22 is in the stereoscopic display mode and a program process for displaying a stereoscopic image is being executed (that is, the 3D adjustment switch is moved from the first position to the second position). (When image processing is performed such that the left-eye image and the right-eye image are different from each other), the light may be turned on. As shown in FIG. 1, the 3D indicator 26 is provided on the inner surface of the upper housing 21 and is provided near the screen of the upper LCD 22. For this reason, when the user views the screen of the upper LCD 22 from the front, the user can easily view the 3D indicator 26. Therefore, the user can easily recognize the display mode of the upper LCD 22 even when the user is viewing the screen of the upper LCD 22.

  A speaker hole 21 </ b> E is provided on the inner surface of the upper housing 21. Sound from a speaker 43 described later is output from the speaker hole 21E.

(Internal configuration of game device 10)
Next, with reference to FIG. 6, an internal electrical configuration of the game apparatus 10 will be described. FIG. 6 is a block diagram showing an internal configuration of the game apparatus 10. As shown in FIG. 6, in addition to the above-described units, the game apparatus 10 includes an information processing unit 31, a main memory 32, an external memory interface (external memory I / F) 33, an external memory I / F 34 for data storage, data It includes electronic components such as an internal memory 35 for storage, a wireless communication module 36, a local communication module 37, a real time clock (RTC) 38, an acceleration sensor 39, a power supply circuit 40, and an interface circuit (I / F circuit) 41. These electronic components are mounted on an electronic circuit board and accommodated in the lower housing 11 (or the upper housing 21).

The information processing unit 31 is information processing means including a CPU (Central Processing Unit) 311 for executing a predetermined program, a GPU (Graphics Processing Unit) 312 for performing image processing, and the like. The CPU 311 of the information processing unit 31 executes the program stored in a memory (for example, the external memory 44 connected to the external memory I / F 33 or the data storage internal memory 35) in the game apparatus 10, thereby executing the program. Processing (for example, shooting processing or image display processing described later) according to the above is executed. Note that the program executed by the CPU 311 of the information processing unit 31 may be acquired from another device through communication with the other device. The information processing unit 31 includes a VRAM (Video RAM) 313. The GPU 312 of the information processing unit 31 generates an image in response to a command from the CPU 311 of the information processing unit 31 and draws it on the VRAM 313. Then, the GPU 312 of the information processing unit 31 outputs the image drawn in the VRAM 313 to the upper LCD 22 and / or the lower LCD 12, and the image is displayed on the upper LCD 22 and / or the lower LCD 12.

  A main memory 32, an external memory I / F 33, a data storage external memory I / F 34, and a data storage internal memory 35 are connected to the information processing section 31. The external memory I / F 33 is an interface for detachably connecting the external memory 44. The data storage external memory I / F 34 is an interface for detachably connecting the data storage external memory 45.

  The main memory 32 is a volatile storage unit used as a work area or a buffer area of the information processing unit 31 (the CPU 311). That is, the main memory 32 temporarily stores various data used for the processing based on the program, or temporarily stores a program acquired from the outside (such as the external memory 44 or another device). In the present embodiment, for example, a PSRAM (Pseudo-SRAM) is used as the main memory 32.

  The external memory 44 is a nonvolatile storage unit for storing a program executed by the information processing unit 31. The external memory 44 is composed of, for example, a read-only semiconductor memory. When the external memory 44 is connected to the external memory I / F 33, the information processing section 31 can read a program stored in the external memory 44. A predetermined process is performed by executing the program read by the information processing unit 31. The data storage external memory 45 is composed of a non-volatile readable / writable memory (for example, a NAND flash memory), and is used for storing predetermined data. For example, the data storage external memory 45 stores an image captured by the outer imaging unit 23 or an image captured by another device. When the data storage external memory 45 is connected to the data storage external memory I / F 34, the information processing section 31 reads an image stored in the data storage external memory 45 and applies the image to the upper LCD 22 and / or the lower LCD 12. An image can be displayed.

  The data storage internal memory 35 is configured by a readable / writable nonvolatile memory (for example, a NAND flash memory), and is used for storing predetermined data. For example, the data storage internal memory 35 stores data and programs downloaded by wireless communication via the wireless communication module 36.

  The wireless communication module 36 is, for example, IEEE 802.11. It has a function of connecting to a wireless LAN by a method compliant with the b / g standard. Further, the local communication module 37 has a function of performing wireless communication with the same type of game device by a predetermined communication method (for example, communication using a unique protocol or infrared communication). The wireless communication module 36 and the local communication module 37 are connected to the information processing unit 31. The information processing unit 31 transmits / receives data to / from other devices via the Internet using the wireless communication module 36, and transmits / receives data to / from other game devices of the same type using the local communication module 37. You can do it.

  An acceleration sensor 39 is connected to the information processing unit 31. The acceleration sensor 39 detects the magnitude of linear acceleration (linear acceleration) along the three-axis (xyz-axis) direction. The acceleration sensor 39 is provided inside the lower housing 11. As shown in FIG. 1, the acceleration sensor 39 is configured such that the long side direction of the lower housing 11 is the x axis, the short side direction of the lower housing 11 is the y axis, and the inner side surface (main surface) of the lower housing 11. With the vertical direction as the z axis, the magnitude of linear acceleration on each axis is detected. The acceleration sensor 39 is, for example, an electrostatic capacitance type acceleration sensor, but other types of acceleration sensors may be used. The acceleration sensor 39 may be an acceleration sensor that detects a uniaxial or biaxial direction. The information processing unit 31 can detect data indicating the acceleration detected by the acceleration sensor 39 (acceleration data) and detect the attitude and movement of the game apparatus 10.

  Further, the RTC 38 and the power supply circuit 40 are connected to the information processing unit 31. The RTC 38 counts the time and outputs it to the information processing unit 31. The information processing unit 31 calculates the current time (date) based on the time counted by the RTC 38. The power supply circuit 40 controls power from a power source (the rechargeable battery housed in the lower housing 11) of the game apparatus 10 and supplies power to each component of the game apparatus 10.

  In addition, an I / F circuit 41 is connected to the information processing unit 31. A microphone 42 and a speaker 43 are connected to the I / F circuit 41. Specifically, a speaker 43 is connected to the I / F circuit 41 via an amplifier (not shown). The microphone 42 detects the user's voice and outputs a voice signal to the I / F circuit 41. The amplifier amplifies the audio signal from the I / F circuit 41 and outputs the audio from the speaker 43. The touch panel 13 is connected to the I / F circuit 41. The I / F circuit 41 includes a voice control circuit that controls the microphone 42 and the speaker 43 (amplifier), and a touch panel control circuit that controls the touch panel. The voice control circuit performs A / D conversion and D / A conversion on the voice signal, or converts the voice signal into voice data of a predetermined format. The touch panel control circuit generates touch position data in a predetermined format based on a signal from the touch panel 13 and outputs it to the information processing unit 31. The touch position data indicates the coordinates of the position where the input is performed on the input surface of the touch panel 13. The touch panel control circuit reads signals from the touch panel 13 and generates touch position data at a rate of once per predetermined time. The information processing unit 31 can know the position where the input is performed on the touch panel 13 by acquiring the touch position data.

  The operation button 14 includes the operation buttons 14 </ b> A to 14 </ b> L and is connected to the information processing unit 31. From the operation button 14 to the information processing section 31, operation data indicating the input status (whether or not the button is pressed) for each of the operation buttons 14A to 14I is output. The information processing unit 31 acquires the operation data from the operation button 14 to execute processing according to the input to the operation button 14.

  The lower LCD 12 and the upper LCD 22 are connected to the information processing unit 31. The lower LCD 12 and the upper LCD 22 display images according to instructions from the information processing unit 31 (the GPU 312). In the present embodiment, the information processing unit 31 causes the upper LCD 22 to display a stereoscopic image (a stereoscopically viewable image).

  Specifically, the information processing section 31 is connected to an LCD controller (not shown) of the upper LCD 22 and controls ON / OFF of the parallax barrier for the LCD controller. When the parallax barrier of the upper LCD 22 is ON, the right-eye image and the left-eye image stored in the VRAM 313 of the information processing unit 31 are output to the upper LCD 22. More specifically, the LCD controller alternately repeats the process of reading pixel data for one line in the vertical direction for the image for the right eye and the process of reading pixel data for one line in the vertical direction for the image for the left eye. Thus, the right-eye image and the left-eye image are read from the VRAM 313. As a result, the image for the right eye and the image for the left eye are divided into strip-like images in which pixels are arranged vertically for each line. The strip-like images for the right-eye image and the strip-like images for the left-eye image are alternately displayed. The image arranged on the upper LCD 22 is displayed on the screen. Then, when the user visually recognizes the image through the parallax barrier of the upper LCD 22, the right eye image is visually recognized by the user's right eye and the left eye image is visually recognized by the user's left eye. As a result, a stereoscopically viewable image is displayed on the screen of the upper LCD 22.

  The outer imaging unit 23 and the inner imaging unit 24 are connected to the information processing unit 31. The outer imaging unit 23 and the inner imaging unit 24 capture an image in accordance with an instruction from the information processing unit 31, and output the captured image data to the information processing unit 31.

  The 3D adjustment switch 25 is connected to the information processing unit 31. The 3D adjustment switch 25 transmits an electrical signal corresponding to the position of the slider 25 a to the information processing unit 31.

  The 3D indicator 26 is connected to the information processing unit 31. The information processing unit 31 controls lighting of the 3D indicator 26. For example, the information processing section 31 turns on the 3D indicator 26 when the upper LCD 22 is in the stereoscopic display mode. The above is the description of the internal configuration of the game apparatus 10.

(Outline of operation of game device 10)
Hereinafter, an outline of the operation of the game apparatus 10 in the present embodiment will be described. In the present embodiment, based on the image display program, a real-world image currently captured by the outer imaging unit 23 (outer imaging unit (left) 23a, outer imaging unit (right) 23b) and a three-dimensional virtual space. Is displayed on the screen of the upper LCD 22 so as to be stereoscopically viewable.

(Stereoscopic display of real world images)
The two captured images captured by the outer imaging unit 23 are supplied to the upper LCD so as to have a predetermined parallax and are displayed in a stereoscopic view.

  FIG. 7 shows an example of a stereoscopic image displayed on the screen of the upper LCD 22 when three balls 60 a to 60 c (real objects) are imaged by the outer imaging unit 23. When three balls 60a to 60c are imaged by the outer imaging unit 23, these balls 60a to 60c are displayed on the upper LCD 22 so as to be stereoscopically viewed. As shown in FIG. 7, on the screen of the upper LCD 22, the ball 60 a closest to the outer imaging unit 23 is visually recognized by the user so that the ball 60 c farthest from the outer imaging unit 23 is farthest away. It is visually recognized by the user so as to be positioned. In FIG. 7, for ease of understanding, the ball protrudes from the screen of the upper LCD 22, but the ball is actually displayed in the screen. The same applies to FIGS. 8 and 10 described later. In FIG. 7, since a marker 61 (see FIG. 8) described later is not captured by the outer imaging unit 23, no virtual object is displayed on the screen of the upper LCD 22. 7, 8, 10, and the like illustrate how the display target is visually recognized as popping out, but when it is displayed in a stereoscopic view, it is not always visible in the popping out direction. This includes not only the case where the image is displayed but also the case where the image is displayed with a depth behind the screen.

(Stereoscopic display of real world images and CG images)
FIG. 8 illustrates an example of a stereoscopic image displayed on the screen of the upper LCD 22 when the marker 61 (real object) is captured by the outer imaging unit 23. As shown in FIG. 8, a square including an arrow is drawn on the marker 61, and the CPU 311 performs, for example, image processing such as pattern matching on the image acquired from the outer imaging unit 23. It can be determined whether or not a marker is included in the image. When the marker 61 is imaged by the outer imaging unit 23, the upper LCD 22 displays the marker 61 as a real world image so as to be stereoscopically viewed, and a virtual object 62 (for example, a virtual model imitating a dog) at the position of the marker 61. Object) is displayed in a composite manner so as to be stereoscopically viewed. As shown in FIG. 9, the marker 61 has a direction (forward direction, right direction, upward direction) defined, and a virtual object can be arranged in a posture corresponding to the posture of the marker 61. For example, the virtual object 62 can be arranged on the marker 61 so that the front direction of the virtual object 62 matches the front direction of the marker 61.

FIG. 10 shows the upper LCD 2 when the marker 61 is imaged by the outer imaging unit 23.
7 shows another example of a stereoscopic image displayed on the second screen. As described above, when the user moves the game apparatus 10 and the position and orientation of the marker 61 displayed on the screen of the upper LCD 22 change, the position and orientation of the virtual object 62 change to follow the change. Therefore, it seems to the user that the virtual object 62 really exists in the real world.

  Hereinafter, with reference to FIG. 11 to FIG. 40, the details of the image display process executed in the game apparatus 10 based on the image display program will be described.

(Memory map)
First, main data stored in the main memory 32 during the image display process will be described. FIG. 11 is a diagram showing a memory map of the main memory 32 of the game apparatus 10. As shown in FIG. 11, in the main memory 32, the image display program 70, the latest left real world image 71L, the latest right real world image 71R, the display left real world image 72L, the display right real world image 72R, the stereoscopic image Visual zero distance 73, virtual object information 74, left transformation matrix 75L, right transformation matrix 75R, left view matrix 76L, right view matrix 76R, left projection matrix 77L, right projection matrix 77R, display mode 78, marker recognition mode 79, main Real world image identification information 80, various variables 81, and the like are stored.

  The image display program 70 is a program for causing the CPU 311 to execute the image display process.

  The latest left real world image 71L is the latest image captured by the outer imaging unit (left) 23a.

  The latest right real world image 71R is the latest image captured by the outer imaging unit (right) 23b.

  The display left real world image 72L is the latest image determined to be displayed on the upper LCD 22 among images captured by the outer imaging unit (left) 23a.

  The display right real world image 72R is the latest image determined to be displayed on the upper LCD 22 among the images captured by the outer imaging unit (right) 23b.

  The stereoscopic zero distance 73 is visually recognized so that an object separated from the outer imaging unit 23 in the shooting direction is in the same depth position as the screen of the upper LCD 22 when stereoscopically displayed on the screen of the upper LCD 22. For example, expressed in units of centimeters. In this embodiment, the stereoscopic vision zero distance 73 is fixed at 25 cm, but this is merely an example, and the stereoscopic vision zero distance 73 is changed as needed according to a user instruction or automatically by a computer. It doesn't matter. The stereoscopic zero distance 73 may be specified by a distance from the marker 61 or a ratio in a distance between the outer imaging unit 23 and the marker 61. Further, it is not necessarily specified as the distance in the real space, but may be specified by the distance in the virtual space. When the size of the marker 61 is known, the unit of length in the real world and the unit of length in the virtual space can be matched. Thus, when both units can be made to correspond, the stereoscopic zero distance 73 can be set in the unit of length in the real world. However, if the units of both cannot be matched, the stereoscopic vision zero distance 73 may be set in units of length in the virtual space. As will be described later, in the present embodiment, the unit of length in the real world and the unit of length in the virtual space can be matched without using the size of the marker 61.

  The virtual object information 74 is information related to the virtual object 62 described above, and includes 3D model data (polygon data) representing the shape of the virtual object 62, texture data representing the pattern of the virtual object 62, and virtual data in the virtual space. Information on the position and orientation of the object 62 is included.

  The left conversion matrix 75L is a matrix calculated by recognizing the position and orientation of the marker 61 in the left real world image, and is expressed in a coordinate system (marker coordinate system) set with the position and orientation of the marker 61 as a reference. This is a coordinate transformation matrix for transforming the coordinates into a coordinate system (outer imaging unit (left) coordinate system) expressed with reference to the position and orientation of the outer imaging unit (left) 23a. The left transformation matrix 75L is a matrix including information on the relative position and orientation of the outer imaging unit (left) 23a with respect to the position and orientation of the marker 61, and more specifically, the outer imaging unit (left ) 23a is a matrix including the position and orientation information.

  The right transformation matrix 75R is a matrix calculated by recognizing the position and orientation of the marker 61 in the right real world image, and the coordinates expressed in the marker coordinate system are used as the position and orientation of the outer imaging unit (right) 23b. This is a coordinate transformation matrix for transformation into a coordinate system (outer imaging unit (right) coordinate system) represented as a reference. The right transformation matrix 75R is a matrix that includes information on the relative position and orientation of the outer imaging unit (right) 23b with respect to the position and orientation of the marker 61, and more specifically, the outer imaging unit (right ) 23b is a matrix including the position and orientation information.

  In this specification, a conversion matrix from the marker coordinate system to the outer imaging unit (left) coordinate system or the outer imaging unit (right) coordinate system is referred to as a “marker / camera conversion matrix”. The left conversion matrix 75L and the right conversion matrix 75R are “marker-camera conversion matrices”.

  The left view matrix 76L is a matrix used when the virtual object 62 viewed from the left virtual camera is drawn, and the coordinates expressed in the world coordinate system of the virtual world are changed to the coordinates expressed in the left virtual camera coordinate system. It is a coordinate transformation matrix for transformation. The left view matrix 76L is a matrix including information on the position and orientation of the left virtual camera in the world coordinate system of the virtual world.

  The right view matrix 76R is a matrix used when the virtual object 62 viewed from the right virtual camera is drawn, and the coordinates expressed in the world coordinate system of the virtual world are changed to the coordinates expressed in the right virtual camera coordinate system. It is a coordinate transformation matrix for transformation. The right view matrix 76R is a matrix including information on the position and orientation of the right virtual camera in the world coordinate system of the virtual world.

  The left projection matrix 77L is a matrix used when drawing the virtual world viewed from the left virtual camera (the virtual object 62 existing in the virtual world), and the coordinates represented by the left virtual camera coordinate system are expressed in the screen coordinate system. It is a coordinate transformation matrix for transforming into expressed coordinates.

  The right projection matrix 77R is a matrix used when drawing the virtual world (virtual object 62 existing in the virtual world) viewed from the right virtual camera, and the coordinates expressed in the right virtual camera coordinate system are expressed in the screen coordinate system. It is a coordinate transformation matrix for transforming into expressed coordinates.

  The display mode 78 is data indicating the current processing mode. More specifically, the display mode 78 is a synchronous display mode in which the real world image and the virtual space image are combined and displayed in synchronization, or the real world image and the virtual space image are displayed. This is data indicating whether the display mode is an asynchronous display mode in which the images are combined and displayed without being synchronized.

The marker recognition mode 79 is data indicating the current processing mode. More specifically, the marker recognition mode 79 is a one-image recognition mode in which marker recognition processing is performed on only one of the left real world image and the right real world image. This is data indicating both image recognition modes in which marker recognition processing is performed for both the left real world image and the right real world image.

  The main real world image identification information 80 is data indicating which of the left real world image and the right real world image is the main real world image. In the one-image recognition mode, the marker recognition process is performed only on the main real world image, and the marker recognition process is not performed on the other real world image (hereinafter referred to as a sub real world image).

  Various variables 81 are variables used when the image display program 70 is executed, and include variables as shown in FIG. The meaning of these variables will be explained as needed in the following description.

  When the power of the game apparatus 10 is turned on, the information processing section 31 (CPU 311) of the game apparatus 10 executes a startup program stored in a ROM (not shown), thereby initializing each unit such as the main memory 32. Is done. Next, the image display program stored in the internal data storage memory 35 is read into the main memory 32, and the CPU 311 of the information processing unit 31 starts executing the image display program.

  Hereinafter, the flow of processing executed based on the image display program will be described with reference to the flowcharts of FIGS. Note that the flowcharts of FIGS. 13 to 19 are merely examples. Therefore, if the same result is obtained, the processing order of each step may be changed. Moreover, the value of the variable and the threshold value used in the determination step are merely examples, and other values may be adopted as necessary. In the present embodiment, the processing of all steps in the flowcharts of FIGS. 13 to 19 is described as being executed by the CPU 311. However, the processing of some steps in the flowcharts of FIGS. It may be executed by a processor or a dedicated circuit.

(Marker processing)
FIG. 13 is a flowchart showing the flow of marker processing executed by the CPU 311 based on the image display program 70. The marker process is a process executed in parallel with a main process to be described later. In the present embodiment, the marker process is executed when the CPU 311 is in an idle state. Details of the marker process will be described below.

  In step S10 of FIG. 13, the CPU 311 determines whether or not both real world images (that is, the left real world image and the right real world image) have been acquired from the outer imaging unit 23, and both real world images are acquired. If yes, the process proceeds to step S11. The latest left real world image acquired from the outer imaging unit 23 is stored in the main memory 32 as the latest left real world image 71L, and the latest right real world image is stored in the main memory 32 as the latest right real world image 71R. Is done.

  As described above, in the upper housing 21, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are separated by a certain distance (for example, 3.5 cm, hereinafter, distance between imaging units). Therefore, when the marker 61 is imaged simultaneously by the outer imaging unit (left) 23a and the outer imaging unit (right) 23b, as shown in FIG. 20, the marker in the left real world image captured by the outer imaging unit (left) 23a. There is a shift due to parallax between the position and orientation of 61 and the position and orientation of the marker 61 in the right real world image captured by the outer imaging unit (right) 23b.

In step S11, the CPU 311 performs marker recognition processing on the main real world image. More specifically, it is determined whether or not the marker 61 is included in the main real world image by a pattern matching method or the like. If the marker 61 is included in the main real world image, Based on the position and orientation of the marker 61, the left conversion matrix 75L or the right conversion matrix 75R is calculated (if the main real world image is a left real world image, the left conversion matrix 75L is calculated and the main real world image is calculated). Is the right real world image, the right transformation matrix 75R is calculated).

  The left conversion matrix 75L is a matrix reflecting the position and orientation of the outer imaging unit (left) 23a calculated based on the position and orientation of the marker 61 in the left real world image. More precisely, as shown in FIG. 21, a marker coordinate system (a coordinate system in which the position of the marker 61 in the real world is the origin and the vertical direction, the horizontal direction, and the normal direction of the marker 61 are axes) Are expressed in the outer imaging unit (left) coordinate system based on the position and orientation of the outer imaging unit (left) 23a calculated based on the position and orientation of the marker 61 in the left real world image. This is a coordinate transformation matrix for transforming into coordinates.

  The right transformation matrix 75R is a matrix reflecting the position and orientation of the outer imaging unit (right) 23b calculated based on the position and orientation of the marker 61 in the right real world image. More precisely, as shown in FIG. 22, the position and orientation of the outer imaging unit (right) 23b calculated based on the coordinates and the coordinates expressed in the marker coordinate system based on the position and orientation of the marker 61 in the right real world image. Is a coordinate transformation matrix for transforming into coordinates expressed in the outer imaging unit (right) coordinate system with reference to.

  Assuming that there is no error in marker recognition accuracy and that there is no error in the accuracy of attaching the outer imaging unit (left) 23a and the outer imaging unit (right) 23b to the game apparatus 10, marker recognition of the right real world image is performed. The position of the outer imaging unit (right) 23b indicated by the right transformation matrix 75R as the result is the position of the outer imaging unit (left) 23a indicated by the left transformation matrix 75L as the marker recognition result of the left real world image. (Left) The outer imaging unit indicated by the right transformation matrix 75R is positioned at a certain distance (distance between imaging units) along the x-axis direction of the coordinate system (the horizontal direction of the game apparatus 10 and the horizontal direction when used). (Right) The posture of 23b and the posture of the outer imaging unit (left) 23a indicated by the left transformation matrix 75L are the same as the attached state of the outer imaging unit (left) 23a and the outer imaging unit (right) 23b in the game apparatus 10. To become. In the present embodiment, since the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are designed to be mounted in parallel in the game apparatus 10, the x axis, y of the outer imaging unit (left) coordinate system is designed. The axes and the z-axis are parallel to the x-axis, y-axis, and z-axis of the outer imaging unit (right) coordinate system, respectively. However, actually, since there is an error in the marker recognition accuracy and the accuracy in attaching the outer imaging unit (left) 23a and the outer imaging unit (right) 23b to the game apparatus 10, the outer imaging unit (shown by the left conversion matrix 75L) The position and orientation of the left) 23a and the position and orientation of the outer imaging unit (right) 23b indicated by the right transformation matrix 75R are not in an ideal relationship. For example, the left transformation matrix 75L and the right transformation matrix 75R indicate that the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are too close or too far apart, the attitude of the outer imaging unit (left) 23a and the outer imaging unit. (Right) The matrix is such that the posture of 23b is different. Further, the matrix is such that the attitude of the outer imaging unit (left) 23a and the attitude of the outer imaging unit (right) 23b are not parallel.

Here, in AR, a CG image is converted to a real world image by designating a matrix for converting the marker coordinate system to the outer imaging unit (left) coordinate system or the outer imaging unit (right) coordinate system in the view matrix of the virtual camera. However, when the left transformation matrix 75L is designated as the left view matrix 76L and the right transformation matrix 75R is designated as the right view matrix 76R and the virtual space image is stereoscopically displayed on the upper LCD 22, the virtual transformation is possible. The object 62 may not be normally stereoscopically viewed. Therefore, in the present embodiment, as will be apparent from the following description, the position of one virtual camera calculated from the marker recognition result (marker / camera conversion matrix) of one of the left real world image and the right real world image And the position and orientation (view matrix) of the other virtual camera so that the position and orientation of the left virtual camera 63L and the position and orientation of the right virtual camera 63R have an ideal relationship based on the orientation and view (view matrix). It is determined.

  In step S11, if the marker 61 is not recognized from the main real world image, a null value is stored in the left conversion matrix 75L or the right conversion matrix 75R, whereby the left real world image or the right real world image is stored. It is recorded that the recognition of the marker 61 is failed.

  In step S12, the CPU 311 determines whether or not the marker recognition mode is the one-image recognition mode. If it is the one-image recognition mode, the process proceeds to step S13. If not (that is, if it is the both-image recognition mode), the process proceeds to step S14.

  In step S13, the CPU 311 regards the marker recognition result of the real world image that is not the main real world image (hereinafter referred to as the sub real world image) as the “failure” among the left real world image and the right real world image. . More specifically, if the main real world image is a left real world image, a null value is stored in the right conversion matrix 75R, and if the main real world image is a right real world image, the left conversion matrix A null value is stored in 75L.

  In step S14, the CPU 311 performs marker recognition processing on the sub real world image. More specifically, it is determined whether or not the marker 61 is included in the sub real world image by a pattern matching method or the like, and when the marker 61 is included in the sub real world image, Based on the position and orientation of the marker 61, the left conversion matrix 75L or the right conversion matrix 75R is calculated (if the sub real world image is a left real world image, the left conversion matrix 75L is calculated and the sub real world image is calculated). Is the right real world image, the right transformation matrix 75R is calculated).

  In step S15, the CPU 311 determines whether the display mode is the synchronous display mode. If the mode is the synchronous display mode, the process proceeds to step S17. If not (that is, the mode is the asynchronous display mode), the process proceeds to step S16.

  In step S16, the CPU 311 stores the latest left real world image 71L and the latest right real world image 71R in the main memory 32 as a display left real world image 72L and a display right real world image 72R, respectively. Then, the process returns to step S10.

  In step S17, the CPU 311 determines whether the marker recognition mode is the one-image recognition mode. If it is the single image recognition mode, the process proceeds to step S18. If not (that is, if it is the both image recognition mode), the process proceeds to step S19.

  In step S <b> 18, the CPU 311 determines whether marker recognition of the main real world image has succeeded. More specifically, when the main real world image is the left real world image, the CPU 311 determines whether a valid matrix that is not a null value is stored in the left conversion matrix 75L, and the main real world image Is a right real world image, it is determined whether or not a valid matrix that is not a null value is stored in the right transformation matrix 75R. If it is determined that marker recognition of the main real world image is successful, the process proceeds to step S16, and if not, the process returns to step S10.

In step S19, the CPU 311 determines whether or not the marker recognition of at least one of the left real world image and the right real world image is successful. More specifically, the CPU 311 determines whether valid matrices that are not null values are stored in both the left conversion matrix 75L and the right conversion matrix 75R. If it is determined that at least one of the left real world image and the right real world image has been successfully recognized, the process proceeds to step S16. If not, the process returns to step S10.

(Main process)
FIG. 14 is a flowchart showing a flow of main processing executed by the CPU 311 based on the image display program 70. The main process is a process executed in parallel with the marker process described above. Hereinafter, details of the main process will be described.

  In step S20 of FIG. 14, the CPU 311 arranges the virtual object 62 at a predetermined position in the three-dimensional virtual space. In the present embodiment, the virtual object 62 is arranged at the origin of the virtual space (the origin of the world coordinate system).

  In step S21, the CPU 311 performs an update process. This update process is a process for updating various variables used to draw the virtual object 62. Details of the update process will be described later.

  In step S22, the CPU 311 performs virtual object processing. The virtual object process is a process related to the virtual object 62 arranged in the virtual space. For example, the size of the virtual object 62 is changed or a predetermined action (in the virtual space is performed in the virtual object 62) as necessary. For example, if a movement is made to walk around the origin of the virtual space, a display is made to walk around the origin of the marker coordinate system). The movement control of the virtual object 62 is realized by changing the position coordinates of the virtual object in the world coordinate system of the virtual space.

  In step S <b> 23, the CPU 311 determines a drawing range which is a range used for display in the left real world image according to the stereoscopic zero distance 73. More specifically, as shown in FIG. 23, the outer imaging unit 23 is separated from the outer imaging unit 23 by a value of a stereoscopic vision zero distance 73 (for example, 30 cm) in the imaging direction, and the outer imaging unit (left) 23a and the outer imaging unit ( (Right) A point equidistant from 23b is set as a stereoscopic zero point, and a range in which the stereoscopic zero point is centered when viewed from the outer imaging unit (left) 23a is determined as a drawing range of the left real world image. . For example, as shown in FIG. 23, the horizontal length of the range is obtained by cutting a straight line perpendicular to the shooting direction of the outer imaging unit (left) 23a with a straight line representing the angle of view of the outer imaging unit (left) 23a. A ratio on the line segment can be obtained, and the ratio is determined as a range on the real world image by making the ratio correspond to the horizontal length of the real world image. The vertical length of the range is determined from the horizontal length so as to match the aspect ratio of the display screen.

  When determining the drawing range of the right real world image in step S27 described later, as shown in FIG. 24, a range in which the stereoscopic zero point is centered when viewed from the outer imaging unit (right) 23b, The drawing range of the right real world image is determined. As a result, the position of the stereoscopic zero point in the left-eye image displayed on the upper LCD 22 and the position of the stereoscopic zero point in the right-eye image match on the screen of the upper LCD 22, and the screen is viewed from the user. It seems to exist at the same depth position.

  In step S24, the CPU 311 stores a predetermined storage area in the VRAM 313 for temporarily storing the left eye image to be supplied to the upper LCD 22 from the drawing range determined in step S23 in the display left real world image 72L. Hereinafter, drawing is performed on the left frame buffer.

  In step S25, the CPU 311 determines whether the value of ARAActive (see FIG. 12) is true. ARAActive is a variable (flag) indicating whether or not a virtual object can be displayed in a synthesized manner on a real world image. In a situation where a virtual object can be synthesized and displayed on a real world image, its value is true. In a situation where a virtual object cannot be synthesized and displayed on the real world image (for example, a situation where the marker 61 cannot be recognized at all), the value is set to false (initial value). If the ARAactive value is true, the process proceeds to step S26; otherwise (ie, the ARAactive value is false), the process proceeds to step S27.

  In step S26, for example, as shown in FIG. 23, the CPU 311 overwrites the left frame buffer with a virtual space viewed from the left virtual camera 63L (hereinafter referred to as a left-eye virtual space image) (in practice, typical Is rendered by the GPU 312 in accordance with an instruction from the CPU 311). As a result, as shown in FIG. 26, the left-eye virtual space image is synthesized with the left real world image drawn in the left frame buffer in step S24. The image drawn in the left frame buffer is supplied to the upper LCD 22 as a left-eye image at a predetermined timing. Note that in the left-eye virtual space image, the background of the virtual space is transparent. Therefore, by combining the virtual space image with the real world image, an image in which the virtual object 62 exists on the real world image. Is generated.

  In step S <b> 27, the CPU 311 determines the drawing range of the right real world image according to the stereoscopic vision zero distance 73. The details of the process for determining the drawing range are the same as the process for determining the drawing range of the left real world image, and are therefore omitted.

  In step S28, the CPU 311 stores a predetermined storage area in the VRAM 313 for temporarily storing the right eye image to be supplied to the upper LCD 22 from the drawing range determined in step S27 in the display right real world image 72R. The image is drawn in the right frame buffer hereinafter.

  In step S29, the CPU 311 determines whether or not the value of ARAActive is true. If the value of ARAactive is true, the process proceeds to step S30, and if not (that is, if the value of ARAactive is false), the process proceeds to step S31.

  In step S30, the CPU 311 overwrites the right frame buffer with a virtual space viewed from the right virtual camera 63R (hereinafter, referred to as a right-eye virtual space image) (in practice, typically, an instruction from the CPU 311 is used). Therefore, it is rendered by the GPU 312). Thereby, the virtual space image for the right eye is combined with the right real world image drawn in the right frame buffer in step S28. The image drawn in the right frame buffer is supplied to the upper LCD 22 as a right eye image at a predetermined timing.

  In step S31, the CPU 311 waits for an interrupt signal (vertical synchronization interrupt) from the upper LCD 22, and when the interrupt signal is generated, the process returns to step S21. Thereby, the process of step S21-S31 is repeated with a fixed period (for example, 1/60 second period).

(Update process)
Next, details of the update process of step S21 in the main process will be described with reference to the flowchart of FIG.

  In step S40 of FIG. 15, the CPU 311 determines whether or not the marker recognition result (that is, the left conversion matrix 75L and the right conversion matrix 75R) in the marker processing described above has been updated. The process proceeds to S41, and if it has not been updated yet, the process proceeds to step S44.

  In step S <b> 41, the CPU 311 determines whether the marker recognition of both the real world images of the left real world image and the right real world image is successful. If the marker recognition of both real world images has succeeded, the process proceeds to step S42. If the marker recognition of one or both real world images has failed, the process proceeds to step S42. Proceed to S44.

  In step S42, the CPU 311 determines whether the value of EyeWidth (see FIG. 12) is 0 or whether the value of EyeMeasure (see FIG. 12) is true. EyeWidth indicates the distance between the imaging units in the positional relationship between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b, which is calculated using the left transformation matrix 75L and the right transformation matrix 75R. . Here, the distance between the two imaging units does not need to be obtained as a real world scale, and may be obtained as a distance in a virtual space. Specifically, the relationship between the position of the marker 61 and the position of the outer imaging section (left) 23a indicated by the left conversion matrix 75L, the position of the marker 61 indicated by the right conversion matrix 75R, and the position of the outer imaging section (right) 23b. From the relationship with the position, the distance when the relationship between the position of the outer imaging unit (left) 23a and the position of the outer imaging unit (right) 23b is obtained with the position of the marker 61 as a reference. When the distance is not yet determined, the value is set to 0 (initial value). In the present embodiment, EyeWidth is a scalar value, but EyeWidth is calculated from the position of the outer imaging unit (left) 23a indicated by the left transformation matrix 75L and the position of the outer imaging unit (right) 23b indicated by the right transformation matrix 75R. You may make it the vector which connects between. Further, a component orthogonal to the shooting direction of a vector connecting EyeWidth to the position of the outer imaging section (left) 23a indicated by the left transformation matrix 75L and the position of the outer imaging section (right) 23b indicated by the right transformation matrix 75R. It may be of length. As will be described later, EyeWidth is used to set the distance (interval) between the left virtual camera 63L and the right virtual camera 63R in the virtual space. EyeMeasure is a variable (flag) indicating whether or not EyeWidth should be recalculated. In the situation where EyeWidth is to be recalculated, its value is set to true. In other cases, the value is false (initial value). Value). If the value of EyeWidth is 0 or the value of EyeMeasure is true, the process proceeds to step S43. Otherwise (ie, the value of EyeWidth is a value other than 0, and the value of EyeMeasure). Is false), the process proceeds to step S44.

  In step S43, the CPU 311 executes virtual camera interval determination processing. In the virtual camera interval determination process, the interval (that is, EyeWidth) between the left virtual camera 63L and the right virtual camera 63R is appropriately determined and updated. Details of the virtual camera interval determination processing will be described later.

  In step S44, the CPU 311 executes view matrix generation processing. In the view matrix generation process, the left view matrix 76L and the right view matrix 76R are calculated based on the already determined interval between the left virtual camera 63L and the right virtual camera 63R (that is, EyeWidth). Details of the view matrix generation processing will be described later.

In step S45, the CPU 311 executes main real world image setting processing. In the main real world image setting process, the main real world image is appropriately switched between the left real world image and the right real world image (this is the difference between the outer imaging unit (left) 23a and the outer imaging unit (right)). It is equivalent to switching the main imaging unit between them). Details of the main real-world image setting process will be described later.

  In step S46, the CPU 311 executes the display mode switching process and ends the update process. In the display mode switching process, the display mode is appropriately switched between the synchronous display mode and the asynchronous display mode. Details of the display mode switching process will be described later.

(Virtual camera interval determination process)
Next, details of the virtual camera interval determination process in step S43 in the update process (FIG. 15) will be described with reference to the flowchart of FIG. Here, as described above, the position of the outer imaging section (left) 23a with respect to the position of the marker 61 indicated by the left conversion matrix 75L and the position of the outer imaging section (right) 23b with respect to the position of the marker 61 indicated by the right conversion matrix 75R. Find the distance. There are several methods for calculating this distance. In this embodiment, the following method is adopted as an example of the calculation method.

  In step S50 of FIG. 16, the CPU 311 calculates the coordinate V0 based on the left conversion matrix 75L and the right conversion matrix 75R. Hereinafter, a method for calculating V0 will be described with reference to FIGS. 27 to 29, the calculation is based on the relative position and orientation of the outer imaging unit (left) 23a with respect to the marker 61, which is calculated based on the left real world image, and the right real world image. In order to emphasize that the relative position and relative orientation of the outer imaging unit (right) 23b with respect to the marker 61 do not necessarily have an ideal relationship as described above, the shooting direction and the outer side of the outer imaging unit (left) 23a are not limited. The imaging directions of the imaging unit (right) 23b are illustrated so as to be greatly different from each other.

First, as shown in FIG. 27, when (0, 0, 0) is multiplied by the left transformation matrix 75L, the coordinate V1 of the origin of the marker coordinate system expressed in the outer imaging unit (left) coordinate system is obtained. . The coordinate V1 represents the relative position of the marker 61 with respect to the outer imaging unit (left) 23a calculated based on the left real world image. Assuming that the coordinate V1 is a coordinate expressed in the outer imaging unit (right) coordinate system, the position indicated by the coordinate V1 moves to the position shown in FIG. 28 in the outer imaging unit (right) coordinate system. The coordinate V1 is further multiplied by the inverse matrix of the right transformation matrix 75R. Multiplying the inverse matrix of the right transformation matrix 75R corresponds to transforming coordinates expressed in the outer imaging unit (right) coordinate system to coordinates expressed in the marker coordinate system. Accordingly, the multiplication of the inverse matrix converts the coordinate V1 (FIG. 28) expressed in the outer imaging unit (right) coordinate system to the coordinate V0 expressed in the marker coordinate system as shown in FIG. become. The coordinates V0 calculated in this way are expressed in the relative position of the marker 61 with respect to the outer imaging section (left) 23a calculated based on the left real world image (that is, the outer imaging section (left) coordinate system). The coordinate of the origin of the marker coordinate system) and the relative position of the marker 61 with respect to the outer imaging unit (right) 23b calculated based on the right real world image (that is, the marker represented by the outer imaging unit (right) coordinate system). The difference from the coordinate of the origin of the coordinate system is shown. In the present embodiment, the difference in the relative position of the marker 61 is considered to be caused by the difference in the attachment position of the outer imaging unit (left) 23a and the outer imaging unit (right) 23b, and the outer imaging unit (left). 23a and the outer imaging unit (right) 23b are estimated to be attached.

  In step S51, the CPU 311 calculates the coordinate V1 of the origin of the marker coordinate system expressed in the outer imaging unit (left) coordinate system based on the left conversion matrix 75L. Specifically, by multiplying (0, 0, 0) by the left transformation matrix 75L, the coordinate V1 of the origin of the marker coordinate system expressed in the outer imaging unit (left) coordinate system is obtained. Here, the coordinate of the origin of the marker coordinate system expressed in the outer imaging unit (left) coordinate system is V1, but instead, the marker coordinate system expressed in the outer imaging unit (right) coordinate system. The origin coordinate may be V1.

In step S52, the CPU 311 stores the magnitude of V0 (distance from the origin) (see FIG. 29) obtained in step S50 in ew (see FIG. 12), and the z-axis value of V1 obtained in step S51. The absolute value (see FIG. 27) is stored in ed (see FIG. 12). However, the value of ew is calculated in the unit of length in the marker coordinate system, and does not coincide with the unit of length in the real world. As described above, since the actual distance between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b is known (for example, 3.5 cm), the actual distance and the value of ew are used. The unit of length in the world and the unit of length in virtual space can be matched. If the size of the marker 61 is known, the unit of length in the real world and the virtual one can be calculated from the correspondence between the size of the marker 61 obtained based on the recognition result of the marker image and the size of the marker 61 in the real world. It is also possible to match the unit of length in space.

  Note that due to an error in the accuracy of attaching the outer imaging unit (left) 23a and the outer imaging unit (right) 23b to the game apparatus 10, the shooting direction of the outer imaging unit (left) 23a and the shooting direction of the outer imaging unit (right) 23b are different. When not parallel, the distance (ew) between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b calculated in this way is between the outer imaging unit 23 and the marker 61. Fluctuates according to the distance in the shooting direction. Therefore, as described later, when the distance in the shooting direction between the outer imaging unit 23 and the marker 61 changes, the distance between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b (described later). EyeWidth) is recalculated using the marker-camera conversion matrix at that time.

In this embodiment, ew is set to the magnitude of V0. However, the position of the outer imaging section (left) 23a indicated by the left transformation matrix 75L and the outer imaging section (right) 23b indicated by the right transformation matrix 75R of V0. It may be a component of “direction connecting positions of”.

  The ew calculated in this way is the “interval in the marker coordinate system between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b”, which is the left virtual camera 63R and the right virtual camera 63R. Used as an interval between (step S65 or S68 described later).

  Note that there are other methods for obtaining ew than the method described above. For example, by multiplying (0, 0, 0) by the left transformation matrix 75L, the coordinates Vl of the origin of the marker coordinate system expressed in the outer imaging unit (left) coordinate system (that is, the outer imaging unit (left) 23a). Relative position of the marker 61 with respect to (0), and by multiplying (0, 0, 0) by the right transformation matrix 75R, the coordinate Vr of the origin of the marker coordinate system represented by the outer imaging unit (right) (ie, Ew may be obtained by obtaining the relative position of the marker 61 with respect to the outer imaging section (right) 23b and calculating the distance between the coordinates Vl and the coordinates Vr thus obtained.

  In step S53, the CPU 311 determines whether or not the value of LogIndex (see FIG. 12) is larger than 1 (initial value). If it is larger than 1, the process proceeds to step S57, and if not (that is, 1). In the case), the process proceeds to step S54. LogIndex is a variable for identifying each element of an array (LogWidth, LogDepth) described later.

  In step S54, the CPU 311 stores the value of ew obtained in step S52 in the array LogWidth [LogIndex]. Further, the value of ed obtained in step S52 is stored in the array LogDepth [LogIndex]. Further, the value of LogIndex is incremented. LogWidth is an array variable for holding a plurality of ew values. Log Depth is an array variable for holding a plurality of ed values.

In step S55, the CPU 311 determines whether or not the value of LogIndex is larger than the value of LogMax. If it is larger, the process proceeds to step S56, and if not (that is, the value of LogIndex is equal to or smaller than the value of LogMax). The virtual camera interval determination process ends.

  In step S56, the CPU 311 calculates the average value of each element of LogWidth, and stores the average value in EyeWidth. Further, the average value of each element of Log Depth is calculated, and the average value is stored in Eye Depth. Further, the value of EyeMeasure is set to false. Further, the marker recognition mode 79 is set to the one-image recognition mode. Eye Depth is a coordinate indicating the position in the marker coordinate system of the outer imaging unit (left) 23a indicated by the left transformation matrix 75L (or a coordinate indicating the position in the marker coordinate system of the outer imaging unit (right) 23b indicated by the right transformation matrix 75R. ) To the origin of the marker coordinate system (depth: distance in the photographing direction), and is used as a reference value for determination in step S72 of FIG. The initial value of EyeDepth is 0. When the process of step S56 ends, the virtual camera interval determination process ends.

  In step S57, the CPU 311 determines that the absolute value of the difference between the ew value obtained in step S52 and the LogWidth [1] value is smaller than 10% of the LogWidth [1] value, and the ed obtained in step S52. It is determined whether the absolute value of the difference between the value of LogDepth [1] and the value of LogDepth [1] is less than 10% of the value of LogDepth [1]. If the determination result in step S57 is affirmative, the process proceeds to step S54. If the determination result in step S57 is negative, the process proceeds to step S58.

  In step S58, the CPU 311 resets the value of LogIndex to 1 (initial value), and ends the virtual camera interval determination process.

  As described above, in the virtual camera interval determination process, the position of the outer imaging unit (left) 23a calculated based on the position and orientation of the marker 61 included in the left real world image, and the marker included in the right real world image Ew and ed are calculated based on the position of the outer imaging unit (right) 23b calculated based on the position and orientation of 61. The value of ew and the value of ed calculated in this way are sequentially stored in LogWidth and LogDepth. At this time, when the newly calculated ew value deviates from the predetermined range (± 10%) based on the ew value first stored in LogWidth (that is, the value of LogWidth [1]), If the value of ed calculated in step 1 is out of a predetermined range (± 10%) based on the value of ed initially stored in Log Depth (that is, the value of Log Depth [1]), LogWidth and LogDepth The ew value and the ed value are stored again from the beginning. Therefore, only when the ew value and the ed value calculated sequentially do not show a large variation (that is, when the ew value and the ed value are stable to some extent for a certain period of time), the values of the ew values. And the average value of those ed values are stored in EyeWidth and EyeDepth.

  By the way, when the user moves or rotates the game apparatus 10, the left real world image and the right real world image captured by the outer imaging unit 23 are easily blurred, and the recognition accuracy of the marker 61 is significantly reduced. The ew value and the ed value that are sequentially detected are likely to fluctuate greatly. Therefore, it should be avoided to determine the value of EyeWidth and the value of EyeDepth based on the value of ew and the value of ed with low reliability detected in such a situation. Therefore, in the present embodiment, the EyeWidth value and the EyeDepth value are determined based on the values of ew and ed, which are sequentially calculated, only when there is no significant change. Further, by using the average value of the ew value and the ed value calculated multiple times as the EyeWidth value and the EyeDepth value, the accuracy of the EyeWidth value and the EyeDepth value can be improved.

(View matrix generation processing)
Next, details of the view matrix generation processing in step S44 in the update processing (FIG. 15) will be described with reference to the flowchart of FIG.

  In step S60 of FIG. 17, the CPU 311 determines whether EyeWidth is greater than 0. If it is greater than 0, the process proceeds to step S63. If not (ie, 0), the process proceeds to step S63. Proceed to S61.

  In step S61, the CPU 311 determines whether the display mode is the synchronous display mode. If the display mode is the synchronous display mode, the view matrix generation process is terminated. If not (that is, the asynchronous display mode is selected), the process proceeds to step S62.

  In step S62, the CPU 311 sets the value of ARAactive to false. Then, the view matrix generation process ends.

  In step S63, the CPU 311 refers to the left transformation matrix 75L to determine whether or not marker recognition of the left real world image has succeeded. If the marker recognition of the left real world image has succeeded, the processing is performed. The process proceeds to step S64, and if not (that is, if the value of the left conversion matrix 75L is a null value), the process proceeds to step S66.

  In step S64, the CPU 311 stores the value of the left conversion matrix 75L in the left view matrix 76L. This is to generate a virtual space image for the left eye based on the position and orientation of the outer imaging unit (left) 23a in the marker coordinate system calculated based on the position and orientation of the marker 61 included in the left real world image. This means that the left virtual camera 63L is handled as it is as the position and orientation.

  In step S65, the CPU 311 stores the value obtained by multiplying the value of the left view matrix 76L by the translation matrix (−EyeWidth, 0, 0) in the right view matrix 76R. As shown in FIG. 30, this is shifted from the position of the left virtual camera 63L in the world coordinate system of the virtual space set in step S64 described above by the value of EyeWidth in the positive direction of the x axis of the left virtual camera coordinate system. This means that the detected position is treated as the position of the right virtual camera 63R for generating the virtual space image for the right eye. Further, the posture of the right virtual camera 63R is the same as the posture of the left virtual camera 63L (that is, the x-axis, y-axis, and z-axis of the left virtual camera coordinate system are the x-axis, y-axis of the right virtual camera coordinate system). , Parallel to the z axis). Thereby, the consistency between the position and orientation of the left virtual camera 63L and the position and orientation of the right virtual camera 63R is maintained, so that the virtual object 62 can be displayed so that it can be normally stereoscopically viewed on the upper LCD 22. .

  In step S66, the CPU 311 refers to the right transformation matrix 75R to determine whether the marker recognition of the right real world image is successful. The process proceeds to step S67, and if not (that is, if the value of the right transformation matrix 75R is a null value), the view matrix generation process ends.

  In step S67, the CPU 311 stores the value of the right transformation matrix 75R in the right view matrix 76R. This is to generate a virtual space image for the right eye based on the position and orientation of the outer imaging unit (right) 23b in the marker coordinate system calculated based on the position and orientation of the marker 61 included in the right real world image. This means that the position and orientation of the right virtual camera 63R are handled as they are.

  In step S68, the CPU 311 stores the value obtained by multiplying the value of the right view matrix 76R by the translation matrix (EyeWidth, 0, 0) in the left view matrix 76L. As shown in FIG. 31, this is shifted from the position of the right virtual camera 63R in the world coordinate system of the virtual space set in step S67 described above by the value of EyeWidth in the negative direction of the x axis of the right virtual camera coordinate system. This position is treated as the position of the left virtual camera 63L for generating the virtual space image for the left eye. Further, the posture of the left virtual camera 63L is the same as the posture of the right virtual camera 63R (that is, the x-axis, y-axis, and z-axis of the left virtual camera coordinate system are the x-axis, y-axis of the right virtual camera coordinate system). , Parallel to the z axis). Thereby, the consistency between the position and orientation of the left virtual camera 63L and the position and orientation of the right virtual camera 63R is maintained, so that the virtual object 62 can be displayed so that it can be normally stereoscopically viewed on the upper LCD 22. .

  Thus, in the present embodiment, the position and orientation of one virtual camera (for example, the left virtual camera 63L) is calculated from the captured image of one outer imaging unit (for example, the outer imaging unit (left) 23a). The “marker / camera conversion matrix” is set (more specifically, it is used as it is), but the position and orientation of the other virtual camera (for example, the right virtual camera 63R) are set on the other outer imaging unit ( For example, the setting is made without using the “marker / camera conversion matrix” calculated from the captured image of the outer imaging section (right) 23b).

  When performing stereoscopic display of AR with a stereo camera, it is necessary to set two virtual cameras, one for the right and one for the left, and the “marker / camera conversion matrix” is converted by the outer imaging unit (left) 23a. There is a matrix (left transformation matrix 25L) and a transformation matrix (right transformation matrix 25R) by the outer imaging unit (right) 23b. In the present embodiment, when setting the virtual cameras 63L and 63R, instead of using the conversion matrices 25L and 25R, one conversion matrix is set (one conversion matrix of the conversion matrices 25L and 25R is set). Use as it is, or generate another new transformation matrix from both transformation matrices 25L and 25R (eg, average position, average orientation, etc.) and use both transformation cameras to create both virtual cameras. The positions and postures of 63L and 63R are set. Thereby, the problem of accuracy in AR recognition can be solved.

  In step S69, the CPU 311 sets the value of ARAActive to true. Thereby, the composite display of the virtual object 62 with respect to the real world image is started or restarted.

  In step S <b> 70, the CPU 311 determines whether the value of EyeMeasure is true. If it is true, the view matrix generation process ends. If not (i.e., if false), the process proceeds to step S <b> 71. .

  In step S71, the CPU 311 calculates the coordinate V of the origin of the virtual space represented by the left virtual camera coordinate system based on the left view matrix 76L determined in step S64 or step S68. Specifically, as shown in FIG. 32, the coordinate V of the origin of the virtual space represented by the left virtual camera coordinate system is obtained by multiplying (0, 0, 0) by the left view matrix 76L. . Here, the coordinate of the origin of the virtual space represented by the left virtual camera coordinate system is V, but instead of this, the calculation is performed based on the right view matrix 76R determined in step S65 or step S67. The coordinate of the origin of the virtual space represented by the right virtual camera coordinate system may be V. The V thus obtained is substantially the same as the position of the marker 61 in the outer imaging unit (left) coordinate system (however, the value of V is a unit of length in the virtual space or the marker coordinate system). And does not match the unit of length in the real world.)

In step S72, the CPU 311 determines the absolute value (| V.z |) of the z coordinate value of V and E
It is determined whether or not the absolute value of the difference from the value of yesDepth is greater than 20% of the value of EyeDepth. If so, the process proceeds to step S73. If not, the view matrix generation process ends. | V. z | is substantially the same as the distance (depth) in the imaging direction from the outer imaging unit 23 to the marker 61. That is, in step S72, the depth distance (depth) from the outer imaging unit 23 to the marker 61 is within a range of ± 20% with reference to the time when the EyeWidth value is calculated in the virtual camera interval determination process (FIG. 16). Judging whether it has changed beyond that.

  In step S73, the CPU 311 sets the value of EyeMeasure to true, further resets the value of LogIndex to 1, sets the marker recognition mode 79 to both image recognition modes, and ends the view matrix generation process. . As a result, the EyeWidth calculation process by the virtual camera interval determination process is started again.

  As described above, in the virtual camera interval determination process, when the depth distance (depth) from the outer imaging unit 23 to the marker 61 has changed beyond a predetermined range on the basis of the previous calculation of the value of EyeWidth, The reason for recalculating EyeWidth is the depth distance (depth) from the outer imaging unit 23 to the marker 61 due to an error in the accuracy of attaching the outer imaging unit (left) 23a and the outer imaging unit (right) 23b to the game apparatus 10. This is because the optimal virtual camera interval (EyeWidth) changes accordingly. For example, in a situation where the shooting direction of the outer imaging unit (left) 23a and the shooting direction of the outer imaging unit (right) 23b are not parallel, the depth distance from the outer imaging unit 23 to the marker 61 is as shown in FIG. Assuming that EyeWidth calculated in the virtual camera interval determination processing when D1 is EyeWidth1, in the virtual space, the left virtual camera 63L and the right virtual camera 63R are arranged with an interval of EyeWidth1 as shown in FIG. (By doing so, for example, a virtual object placed at the origin of the virtual space is appropriately displayed in a three-dimensional manner as if it existed on the marker 61 of the real world image. ). Then, when the depth distance from the outer imaging unit 23 to the marker 61 is changed to D2 smaller than D1, as shown in FIG. 35, the EyeWidth value calculated in the virtual camera interval determination process is smaller than EyeWidth1. As shown in FIG. 36, it is appropriate that the left virtual camera 63L and the right virtual camera 63R are arranged with an interval of EyeWidth 2 as shown in FIG.

(Main real world image setting process)
Next, details of the main real world image setting process of step S45 in the update process (FIG. 15) will be described with reference to the flowchart of FIG.

  In step S80 of FIG. 18, the CPU 311 determines whether or not the value of ARAactive is true. If true, the process proceeds to step S81, and if not (that is, if the value of ARAactive is false). Ends the main real-world image setting process.

  In step S81, the CPU 311 generates a left projection matrix 77L and a right projection matrix 77R. The left projection matrix 77L is a matrix that defines the drawing range of the virtual space viewed from the left virtual camera 63L, and in this embodiment, as shown in FIG. 37, the left real world image determined in step S23 of the main process. A left projection matrix 77L corresponding to the drawing range is generated. The right projection matrix 77R is a matrix that defines the drawing range of the virtual space viewed from the right virtual camera 63R. In this embodiment, as shown in FIG. 38, the right real world image determined in step S27 of the main process is shown in FIG. A right projection matrix 77R corresponding to the drawing range is generated.

Specifically, the projection matrix of the left virtual camera 63L indicates that the ratio of the horizontal field angle of the outer imaging unit (left) 23a and the horizontal field angle of the left virtual camera is the horizontal direction of the left real world image. It is set as a projection matrix that defines a visual volume having a field angle that is the same as the ratio representing the horizontal position and size of the drawing range in the length of the direction.

  In step S82, the CPU 311 determines whether or not the main real world image is a left real world image. If the main real world image is a left real world image, the process proceeds to step S83; If the real world image is a right real world image), the process proceeds to step S86.

  In step S83, the CPU 311 obtains a vector V by multiplying (0, 0, 0, 1) by the left view matrix 76L and the left projection matrix 77L.

  In step S84, the CPU 311 determines whether the value (V.x / V.w) obtained by multiplying the first component (x) of the vector V by the fourth component (w) is greater than 0.5. If it is greater than 0.5, the process proceeds to step S85; otherwise, the main real world image setting process is terminated. V. above. x / V. The value of w indicates the position of the origin of the world coordinate system of the virtual space in the left-right direction on the left-eye virtual space image (in the left-eye virtual space image Thus, “where the origin of the world coordinate system in the virtual space is located in the horizontal direction of the image” is “on the captured image of the outer imaging unit (left) 23a, the origin of the marker coordinate system is the left and right of the image. Is equivalent to “where in the direction”). When the origin of the virtual space world coordinate system is located at the center of the virtual space image for the left eye, x / V. As the value of w becomes 0 and the origin of the world coordinate system of the virtual space approaches the left end of the virtual space image for the left eye, V. x / V. As the value of w approaches -1.0 and the origin of the virtual space world coordinate system approaches the right end of the virtual space image for the left eye, V. x / V. The value of w approaches +1.0. V. x / V. That the value of w is larger than 0.5 means that the origin of the world coordinate system of the virtual space is located in the right end region (shaded region in FIG. 39) of the virtual space image for the left eye. In other words, it means that the marker 61 is located in the right end region of the left real world image imaged by the outer imaging unit (left) 23a.

  In step S85, the CPU 311 changes the main real world image from the left real world image to the right real world image. That is, on the virtual space image by the left virtual camera 63L, when the origin of the virtual space exists on the right side of the right position by a predetermined distance (or a predetermined ratio of the left and right width of the image) from the center of the image, Change the main real world image to the right real world image. Alternatively, the origin of the marker coordinate system exists on the right side of the right position by a predetermined distance from the center of the image (or by a predetermined ratio of the horizontal width of the image) on the image captured by the outer imaging unit (left) 23a. When changing the main real world image to the right real world image. Thereby, for example, even if the position of the marker 61 gradually moves to the right in the left real world image and the marker 61 disappears from the left real world image, the marker 61 disappears from the left real world image. Since the main real world image is changed to the right real world image before it is lost, the marker 61 can be continuously recognized in the one-image recognition mode. When the process of step S85 is completed, the main real world image setting process is terminated.

  In step S86, the CPU 311 obtains the vector V by multiplying (0, 0, 0, 1) by the right view matrix 76R and the right projection matrix 77R.

In step S87, the CPU 311 determines whether the value (V.x / V.w) obtained by multiplying the first component (x) of the vector V by the fourth component (w) is less than -0.5. If the value is smaller than -0.5, the process proceeds to step S87. If not, the main real world image setting process ends. V. above. x / V. The value of w indicates the position of the origin of the world coordinate system of the virtual space in the left-right direction on the virtual space image for the right eye. When the origin of the virtual space world coordinate system is located at the center of the virtual space image for the right eye, x / V. As the value of w becomes 0 and the origin of the world coordinate system of the virtual space approaches the left end of the virtual space image for the right eye, V. x / V. As the value of w approaches -1.0 and the origin of the virtual space world coordinate system approaches the right end of the virtual space image for the right eye, V.V. x / V. The value of w approaches +1.0. V. x / V. That the value of w is smaller than −0.5 means that the origin of the virtual space world coordinate system is located in the left end region (shaded region in FIG. 40) of the right-eye virtual space image. In other words, it means that the marker 61 is located in the left end region of the right real world image imaged by the outer imaging unit (right) 23b.

  In step S88, the CPU 311 changes the main real world image from the right real world image to the left real world image. Thereby, for example, even if the position of the marker 61 gradually moves leftward in the right real world image, and the marker 61 disappears from the right real world image, the marker 61 disappears from the right real world image. Since the main real world image is changed to the left real world image before it is lost, the marker 61 can be continuously recognized in the one-image recognition mode. When the process of step S88 ends, the main real world image setting process ends.

(Display mode switching process)
Next, details of the display mode switching process of step S46 in the update process (FIG. 15) will be described with reference to the flowchart of FIG.

  In step S90 of FIG. 19, the CPU 311 determines whether or not the display mode is the synchronous display mode. If the display mode is the synchronous display mode, the process proceeds to step S91; otherwise (that is, the asynchronous display mode). In step S96, the process proceeds to step S96.

  In step S <b> 91, the CPU 311 determines whether EyeWidth is greater than 0 and whether the marker recognition of either the left real world image or the right real world image is successful. If the determination result is affirmative, the process proceeds to step S92, and if the determination result is negative (that is, EyeWidth is 0, or both the left real world image and the right real world image are both). If the real world image marker recognition has failed), the process proceeds to step S93.

  In step S92, the CPU 311 sets the value of SyncCount (see FIG. 12) to 20. SyncCount is a variable for determining the timing for switching the display mode from the synchronous display mode to the asynchronous display mode. When the process of step S92 ends, the display mode switching process ends.

  In step S93, the CPU 311 decrements the value of SyncCount.

  In step S94, the CPU 311 determines whether or not the value of SyncCount is greater than 0. If it is greater than 0, the display mode switching process is terminated. If not (ie, 0), the process proceeds to step S95. Proceed to

  In step S95, the CPU 311 sets the value of LogIndex to 1, further sets the values of EyeWidth and EyeDepth to 0, further changes the display mode 78 from the synchronous display mode to the asynchronous display mode, and further sets the marker recognition mode 79. Is set to both image recognition mode. When the process of step S95 ends, the display mode switching process ends.

  In step S <b> 96, the CPU 311 determines whether EyeWidth is greater than 0 and whether the real world image of either the left real world image or the right real world image has been successfully recognized. If the determination result is affirmative, the process proceeds to step S97, and if the determination result is negative (that is, EyeWidth is 0, or both the left real world image and the right real world image are displayed). When the real world image marker recognition has failed), the display mode switching process is terminated.

  In step S97, the CPU 311 sets the value of SyncCount to 20, and further changes the display mode from the asynchronous display mode to the synchronous display mode. When the process of step S97 ends, the display mode switching process ends.

  As described above, the display mode is appropriately switched between the synchronous display mode and the asynchronous display mode by the display mode switching process. More specifically, immediately after the execution of the image display program is started, the display mode is the asynchronous display mode, and the latest real world image is always displayed on the upper LCD 22. Thereafter, when the marker 61 is recognized and the virtual object 62 can be synthesized and displayed on the real world image displayed on the upper LCD 22, the display mode changes from the asynchronous display mode to the synchronous display mode. In the synchronous display mode, in order to synthesize and display the virtual object 62 at the correct position in the real world image, the real world image (which is not necessarily the latest real world image) that can finally recognize the marker 61 is displayed. A synthesized image obtained by synthesizing the virtual object 62 is displayed on the upper LCD 22. As a result, it is possible to prevent positional deviation of the virtual object 62 that is synthesized and displayed on the real world image. Thereafter, when the state in which the marker 61 cannot be recognized continues for a certain period of time, the display mode changes from the synchronous display mode to the asynchronous display mode, and the uppermost LCD 22 always displays the latest real world image. Become. Thereby, it is possible to prevent an old image from being continuously displayed on the upper LCD 22 when a state in which the marker 61 cannot be recognized continues.

(Effect of this embodiment)
As described above, according to the present embodiment, one outer imaging unit (the outer imaging unit (left) 23a or the outer imaging unit) calculated from the marker recognition result of one of the left real world image and the right real world image. (Right) The position and orientation of one virtual camera is determined based on the position and orientation in the marker coordinate system of 23b), and the position and orientation of the left virtual camera 63L and the position and orientation of the right virtual camera 63R are ideal. The position and orientation of the other virtual camera are determined so as to achieve a general relationship. Accordingly, the virtual object 62 can be displayed on the display device capable of stereoscopic viewing so that the stereoscopic viewing can be performed normally.

  The position of the outer imaging unit (left) 23a calculated based on the marker recognition result for the left real world image and the position of the outer imaging unit (right) 23b calculated based on the marker recognition result for the right real world image Based on the above, the interval between the two outer imaging units is calculated, the virtual camera interval (EyeWidth) is determined based on the result, and the direction of one virtual camera is orthogonal to the imaging direction of the coordinate system of the virtual camera The position of the other virtual camera is determined by moving it by the interval. Thereby, it is possible to set so that both virtual cameras are aligned in a direction perpendicular to the shooting direction. Also, if the interval between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b is not known, or the accuracy of attaching the outer imaging unit (left) 23a and the outer imaging unit (right) 23b to the game apparatus 10 is high. Even if it is bad, the left virtual camera 63L and the right virtual camera 63R can be arranged at ideal intervals.

In addition, after the virtual camera interval (EyeWidth) is determined by the virtual camera interval determination process, it is only necessary to perform the marker recognition process on the main real world image. Compared with the case where marker recognition processing is performed on the processing load, the processing load can be reduced.

  When the interval between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b is known, the outer imaging unit (left) 23a and the outer side in the real world are determined based on the result of the virtual camera interval determination process. The interval (EyeWidth) between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b in the marker coordinate system corresponding to the interval (for example, 3.5 cm) between the imaging unit (right) 23b is found. Therefore, for example, a character (virtual object) having a height of 30 cm is displayed in a synthesized manner on a real world image, or a character (virtual object) synthesized and displayed on a real world image is moved at a speed of 10 cm per second in a virtual space. It becomes possible to perform processing based on the real world scale.

(Modification)
In the above embodiment, the position and orientation of the marker 61 included in the real world image are recognized, and the virtual object 62 is combined with the real world image according to the recognition result, but in other embodiments, Not only the marker 61 but also the position and / or orientation of an arbitrary recognition target may be recognized, and the virtual object 62 may be combined with the real world image according to the recognition result. An example of a recognition target is a human face.

  In the above embodiment, a stereoscopic image is displayed on the upper LCD 22 based on a real world image captured in real time by the outer imaging unit 23. In other embodiments, the outer imaging unit 23 or an external stereo camera is displayed. For example, a stereoscopic image may be displayed on the upper LCD 22 based on moving image data captured in the past.

  In the above embodiment, the outer imaging unit 23 is mounted in advance on the game apparatus 10. However, in other embodiments, an external camera that can be attached to and detached from the game apparatus 10 may be used.

  In the above embodiment, the upper LCD 22 is mounted on the game apparatus 10 in advance. However, in another embodiment, an external stereoscopic display that can be attached to and detached from the game apparatus 10 may be used.

  In the above embodiment, the virtual object 62 is arranged at the origin position of the marker coordinate system. However, in another embodiment, the virtual object 62 may be arranged at a position away from the origin of the marker coordinate system. .

  In the above embodiment, only one virtual object is arranged in the virtual space. However, in other embodiments, a plurality of virtual objects may be arranged in the virtual space.

In the above embodiment, in the virtual camera interval determination process, after calculating the interval (EyeWidth) between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b in the marker coordinate system, the marker is determined based on the interval. Although the position and orientation of the other virtual camera are determined from the position and orientation of one of the left virtual camera 63L and the right virtual camera 63R calculated based on the recognition result, in other embodiments The position and orientation of the outer imaging unit (left) 23a calculated based on the marker recognition result for the left real world image and the outer imaging unit (right) 23b calculated based on the marker recognition result for the right real world image Based on the position and orientation, the position and orientation of the outer imaging unit 23 (for example, the position of the outer imaging unit (right) 23a and the outer imaging unit (left) 23). And the average position of the outer imaging unit (right) 23a and the outer imaging unit (left) 23b are calculated based on the positions of the left virtual camera 63L and the right virtual camera 63R. Alternatively, the posture may be determined. For example, the postures of the left virtual camera 63L and the right virtual camera 63R are both calculated based on the marker recognition result for the left real world image, and the marker recognition for the right real world image. You may make it determine the attitude | position of the left virtual camera 63L and the right virtual camera 63R so that it may become an attitude | position just in the middle of the attitude | position of the outer side imaging part (right) 23b calculated based on the result. Further, for example, after calculating the interval (EyeWidth) between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b in the virtual camera interval determination process, the outer imaging calculated based on the marker recognition result for the left real world image The EyeWidth / direction in the opposite direction from the position in the virtual space corresponding to the average position of the position of the portion (left) 23a and the position of the outer imaging portion (right) 23b calculated based on the marker recognition result for the right real world image The positions of the left virtual camera 63L and the right virtual camera 63R may be determined by shifting by 2 in a direction orthogonal to the shooting direction of the virtual camera.

  Further, in the above embodiment, after the virtual camera interval (EyeWidth) is determined by the virtual camera interval determination process, the marker recognition process is performed only on the main real world image. Marker recognition processing may be performed on both the left real world image and the right real world image.

  In the above embodiment, the upper LCD 22 is a parallax barrier type stereoscopic display device. However, in other embodiments, the upper LCD 22 may be another arbitrary type of stereoscopic display device such as a lenticular method. For example, when a lenticular stereoscopic display device is used, the left-eye image and the right-eye image are combined by the CPU 311 or another processor, and then the combined image is supplied to the lenticular stereoscopic display device. Also good.

  In the above embodiment, the virtual object is synthesized and displayed on the real world image using the game apparatus 10. However, in other embodiments, any information processing apparatus or information processing system (for example, PDA (Personal Digital Assistant) is used. ), A mobile phone, a personal computer, a camera, etc.) may be used to synthesize and display the virtual object on the real world image.

  In the above embodiment, the image display process is executed by only one information processing device (game device 10). In other embodiments, the image display system includes a plurality of information processing devices that can communicate with each other. The plurality of information processing apparatuses may share and execute the image display process.

10 Game device 11 Lower housing 12 Lower LCD
13 Touch Panel 14 Operation Buttons 15 Analog Stick 16 LED
21 Upper housing 22 Upper LCD
23 Outside imaging unit 23a Outside imaging unit (left)
23b Outside imaging unit (right)
24 Inner imaging unit 25 3D adjustment switch 26 3D indicator 28 Touch pen 31 Information processing unit 311 CPU
312 GPU
32 Main memory 60a-60c Ball 61 Marker 62 Virtual object 63L Left virtual camera 63R Right virtual camera

Claims (23)

This is an image display program for stereoscopically displaying a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing, using outputs from a real camera for the right eye and a real camera for the left eye. Computer
Of the real world image data output from the real camera for the right eye and the real world image data output from the real camera for the left eye, a predetermined imaging target in the real world image data output from one real camera A first position / orientation calculation means for calculating position / orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging target,
Using the position and orientation information calculated by the first position and orientation calculation means, a right virtual camera for generating an image for the right eye in a predetermined virtual space and a left virtual camera for generating an image for the left eye Virtual camera setting means for determining the position and orientation for both,
Right virtual space image generation means for generating a right virtual space image indicating the virtual space viewed from the right virtual camera;
A left virtual space image generating means for generating a left virtual space image indicating the virtual space viewed from the left virtual camera; and
Combining the right virtual space image with real world image data output from the real camera for the right eye and combining the left virtual space image with real world image data output from the real camera for the left eye An image display program for causing the stereoscopic display device to function as display control means for outputting an image for stereoscopic viewing.   The virtual camera setting means is configured such that a relative posture relationship between the right virtual camera and the left virtual camera is a design relative posture relationship between the real camera for the right eye and the real camera for the left eye. The image display program according to claim 1, wherein positions and orientations of the right virtual camera and the left virtual camera are determined so as to be the same. The virtual camera setting means includes
Using the position / orientation information calculated by the first position / orientation calculation unit, one of the right virtual camera and the left virtual camera corresponding to the one real camera in the first position / orientation calculation unit A first virtual camera posture determination unit for determining the posture of the camera;
Based on the posture of the one virtual camera determined by the first virtual camera posture determination unit, the relationship of the relative posture between the right virtual camera and the left virtual camera is the real camera for the right eye and the The image display of Claim 2 including the 2nd camera attitude | position determination part which determines the attitude | position of the other virtual camera so that it may become the same as the relationship of the design relative attitude | position with the real camera for left eyes. program. The image display program causes the computer to
Further function as a virtual camera relative positional relationship determining means for determining a relative positional relationship between the right virtual camera and the left virtual camera,
The virtual camera setting means includes
Using the position / orientation information calculated by the first position / orientation calculation unit, one of the right virtual camera and the left virtual camera corresponding to the one real camera in the first position / orientation calculation unit A first virtual camera position determining unit for determining the position of the camera;
The position of the other virtual camera is determined at a position separated from the position of the one virtual camera determined by the first virtual camera position determining unit by the relative position determined by the virtual camera relative positional relationship determining means. The image display program according to claim 2, further comprising: a second virtual camera position determination unit that performs the operation. The virtual camera setting means includes
Using the position / orientation information calculated by the first position / orientation calculation unit, one of the right virtual camera and the left virtual camera corresponding to the one real camera in the first position / orientation calculation unit A first virtual camera position determining unit for determining the position of the camera;
Using the position / orientation information calculated by the first position / orientation calculation means, the one of the right virtual camera and the left virtual camera corresponding to the one real camera in the first position / orientation calculation means. A first virtual camera attitude determination unit that determines the attitude of the virtual camera;
When viewed from the position of the one virtual camera determined by the first virtual camera position determining unit, the other virtual camera is set in a direction based on the posture of the one virtual camera determined by the first virtual camera attitude determining unit. The image display program according to claim 2, further comprising a second virtual camera position determination unit that determines a position of the camera. The computer,
Further function as a virtual camera distance determining means for determining a distance between the right virtual camera and the left virtual camera,
The second camera position determination unit is configured to display the one virtual camera determined by the first virtual camera attitude determination unit as viewed from the position of the one virtual camera determined by the first virtual camera position determination unit. The image display program according to claim 5, wherein the position of the other virtual camera is determined at a position separated by a distance determined by the inter-virtual camera distance determining means in a direction based on a posture. The virtual camera relative positional relationship determining means is
Relative positional relationship between the right virtual camera and the left virtual camera is determined based on the parallax between the real world image data output from the real camera for the right eye and the real world image data output from the real camera for the left eye. The image display program according to claim 4.   The image display program according to claim 7, wherein the relative positional relationship determined by the relative positional relationship determining unit is an interval between the right virtual camera and the left virtual camera. The image display program causes the computer to
Of the two real world image data respectively output from the two real cameras, the real position image data in the real world image data output from the other real camera different from the one real camera in the first position and orientation calculation means. By recognizing a predetermined imaging target, it further functions as second position and orientation calculation means for calculating position and orientation information indicating a relative position and orientation between the other real camera and the imaging target,
The virtual camera relative positional relationship determining means includes information on a relative position between the one real camera calculated by the first position / orientation calculating means and the imaging target, and the second position / orientation calculating means. 5. The distance between the one real camera and the other real camera is calculated using the calculated relative position information between the other real camera and the imaging target. The image display program described in 1. The first position / orientation calculation means uses the one real camera to obtain a coordinate value represented by a coordinate system with the predetermined imaging target as an origin, based on real world image data output from the one real camera. Including first conversion matrix generation means for generating a first conversion matrix for conversion into a coordinate value represented by a first imaging unit coordinate system as an origin,
The second position / orientation calculation means uses the other real camera as a coordinate value represented by a coordinate system with the predetermined imaging object as an origin, based on real world image data output from the other real camera. The image display program according to claim 9, further comprising second conversion matrix generation means for generating a second conversion matrix for conversion into a coordinate value represented by a second imaging unit coordinate system as an origin. The virtual camera relative positional relationship determining means executes a process of calculating a relative positional relationship between the right virtual camera and the left virtual camera each time new real world image data is output from the two real cameras, The image display program according to claim 7, wherein the relative positional relationship between the right virtual camera and the left virtual camera is determined based on a plurality of relative positional relationship calculation results obtained by executing the processing a plurality of times.   The virtual camera relative positional relationship determining means is configured so that only when the plurality of relative positional relationship calculation results are all within a predetermined range, the virtual camera relative positional relationship determination unit and the right virtual camera based on the multiple relative positional relationship calculation results. The image display program according to claim 11, wherein the relative positional relationship of the left virtual camera is determined. The image display program causes the computer to
After the relative positional relationship between the right virtual camera and the left virtual camera is determined by the virtual camera relative positional relationship determining means, the predetermined imaging target for the two real cameras from the time when the relative positional relationship is determined Further functioning as a depth change determination means for determining whether or not the depth distance has changed beyond a predetermined range,
The image according to claim 7, wherein the virtual camera relative positional relationship determination unit re-determines a relative positional relationship between the right virtual camera and the left virtual camera when a determination result of the depth change determination unit is affirmative. Display program. The image display program causes the computer to
Based on real world image data including the predetermined imaging target, further function as a depth distance calculating means for calculating a depth distance of the predetermined imaging target with respect to the two real cameras,
The depth change determination means includes a reference depth distance calculated by the depth distance calculation means when the relative position relation between the right virtual camera and the left virtual camera is determined by the virtual camera relative position relationship determination means, and thereafter Comparing whether or not the depth distance of the predetermined imaging target with respect to the two real cameras has changed beyond a predetermined range by comparing with the latest depth distance calculated by the depth distance calculating means. Item 14. The image display program according to Item 13.   The depth distance calculating unit executes a process of calculating a depth distance of the predetermined imaging target with respect to the two real cameras each time new real world image data is output from the two real cameras. The image display program according to claim 14, wherein the reference depth distance is calculated based on a plurality of depth distance calculation results obtained by executing a plurality of times.   16. The depth distance calculation unit calculates the reference depth distance based on the plurality of depth distance calculation results only when the plurality of depth distance calculation results are all within a predetermined range. The image display program described in 1. The image display program causes the computer to
Of the two real world image data respectively output from the two real cameras, the real position image data in the real world image data output from the other real camera different from the one real camera in the first position and orientation calculation means. Second position and orientation calculation means for calculating position and orientation information indicating a relative position and orientation between the other real camera and the predetermined imaging target by recognizing the predetermined imaging target; and
Further function as real camera selection means for selecting one of the two real cameras according to the position of the predetermined imaging target in at least one of the two real world image data respectively output from the two real cameras Let
The virtual camera setting means includes
When the real camera selected by the real camera selection unit is the one real camera, the relative position between the one real camera calculated by the first position and orientation calculation unit and the predetermined imaging target is calculated. The position and orientation of the right virtual camera and the left virtual camera are determined using position and orientation information indicating a specific position and orientation, and a real camera,
When the real camera selected by the real camera selection unit is the other real camera, the relative position between the other real camera calculated by the second position and orientation calculation unit and the predetermined imaging target is calculated. The image display program according to claim 1, wherein the real camera determines the positions and orientations of the right virtual camera and the left virtual camera using position and orientation information indicating a specific position and orientation. The real camera selection means includes
In response to the position of the predetermined imaging object in the real world image data output from the real camera for the left eye of the two real cameras entering the right end region of the real world image data, Switch from the real camera to the real camera for the right eye,
In response to the position of the predetermined imaging target in the real world image data output from the real camera for the right eye entering the left end region of the real world image data, the real camera for the left eye from the real camera for the right eye The image display program according to claim 17, wherein the image display program is switched to a real camera.   The image display program according to claim 1, wherein the real world image data used by the first position and orientation calculation unit and the display control unit is real world image data output in real time from the real camera.   The image display program according to claim 1, wherein the computer is built in an information processing apparatus including the two real cameras and the stereoscopic display device. This is an image display device that stereoscopically displays a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device that can be viewed stereoscopically, using outputs from the real camera for the right eye and the real camera for the left eye. And
Of the real world image data output from the real camera for the right eye and the real world image data output from the real camera for the left eye, a predetermined imaging target in the real world image data output from one real camera A first position / orientation calculation means for calculating position / orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging target,
Using the position and orientation information calculated by the first position and orientation calculation means, a right virtual camera for generating an image for the right eye in a predetermined virtual space and a left virtual camera for generating an image for the left eye Virtual camera setting means for determining the position and orientation for both,
Right virtual space image generation means for generating a right virtual space image indicating the virtual space viewed from the right virtual camera;
A left virtual space image generating means for generating a left virtual space image indicating the virtual space viewed from the left virtual camera; and
Combining the right virtual space image with real world image data output from the real camera for the right eye and combining the left virtual space image with real world image data output from the real camera for the left eye An image display device comprising display control means for outputting an image for stereoscopic viewing to the stereoscopic display device. This is an image display system that stereoscopically displays a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing, using outputs from a real camera for the right eye and a real camera for the left eye. And
Of the real world image data output from the real camera for the right eye and the real world image data output from the real camera for the left eye, a predetermined imaging target in the real world image data output from one real camera A first position / orientation calculation means for calculating position / orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging target,
Using the position and orientation information calculated by the first position and orientation calculation means, a right virtual camera for generating an image for the right eye in a predetermined virtual space and a left virtual camera for generating an image for the left eye Virtual camera setting means for determining the position and orientation for both,
Right virtual space image generation means for generating a right virtual space image indicating the virtual space viewed from the right virtual camera;
A left virtual space image generating means for generating a left virtual space image indicating the virtual space viewed from the left virtual camera; and
Combining the right virtual space image with real world image data output from the real camera for the right eye and combining the left virtual space image with real world image data output from the real camera for the left eye An image display system comprising display control means for outputting an image for stereoscopic viewing to the stereoscopic display device. This is an image display method for stereoscopically displaying a real world image obtained by synthesizing a three-dimensional virtual object on the screen of a stereoscopic display device capable of stereoscopic viewing, using outputs from a real camera for the right eye and a real camera for the left eye. And
Of the real world image data output from the real camera for the right eye and the real world image data output from the real camera for the left eye, a predetermined imaging target in the real world image data output from one real camera A first position and orientation calculation step for calculating position and orientation information indicating a relative position and orientation between the one real camera and the predetermined imaging target by recognizing
Using the position and orientation information calculated in the first position and orientation calculation step, a right virtual camera for generating a right-eye image in a predetermined virtual space and a left virtual camera for generating an image for the left eye Virtual camera setting step to determine the position and orientation for both,
A right virtual space image generation step of generating a right virtual space image indicating the virtual space viewed from the right virtual camera;
A left virtual space image generation step of generating a left virtual space image indicating the virtual space viewed from the left virtual camera; and
Combining the right virtual space image with real world image data output from the real camera for the right eye and combining the left virtual space image with real world image data output from the real camera for the left eye An image display method comprising a display control step of outputting an image for stereoscopic viewing to the stereoscopic display device.

Images