JP2012069126A - Image processing program, image processor, image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To take an image with which a user feels as if a virtual object exists in a real space.SOLUTION: When a marker existing in a real space is imaged with an outer camera, an image obtained by imaging plural virtual characters with a virtual camera is superimposed on an actual image obtained by imaging the real space and then displayed on an upper LCD. The virtual characters are positioned in a marker coordinate system with the marker being a reference, and the position and attitude of each virtual character change according to a button operation on a game device by a user. When the user performs a button operation for indicating an imaging instruction, the displayed image is saved in storage means.

Description

  The present invention relates to an image processing program, an image processing apparatus, an image processing system, and an image processing method for generating an image in which a virtual object is superimposed on an image captured by a real camera.

  2. Description of the Related Art Conventionally, there is an image composition device that composes an image of a virtual object with an image captured by a real camera (for example, Patent Document 1). Specifically, the apparatus described in Patent Document 1 detects a subject in a captured image, and determines the type and composite state of a virtual object according to the type and state of the detected subject. Then, a synthesized image obtained by synthesizing the image of the virtual object with the captured image is generated, and when the user gives a shooting instruction, the synthesized image at that time is recorded.

JP 2009-290842 A

  However, in Patent Document 1, a virtual object image whose size, orientation, and the like are set in advance is simply combined with a captured image, and the photograph taken with the apparatus of Patent Document 1 is as if the virtual object is real. It was not something that would give the user the feeling of being in space.

  Therefore, an object of the present invention is to provide an image processing technique for photographing an image that can give a user a feeling that a virtual object exists in real space.

  The present invention employs the following configuration in order to solve the above problems.

  The present invention is an image processing program executed in a computer of an image processing apparatus that generates an image obtained by superimposing a virtual object image on an image captured by a real camera. The image processing program causes the computer to function as a real image acquisition unit, a position / orientation information acquisition unit, a virtual camera setting unit, a virtual object change unit, a virtual object image generation unit, a superimposed image generation unit, and a storage unit. The real image acquisition means acquires a real image of the subject imaged by the real camera. The position and orientation information acquisition means acquires position and orientation information corresponding to the position and orientation of the real camera in the real space. The virtual camera setting means sets the position and orientation of the virtual camera in the virtual space based on the position and orientation information acquired by the position and orientation information acquisition means. The virtual object changing unit changes at least one of the position and orientation of at least a part of the virtual object arranged in the virtual space in response to an input to the input unit by the user. The virtual object image generation means generates a virtual object image by capturing the virtual object, at least one of the position and orientation of which has been changed by the virtual object changing means, with the virtual camera. The superimposed image generating unit generates a superimposed image by superimposing the real image acquired by the real image acquiring unit and the virtual object image generated by the virtual object image generating unit. The storage unit stores the superimposed image in the storage unit in accordance with a shooting instruction from the user.

  Note that the position / orientation information corresponding to the position and orientation of the real camera acquired by the position / orientation information acquisition means may be the relative position and orientation of the predetermined object existing in the real space and the actual camera. Good. That is, the position and orientation information may be the position and orientation of a real camera based on a predetermined object in the real space, or the position and orientation of the predetermined object in the real space based on the real camera. There may be. The position / orientation information is the absolute value of an actual camera detected by means for detecting an absolute position (for example, GPS) or attitude detection means (for example, an angular velocity sensor, an acceleration sensor, or a means for detecting geomagnetism). May be in any position and orientation.

  According to the above, a user operates a virtual object in a virtual space, and a superimposed image in which the operated virtual object is superimposed on a real image can be generated and stored. As a result, it is possible to capture an image that can give the user the feeling that the virtual object exists in real space.

  Further, in another configuration of the invention, the position / orientation information acquisition unit detects a specific object included in the real image acquired by the real image acquisition unit, and based on the detection result, the specific object You may acquire the position and attitude | position of the said real camera relative to.

  Based on the above, based on the specific object included in the real image captured by the real camera, the relative position and orientation of the specific object and the real camera in real space can be acquired.

  In another configuration of the present invention, the image processing apparatus may be connectable to display means. The superimposed image generation unit repeatedly generates a superimposed image obtained by superimposing the real image acquired by the real image acquisition unit and the virtual object image generated by the virtual object image generation unit until the shooting instruction is performed. To do. The image processing program further causes the computer to function as display control means. The display control means causes the display means to display the superimposed image generated by the superimposed image generation means until the photographing instruction is performed.

  According to the above, it is possible to display the superimposed image on the display unit until the user gives a shooting instruction. Thereby, the user can confirm the superimposed image to be photographed (saved).

  In another configuration of the present invention, the virtual object may include a first part and a second part. In this case, the virtual object changing unit changes the posture of the first part so that the first part of the virtual object faces the virtual camera in response to an input to the input unit by a user. Including means.

  According to the above, the posture of the first part can be changed so that the first part of the virtual object faces the direction of the virtual camera. For example, the face of the virtual object can be directed toward the virtual camera, and the user can feel as if the virtual object exists in real space.

  In another configuration of the present invention, the part changing means may change the first part based on the second part when the posture of the first part is changed so that the first part faces the virtual camera. When the posture of one part is outside the first range, the posture of the first part may be set to the upper limit or the lower limit of the first range.

  Based on the above, the posture of the first part with respect to the second part can be limited to the first range. For example, when the body and face of the virtual object are defined, the face direction can be controlled so that the face direction does not become unnatural compared to the body direction.

  In another configuration of the present invention, when the user inputs an input to the input unit, the part changing unit is configured such that the first part of the virtual object is the virtual camera until the photographing instruction is performed. You may change the attitude | position of the said 1st site | part repeatedly so that it may face.

  According to the above, once the user inputs, the posture of the first part is changed so that the first part of the virtual object always faces the virtual camera until a photographing instruction is given, so that useless input can be omitted. .

  In another configuration of the present invention, the real camera may be a stereo camera. The real image acquisition unit acquires a real image for the left eye captured by the real camera for the left eye of the stereo camera and a real image for the right eye captured by the real camera for the right eye of the stereo camera. The position / orientation information acquisition means acquires position / orientation information corresponding to the positions and orientations of the real camera for the left eye and the real camera for the right eye of the stereo camera. The virtual camera setting unit sets the position and posture of the left virtual camera in accordance with the position and orientation information of the real camera for the left eye acquired by the position and orientation information acquisition unit, and is acquired by the position and orientation information acquisition unit. The position and orientation of the right virtual camera are set according to the position and orientation information of the right-eye real camera. The part changing means changes the posture of the first part so that the first part of the virtual object is directed to an intermediate position between the left virtual camera and the right virtual camera. The virtual object image generating means captures the virtual object whose posture of the first part has been changed by the part changing means with the left virtual camera and the right virtual camera, thereby obtaining a virtual object image for the left eye and a right eye. A virtual object image is generated. The superimposed image generating means includes a left-eye superimposed image obtained by superimposing the left-eye real image and the left-eye virtual object image, and a right-eye superimposed image obtained by superimposing the right-eye real image and the right-eye virtual object image. A superimposed image is generated.

  Based on the above, it is possible to generate a stereoscopic image in which a virtual object is superimposed on an image obtained by capturing a real space. In this case, the first part of the virtual object can be directed to the left and right virtual cameras. For this reason, the user can shoot an image with a stereoscopic effect and can obtain a stronger augmented reality.

  In another configuration of the present invention, the virtual object changing unit may include a moving unit that moves the virtual object in response to an input to the input unit by a user.

  According to the above, the user can move the virtual object in the virtual space, and can obtain a feeling that the virtual object moves in the real space.

  In another configuration of the present invention, the virtual object changing means has the same input direction and the moving direction of the virtual object on the display means in response to an input by the user to the input means. As described above, a moving means for moving the virtual object may be included.

  According to the above, since the input direction with respect to the input unit matches the moving direction on the display, the user can move the virtual object in a desired direction by an intuitive operation.

  In another configuration of the present invention, when the moving means moves the virtual object in response to an input to the input means by a user, the moving means exceeds the second range in the virtual space. The position of the virtual object may be corrected within the range.

  Based on the above, it is possible to limit the movement range of the virtual object.

  In another configuration of the present invention, the virtual object changing unit may include a pose changing unit that changes a pose of the virtual object in response to an input to the input unit by a user.

  According to the above, the user can change the pose of the virtual object, and can obtain a feeling that the virtual object exists in the real space.

  In another configuration of the present invention, the image processing program may further cause the computer to function as selection means. The selection means selects one virtual object from a plurality of virtual objects arranged in the virtual space. The virtual object changing unit changes at least one of the position and the posture of at least a part of the virtual object selected by the selection unit in response to an input to the input unit by the user.

  According to the above, the user can select a virtual object and change the position or orientation of at least a part of the selected virtual object.

  In another configuration of the present invention, the image processing program may further cause the computer to function as selection means. The selection means selects one virtual object from a plurality of virtual objects arranged in the virtual space. The superimposed image generation unit generates a superimposed image in which a cursor indicating that the virtual object has been selected by the selection unit is further superimposed. The virtual object changing unit changes at least one of a position and a posture of at least a part of the virtual object selected by the selection unit in response to an input by the user to the input unit. The storage means hides the cursor and stores the superimposed image in the storage means in response to a shooting instruction from the user.

  According to the above, the user can select and change the virtual object while confirming the superimposed image displayed on the display unit until the superimposed image is stored according to the shooting instruction. In addition, since the cursor is superimposed on the display means, the user can confirm the selected virtual object. Further, when the superimposed image is stored, the cursor can be hidden, and a photograph can be taken as if the virtual object exists in real space.

  In another embodiment, an image processing apparatus that realizes each of the above means may be configured. In another form, a plurality of elements that realize each of the above units may operate to form one image processing system. The image processing system may be configured by a single device or a plurality of devices.

  According to the present invention, it is possible to capture an image that gives a user a feeling that a virtual object exists in real space.

Front view of game device 10 in the open state Right side view of game device 10 in open state Left side view, front view, right side view, and rear view of game device 10 in the closed state Block diagram showing the internal configuration of the game apparatus 10 The figure which shows an example of the image displayed on upper LCD22 when the imaging | photography process which concerns on this embodiment is performed, and the marker 61 previously arrange | positioned in real space is imaged with the outer side imaging part 23. FIG. Diagram showing definition of coordinate system in virtual space The figure which shows a mode that the virtual character 51 has been arrange | positioned in virtual space. The figure which shows an example of the image displayed on upper LCD22 when the marker 61 is imaged from the opposite direction to FIG. The figure which shows a mode that the virtual character 52 moves according to operation with respect to the analog stick 15. FIG. The figure which shows an example of the image displayed on upper LCD22 when the virtual character 52 is made to pay attention to the direction of a virtual camera. The figure which shows the memory map of RAM of the game device 10 Main flowchart showing details of photographing processing according to the present embodiment Flow chart showing details of marker recognition processing (step S3) The flowchart which shows the detail of a preservation | save process (step S7) The flowchart which shows the detail of a character movement process (step S8) Flow chart showing details of attention processing (step S10) The figure which shows the positional relationship of the marker 61 and the outer side imaging part (left) 23a. The figure which shows the camera direction calculated by step S42 The figure which shows the XZ plane projection vector which projected the camera direction vector on the XZ plane The figure which shows the input direction of the analog stick 15 The figure which shows the XZ plane projection vector when the imaging direction of a virtual camera is a direction parallel to a Y-axis. The figure which shows the upward direction of a virtual camera, and the input direction of the analog stick 15 when a camera direction is parallel to a Y-axis. The figure which shows the gaze direction of a virtual character

(Configuration of game device)
Hereinafter, a game device according to an embodiment of the present invention will be described. 1 to 3 are plan views showing the appearance of the game apparatus 10. The game apparatus 10 is a portable game apparatus, and is configured to be foldable as shown in FIGS. 1 and 2 show the game apparatus 10 in an open state (open state), and FIG. 3 shows the game apparatus 10 in a closed state (closed state). FIG. 1 is a front view of the game apparatus 10 in the open state, and FIG. 2 is a right side view of the game apparatus 10 in the open state. The game apparatus 10 can capture an image with an imaging unit, display the captured image on a screen, and store data of the captured image. In addition, the game apparatus 10 is capable of executing a game program stored in a replaceable memory card or received from a server or another game apparatus, such as an image captured by a virtual camera set in a virtual space. An image generated by computer graphics processing can be displayed on the screen.

  First, the external configuration of the game apparatus 10 will be described with reference to FIGS. As shown in FIGS. 1 to 3, the game apparatus 10 includes a lower housing 11 and an upper housing 21. The lower housing 11 and the upper housing 21 are connected so as to be openable and closable (foldable). In the present embodiment, each of the housings 11 and 21 has a horizontally long rectangular plate shape, and is connected so as to be rotatable at the long side portions of each other.

  As shown in FIGS. 1 and 2, the upper long side portion of the lower housing 11 is provided with a protruding portion 11 </ b> A that protrudes in a direction perpendicular to the inner surface (main surface) 11 </ b> B of the lower housing 11. . In addition, the lower long side portion of the upper housing 21 is provided with a protruding portion 21A that protrudes from the lower side surface of the upper housing 21 in a direction perpendicular to the lower side surface. By connecting the protrusion 11A of the lower housing 11 and the protrusion 21A of the upper housing 21, the lower housing 11 and the upper housing 21 are foldably connected.

(Description of lower housing)
First, the configuration of the lower housing 11 will be described. As shown in FIGS. 1 to 3, the lower housing 11 includes a lower LCD (Liquid Crystal Display) 12, a touch panel 13, operation buttons 14A to 14L (FIGS. 1 and 3), an analog stick. 15, LED16A-16B, the insertion port 17, and the hole 18 for microphones are provided. Details of these will be described below.

  As shown in FIG. 1, the lower LCD 12 is housed in the lower housing 11. The lower LCD 12 has a horizontally long shape, and is arranged such that the long side direction coincides with the long side direction of the lower housing 11. The lower LCD 12 is disposed at the center of the lower housing 11. The lower LCD 12 is provided on the inner surface (main surface) of the lower housing 11, and the screen of the lower LCD 12 is exposed from an opening provided in the lower housing 11. When the game apparatus 10 is not used, it is possible to prevent the screen of the lower LCD 12 from becoming dirty or damaged by keeping the game apparatus 10 closed. The number of pixels of the lower LCD 12 may be, for example, 256 dots × 192 dots (horizontal × vertical). Unlike the upper LCD 22 described later, the lower LCD 12 is a display device that displays an image in a planar manner (not stereoscopically viewable). In the present embodiment, an LCD is used as the display device, but other arbitrary display devices such as a display device using EL (Electro Luminescence) may be used. Further, as the lower LCD 12, a display device having an arbitrary resolution can be used.

  As shown in FIG. 1, the game apparatus 10 includes a touch panel 13 as an input device. The touch panel 13 is mounted on the screen of the lower LCD 12. In the present embodiment, the touch panel 13 is a resistive film type touch panel. However, the touch panel is not limited to the resistive film type, and any type of touch panel such as a capacitance type can be used. In the present embodiment, the touch panel 13 having the same resolution (detection accuracy) as that of the lower LCD 12 is used. However, the resolution of the touch panel 13 and the resolution of the lower LCD 12 do not necessarily match. An insertion port 17 (dotted line shown in FIGS. 1 and 3D) is provided on the upper side surface of the lower housing 11. The insertion slot 17 can accommodate a touch pen 28 used for performing an operation on the touch panel 13. In addition, although the input with respect to the touchscreen 13 is normally performed using the touch pen 28, it is also possible to input with respect to the touchscreen 13 not only with the touch pen 28 but with a user's finger | toe.

  Each operation button 14A-14L is an input device for performing a predetermined input. As shown in FIG. 1, on the inner surface (main surface) of the lower housing 11, among the operation buttons 14A to 14L, a cross button 14A (direction input button 14A), a button 14B, b button 14C, x A button 14D, a y button 14E, a power button 14F, a select button 14J, a HOME button 14K, and a start button 14L are provided. The cross button 14 </ b> A has a cross shape, and has buttons for instructing up, down, left, and right directions. The button 14B, the button 14C, the button 14D, and the button 14E are arranged in a cross shape. Functions according to a program executed by the game apparatus 10 are appropriately assigned to the buttons 14A to 14E, the select button 14J, the HOME button 14K, and the start button 14L. For example, the cross button 14A is used for a selection operation or the like, and the operation buttons 14B to 14E are used for a determination operation or a cancel operation, for example. The power button 14F is used to turn on / off the power of the game apparatus 10.

  The analog stick 15 is a device that indicates a direction, and is provided in an upper area of the left area from the lower LCD 12 of the inner surface of the lower housing 11. As shown in FIG. 1, since the cross button 14A is provided in the lower region on the left side of the lower LCD 12, the analog stick 15 is provided above the cross button 14A. In addition, the analog stick 15 and the cross button 14A are designed to be operable with the thumb of the left hand holding the lower housing. Further, by providing the analog stick 15 in the upper region, the analog stick 15 is arranged where the thumb of the left hand holding the lower housing 11 is naturally positioned, and the cross button 14A has the thumb of the left hand slightly below. Arranged at shifted positions. The analog stick 15 is configured such that its key top slides parallel to the inner surface of the lower housing 11. The analog stick 15 functions according to a program executed by the game apparatus 10. For example, when a game in which a predetermined object appears in the three-dimensional virtual space is executed by the game apparatus 10, the analog stick 15 functions as an input device for moving the predetermined object in the three-dimensional virtual space. In this case, the predetermined object is moved in the direction in which the key top of the analog stick 15 slides. In addition, as the analog stick 15, an analog stick that allows analog input by being tilted by a predetermined amount in any direction of up / down / left / right and oblique directions may be used.

  Four buttons, a button 14B, b button 14C, x button 14D, and y button 14E, which are arranged in a cross shape, are arranged where the thumb of the right hand holding the lower housing 11 is naturally positioned. Also, these four buttons and the analog stick 15 are arranged symmetrically with the lower LCD 12 in between. Thus, depending on the game program, for example, a left-handed person can use these four buttons to input a direction instruction.

  A microphone hole 18 is provided on the inner surface of the lower housing 11. A microphone (see FIG. 4) as a voice input device described later is provided below the microphone hole 18, and the microphone detects sound outside the game apparatus 10.

  3A is a left side view of the game apparatus 10 in the closed state, FIG. 3B is a front view of the game apparatus 10 in the closed state, and FIG. 3C is a view of the game apparatus 10 in the closed state. FIG. 3D is a right side view, and FIG. 3D is a rear view of the game apparatus 10 in the closed state. As shown in FIGS. 3B and 3D, an L button 14 </ b> G and an R button 14 </ b> H are provided on the upper side surface of the lower housing 11. The L button 14 </ b> G is provided at the left end portion of the upper surface of the lower housing 11, and the R button 14 </ b> H is provided at the right end portion of the upper surface of the lower housing 11. Further, as shown in FIG. 3A, a volume button 14 </ b> I is provided on the left side surface of the lower housing 11. The volume button 14I is used to adjust the volume of a speaker provided in the game apparatus 10.

  As shown in FIG. 3A, an openable / closable cover portion 11 </ b> C is provided on the left side surface of the lower housing 11. A connector (not shown) for electrically connecting the game apparatus 10 and the data storage external memory 45 is provided inside the cover portion 11C. The data storage external memory 45 is detachably attached to the connector. The data storage external memory 45 is used, for example, for storing (saving) data of an image captured by the game apparatus 10. The connector and its cover portion 11 </ b> C may be provided on the right side surface of the lower housing 11.

  As shown in FIG. 3D, an insertion port 11D for inserting the game apparatus 10 and an external memory 44 in which a game program is recorded is provided on the upper side surface of the lower housing 11, and the insertion port Inside the 11D, a connector (not shown) for electrically detachably connecting to the external memory 44 is provided. When the external memory 44 is connected to the game apparatus 10, a predetermined game program is executed. The connector and its insertion port 11D may be provided on the other side surface of the lower housing 11 (for example, the right side surface).

  Further, as shown in FIG. 1 and FIG. 3C, on the lower side surface of the lower housing 11, the first LED 16A for notifying the user of the power ON / OFF state of the game apparatus 10 and the right side surface of the lower housing 11 Is provided with a second LED 16B for notifying the user of the wireless communication establishment status of the game apparatus 10. The game apparatus 10 can perform wireless communication with other devices, and the second LED 16B lights up when wireless communication is established. The game apparatus 10 is, for example, IEEE 802.11. It has a function of connecting to a wireless LAN by a method compliant with the b / g standard. A wireless switch 19 for enabling / disabling this wireless communication function is provided on the right side surface of the lower housing 11 (see FIG. 3C).

  Although not shown, the lower housing 11 stores a rechargeable battery that serves as a power source for the game apparatus 10, and the terminal is provided via a terminal provided on a side surface (for example, the upper side surface) of the lower housing 11. The battery can be charged.

(Description of upper housing)
Next, the configuration of the upper housing 21 will be described. As shown in FIGS. 1 to 3, the upper housing 21 includes an upper LCD (Liquid Crystal Display) 22, an outer imaging unit 23 (an outer imaging unit (left) 23a and an outer imaging unit (right) 23b). , An inner imaging unit 24, a 3D adjustment switch 25, and a 3D indicator 26 are provided. Details of these will be described below.

  As shown in FIG. 1, the upper LCD 22 is accommodated in the upper housing 21. The upper LCD 22 has a horizontally long shape and is arranged such that the long side direction coincides with the long side direction of the upper housing 21. The upper LCD 22 is disposed in the center of the upper housing 21. The area of the screen of the upper LCD 22 is set larger than the area of the screen of the lower LCD 12. Further, the screen of the upper LCD 22 is set to be horizontally longer than the screen of the lower LCD 12. That is, the ratio of the horizontal width in the aspect ratio of the screen of the upper LCD 22 is set larger than the ratio of the horizontal width in the aspect ratio of the screen of the lower LCD 12.

  The screen of the upper LCD 22 is provided on the inner surface (main surface) 21 </ b> B of the upper housing 21, and the screen of the upper LCD 22 is exposed from an opening provided in the upper housing 21. As shown in FIG. 2, the inner surface of the upper housing 21 is covered with a transparent screen cover 27. The screen cover 27 protects the screen of the upper LCD 22 and is integrated with the upper LCD 22 and the inner surface of the upper housing 21, thereby providing a sense of unity. The number of pixels of the upper LCD 22 may be, for example, 640 dots × 200 dots (horizontal × vertical). In the present embodiment, the upper LCD 22 is a liquid crystal display device. However, for example, a display device using EL (Electro Luminescence) may be used. In addition, a display device having an arbitrary resolution can be used as the upper LCD 22.

  The upper LCD 22 is a display device capable of displaying a stereoscopically visible image (also referred to as a stereoscopic image or a stereoscopic image). In the present embodiment, the left-eye image and the right-eye image are displayed using substantially the same display area. Specifically, the display device uses a method in which a left-eye image and a right-eye image are alternately displayed in a horizontal direction in a predetermined unit (for example, one column at a time). Alternatively, a display device of a type in which the left-eye image and the right-eye image are alternately displayed may be used. In this embodiment, the display device is capable of autostereoscopic viewing. A lenticular method or a parallax barrier method (parallax barrier method) is used so that the left-eye image and the right-eye image that are alternately displayed in the horizontal direction appear to be decomposed into the left eye and the right eye, respectively. In the present embodiment, the upper LCD 22 is a parallax barrier type. The upper LCD 22 uses the right-eye image and the left-eye image to display an image (stereoscopic image) that can be stereoscopically viewed with the naked eye. In other words, the upper LCD 22 displays a stereoscopic image (stereoscopically viewable) having a stereoscopic effect for the user by using the parallax barrier to make the left eye image visible to the user's left eye and the right eye image to the user's right eye. be able to. Further, the upper LCD 22 can invalidate the parallax barrier. When the parallax barrier is invalidated, the upper LCD 22 can display an image in a planar manner (in the sense opposite to the above-described stereoscopic view, the planar LCD is planar). (This is a display mode in which the same displayed image can be seen by both the right eye and the left eye). As described above, the upper LCD 22 is a display device capable of switching between a stereoscopic display mode for displaying a stereoscopically viewable image and a planar display mode for displaying an image in a planar manner (displaying a planar view image). . This display mode switching is performed by a 3D adjustment switch 25 described later.

  The outer imaging unit 23 is a general term for two imaging units (23 a and 23 b) provided on the outer surface (back surface opposite to the main surface on which the upper LCD 22 is provided) 21 D of the upper housing 21. The imaging directions of the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are both normal directions of the outer surface 21D. Further, all of these imaging units are designed in a direction 180 degrees opposite to the normal direction of the display surface (inner side surface) of the upper LCD 22. That is, the imaging direction of the outer imaging unit (left) 23a and the imaging direction of the outer imaging unit (right) 23b are parallel. The outer imaging unit (left) 23a and the outer imaging unit (right) 23b can be used as a stereo camera by a program executed by the game apparatus 10. Further, depending on the program, it is possible to use one of the two outer imaging units (23a and 23b) alone and use the outer imaging unit 23 as a non-stereo camera. Further, depending on the program, it is possible to perform imaging with an expanded imaging range by combining or complementarily using images captured by the two outer imaging units (23a and 23b). In the present embodiment, the outer imaging unit 23 includes two imaging units, an outer imaging unit (left) 23a and an outer imaging unit (right) 23b. The outer imaging unit (left) 23a and the outer imaging unit (right) 23b each include an imaging element (for example, a CCD image sensor or a CMOS image sensor) having a predetermined common resolution and a lens. The lens may have a zoom mechanism.

  As indicated by the broken line in FIG. 1 and the solid line in FIG. 3B, the outer imaging unit (left) 23 a and the outer imaging unit (right) 23 b that constitute the outer imaging unit 23 are arranged in the horizontal direction of the screen of the upper LCD 22. Arranged in parallel. That is, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are arranged so that a straight line connecting the two imaging units is parallel to the horizontal direction of the screen of the upper LCD 22. 1 indicate the outer imaging unit (left) 23a and the outer imaging unit (right) 23b existing on the outer surface opposite to the inner surface of the upper housing 21, respectively. As shown in FIG. 1, when the user views the screen of the upper LCD 22 from the front, the outer imaging unit (left) 23a is positioned on the left side and the outer imaging unit (right) 23b is positioned on the right side. When a program that causes the outer imaging unit 23 to function as a stereo camera is executed, the outer imaging unit (left) 23a captures an image for the left eye that is visually recognized by the user's left eye, and the outer imaging unit (right) 23b A right-eye image that is visually recognized by the user's right eye is captured. The interval between the outer imaging unit (left) 23a and the outer imaging unit (right) 23b is set to be approximately the interval between both eyes of the human, and may be set in a range of 30 mm to 70 mm, for example. In addition, the space | interval of the outer side imaging part (left) 23a and the outer side imaging part (right) 23b is not restricted to this range.

  In the present embodiment, the outer imaging unit (left) 23a and the outer imaging unit (right) 23 are fixed to the housing, and the imaging direction cannot be changed.

  Further, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are respectively arranged at positions symmetrical from the center with respect to the left-right direction of the upper LCD 22 (upper housing 21). That is, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are respectively arranged at symmetrical positions with respect to a line dividing the upper LCD 22 into two equal parts. In addition, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are on the upper side of the upper housing 21 in a state where the upper housing 21 is opened, and on the back side of the upper side of the screen of the upper LCD 22. Placed in. That is, the outer imaging unit (left) 23a and the outer imaging unit (right) 23b are the outer surfaces of the upper housing 21, and when the upper LCD 22 is projected on the outer surface, the upper image 22 is higher than the upper end of the projected upper LCD 22 screen. Placed in.

  As described above, the two image pickup units (23a and 23b) of the outer image pickup unit 23 are arranged at symmetrical positions from the center with respect to the left-right direction of the upper LCD 22, so that when the user views the upper LCD 22 from the front, the outer image pickup is performed. The imaging direction of the unit 23 can be matched with the user's line-of-sight direction. Further, since the outer imaging unit 23 is disposed at a position on the back side above the upper end of the screen of the upper LCD 22, the outer imaging unit 23 and the upper LCD 22 do not interfere inside the upper housing 21. Accordingly, it is possible to make the upper housing 21 thinner than in the case where the outer imaging unit 23 is disposed on the back side of the screen of the upper LCD 22.

  The inner imaging unit 24 is an imaging unit that is provided on the inner side surface (main surface) 21B of the upper housing 21 and has an inward normal direction of the inner side surface as an imaging direction. The inner imaging unit 24 includes an imaging element (for example, a CCD image sensor or a CMOS image sensor) having a predetermined resolution, and a lens. The lens may have a zoom mechanism.

  As shown in FIG. 1, the inner imaging unit 24 is disposed above the upper end of the upper LCD 22 and in the left-right direction of the upper housing 21 when the upper housing 21 is opened. Is arranged at the center position (on the line dividing the upper housing 21 (the screen of the upper LCD 22) into left and right halves). Specifically, as shown in FIG. 1 and FIG. 3B, the inner imaging unit 24 is an inner surface of the upper housing 21, and the left and right imaging units (outer imaging unit (left ) 23a and the outer imaging unit (right) 23b) are arranged at positions on the back side in the middle. That is, when the left and right imaging units of the outer imaging unit 23 provided on the outer surface of the upper housing 21 are projected onto the inner surface of the upper housing 21, the inner imaging unit 24 is provided in the middle of the projected left and right imaging units. It is done. A broken line 24 shown in FIG. 3B represents the inner imaging unit 24 existing on the inner surface of the upper housing 21.

  Thus, the inner imaging unit 24 images in the opposite direction to the outer imaging unit 23. The inner imaging unit 24 is provided on the inner surface of the upper housing 21 and on the back side of the intermediate position between the left and right imaging units of the outer imaging unit 23. Thereby, when the user views the upper LCD 22 from the front, the inner imaging unit 24 can capture the user's face from the front. Further, since the left and right imaging units of the outer imaging unit 23 and the inner imaging unit 24 do not interfere inside the upper housing 21, the upper housing 21 can be configured to be thin.

  The 3D adjustment switch 25 is a slide switch, and is a switch used to switch the display mode of the upper LCD 22 as described above. The 3D adjustment switch 25 is used to adjust the stereoscopic effect of a stereoscopically viewable image (stereoscopic image) displayed on the upper LCD 22. As shown in FIGS. 1 to 3, the 3D adjustment switch 25 is provided on the inner side surface and the right side end of the upper housing 21, and when the user views the upper LCD 22 from the front, the 3D adjustment switch 25 is visually recognized. It is provided at a position where it can be made. Moreover, the operation part of 3D adjustment switch 25 protrudes in both the inner surface and the right side surface, and can be visually recognized and operated from both. All switches other than the 3D adjustment switch 25 are provided in the lower housing 11.

  As shown in FIGS. 1 and 2, the 3D adjustment switch 25 is disposed so as to be visible from the front surface and the right side surface of the upper housing 21. The slider 25a of the 3D adjustment switch 25 can be slid to an arbitrary position in a predetermined direction (vertical direction), and the display mode of the upper LCD 22 is set or the stereoscopic effect of the stereoscopic image is adjusted according to the position of the slider 25a. May be. For example, the inter-camera distance of a virtual camera (virtual stereo camera) described later may be set according to the position of the slider 25a of the 3D adjustment switch 25. The positional relationship between the left-eye image captured by the left virtual camera of the virtual stereo camera and the right-eye image captured by the right virtual camera may be adjusted. Specifically, for example, when the slider 25a of the 3D adjustment switch 25 exists at the uppermost point (upward in FIGS. 1 and 2), the horizontal direction of the left-eye image and the right-eye image (the screen of the upper LCD 22) The positional deviation in the horizontal direction (left and right direction in FIG. 1) is set to the upper limit value. Thus, when the horizontal shift between the left-eye image and the right-eye image is set to the upper limit value, the parallax between the two images becomes larger. For this reason, when the user sees two images displayed on the upper LCD 22 through the parallax barrier, it appears that the image has jumped out further from the screen of the upper LCD 22. As described above, the parallax between the two images may be adjusted using the 3D adjustment switch 25.

  The 3D indicator 26 indicates whether or not the upper LCD 22 is in the stereoscopic display mode. The 3D indicator 26 is an LED, and lights up when the stereoscopic display mode of the upper LCD 22 is valid. Note that the 3D indicator 26 may be lit only when the upper LCD 22 is in the stereoscopic display mode and a program process for displaying a stereoscopic image is being executed. As shown in FIG. 1, the 3D indicator 26 is provided on the inner surface of the upper housing 21 and is provided near the screen of the upper LCD 22. For this reason, when the user views the screen of the upper LCD 22 from the front, the user can easily view the 3D indicator 26. Therefore, the user can easily recognize the display mode of the upper LCD 22 even when the user is viewing the screen of the upper LCD 22.

  A speaker hole 21 </ b> E is provided on the inner surface of the upper housing 21. Sound from a speaker 43 described later is output from the speaker hole 21E.

(Internal configuration of game device 10)
Next, with reference to FIG. 4, an internal electrical configuration of the game apparatus 10 will be described. FIG. 4 is a block diagram showing an internal configuration of the game apparatus 10. As shown in FIG. 4, in addition to the above-described units, the game apparatus 10 includes an information processing unit 31, a main memory 32, an external memory interface (external memory I / F) 33, an external memory I / F 34 for data storage, data It includes electronic components such as an internal memory 35 for storage, a wireless communication module 36, a local communication module 37, a real time clock (RTC) 38, an acceleration sensor 39, a power supply circuit 40, and an interface circuit (I / F circuit) 41. These electronic components are mounted on an electronic circuit board and accommodated in the lower housing 11 (or the upper housing 21).

  The information processing unit 31 is information processing means including a CPU (Central Processing Unit) 311 for executing a predetermined program, a GPU (Graphics Processing Unit) 312 for performing image processing, and the like. In the present embodiment, a predetermined program is stored in a memory (for example, the external memory 44 connected to the external memory I / F 33 or the data storage internal memory 35) in the game apparatus 10. The CPU 311 of the information processing section 31 executes processing (described later) (FIG. 12) by executing the predetermined program. Note that the program executed by the CPU 311 of the information processing unit 31 may be acquired from another device through communication with the other device. The information processing unit 31 includes a VRAM (Video RAM) 313. The GPU 312 of the information processing unit 31 generates an image in response to a command from the CPU 311 of the information processing unit 31 and draws it on the VRAM 313. Then, the GPU 312 of the information processing unit 31 outputs the image drawn in the VRAM 313 to the upper LCD 22 and / or the lower LCD 12, and the image is displayed on the upper LCD 22 and / or the lower LCD 12.

  A main memory 32, an external memory I / F 33, a data storage external memory I / F 34, and a data storage internal memory 35 are connected to the information processing section 31. The external memory I / F 33 is an interface for detachably connecting the external memory 44. The data storage external memory I / F 34 is an interface for detachably connecting the data storage external memory 45.

  The main memory 32 is a volatile storage unit used as a work area or a buffer area of the information processing unit 31 (the CPU 311). That is, the main memory 32 temporarily stores various data used for the image processing, and temporarily stores programs acquired from the outside (external memory 44, other devices, etc.). In the present embodiment, for example, a PSRAM (Pseudo-SRAM) is used as the main memory 32.

  The external memory 44 is a nonvolatile storage unit for storing a program executed by the information processing unit 31. The external memory 44 is composed of, for example, a read-only semiconductor memory. When the external memory 44 is connected to the external memory I / F 33, the information processing section 31 can read a program stored in the external memory 44. A predetermined process is performed by executing the program read by the information processing unit 31. The data storage external memory 45 is composed of a non-volatile readable / writable memory (for example, a NAND flash memory), and is used for storing predetermined data. For example, the data storage external memory 45 stores an image captured by the outer imaging unit 23 or an image captured by another device. When the data storage external memory 45 is connected to the data storage external memory I / F 34, the information processing section 31 reads an image stored in the data storage external memory 45 and applies the image to the upper LCD 22 and / or the lower LCD 12. An image can be displayed.

  The data storage internal memory 35 is configured by a readable / writable nonvolatile memory (for example, a NAND flash memory), and is used for storing predetermined data. For example, the data storage internal memory 35 stores data and programs downloaded by wireless communication via the wireless communication module 36.

  The wireless communication module 36 is, for example, IEEE 802.11. It has a function of connecting to a wireless LAN by a method compliant with the b / g standard. The local communication module 37 has a function of performing wireless communication with the same type of game device by a predetermined communication method (for example, infrared communication). The wireless communication module 36 and the local communication module 37 are connected to the information processing unit 31. The information processing unit 31 transmits / receives data to / from other devices via the Internet using the wireless communication module 36, and transmits / receives data to / from other game devices of the same type using the local communication module 37. You can do it.

  An acceleration sensor 39 is connected to the information processing unit 31. The acceleration sensor 39 detects the magnitude of linear acceleration (linear acceleration) along the three-axis (xyz-axis) direction. The acceleration sensor 39 is provided inside the lower housing 11. As shown in FIG. 1, the acceleration sensor 39 is configured such that the long side direction of the lower housing 11 is the x axis, the short side direction of the lower housing 11 is the y axis, and the inner side surface (main surface) of the lower housing 11. With the vertical direction as the z axis, the magnitude of linear acceleration on each axis is detected. The acceleration sensor 39 is, for example, an electrostatic capacitance type acceleration sensor, but other types of acceleration sensors may be used. The acceleration sensor 39 may be an acceleration sensor that detects a uniaxial or biaxial direction. The information processing section 31 receives data (acceleration data) indicating the acceleration detected by the acceleration sensor 39 and detects the posture and movement of the game apparatus 10. In the present embodiment, the information processing section 31 determines the attitude (tilt) of the game apparatus 10 based on the acceleration detected by the acceleration sensor 39.

  Further, the RTC 38 and the power supply circuit 40 are connected to the information processing unit 31. The RTC 38 counts the time and outputs it to the information processing unit 31. The information processing unit 31 calculates the current time (date) based on the time counted by the RTC 38. The power supply circuit 40 controls power from a power source (the rechargeable battery housed in the lower housing 11) of the game apparatus 10 and supplies power to each component of the game apparatus 10.

  In addition, an I / F circuit 41 is connected to the information processing unit 31. A microphone 42 and a speaker 43 are connected to the I / F circuit 41. Specifically, a speaker 43 is connected to the I / F circuit 41 via an amplifier (not shown). The microphone 42 detects the user's voice and outputs a voice signal to the I / F circuit 41. The amplifier amplifies the audio signal from the I / F circuit 41 and outputs the audio from the speaker 43. The touch panel 13 is connected to the I / F circuit 41. The I / F circuit 41 includes a voice control circuit that controls the microphone 42 and the speaker 43 (amplifier), and a touch panel control circuit that controls the touch panel. The voice control circuit performs A / D conversion and D / A conversion on the voice signal, or converts the voice signal into voice data of a predetermined format. The touch panel control circuit generates touch position data in a predetermined format based on a signal from the touch panel 13 and outputs it to the information processing unit 31. The touch position data indicates the coordinates of the position where the input is performed on the input surface of the touch panel 13. The touch panel control circuit reads signals from the touch panel 13 and generates touch position data at a rate of once per predetermined time. The information processing unit 31 can know the position where the input is performed on the touch panel 13 by acquiring the touch position data.

  The operation button 14 includes the operation buttons 14 </ b> A to 14 </ b> L and is connected to the information processing unit 31. From the operation button 14 to the information processing section 31, operation data indicating the input status (whether or not the button is pressed) for each of the operation buttons 14A to 14I is output. The information processing unit 31 acquires the operation data from the operation button 14 to execute processing according to the input to the operation button 14.

  The lower LCD 12 and the upper LCD 22 are connected to the information processing unit 31. The lower LCD 12 and the upper LCD 22 display images according to instructions from the information processing unit 31 (the GPU 312).

  Specifically, the information processing section 31 is connected to an LCD controller (not shown) of the upper LCD 22 and controls ON / OFF of the parallax barrier for the LCD controller. When the parallax barrier of the upper LCD 22 is ON, the right-eye image and the left-eye image stored in the VRAM 313 of the information processing unit 31 are output to the upper LCD 22. More specifically, the LCD controller alternately repeats the process of reading pixel data for one line in the vertical direction for the image for the right eye and the process of reading pixel data for one line in the vertical direction for the image for the left eye. Thus, the right-eye image and the left-eye image are read from the VRAM 313. As a result, the image for the right eye and the image for the left eye are divided into strip-like images in which pixels are arranged vertically for each line. The strip-like images for the right-eye image and the strip-like images for the left-eye image are alternately displayed. The image arranged on the upper LCD 22 is displayed on the screen. Then, when the user visually recognizes the image through the parallax barrier of the upper LCD 22, the right eye image is visually recognized by the user's right eye and the left eye image is visually recognized by the user's left eye. As a result, a stereoscopically viewable image is displayed on the screen of the upper LCD 22.

  The outer imaging unit 23 and the inner imaging unit 24 are connected to the information processing unit 31. The outer imaging unit 23 and the inner imaging unit 24 capture an image in accordance with an instruction from the information processing unit 31, and output the captured image data to the information processing unit 31. For example, the information processing unit 31 issues an imaging instruction to one of the outer imaging unit 23 and the inner imaging unit 24, and the imaging unit that receives the imaging instruction captures an image and sends the image data to the information processing unit 31. . For example, an imaging unit to be used is selected by a user's touch operation using the touch panel 13. The information processing unit 31 (CPU 311) detects that the imaging unit has been selected, and the information processing unit 31 issues an imaging instruction to the outer imaging unit 23 or the inner imaging unit 24.

  The 3D adjustment switch 25 is connected to the information processing unit 31. The 3D adjustment switch 25 transmits an electrical signal corresponding to the position of the slider 25 a to the information processing unit 31.

  The 3D indicator 26 is connected to the information processing unit 31. The information processing unit 31 controls lighting of the 3D indicator 26. In the present embodiment, the information processing section 31 lights the 3D indicator 26 when the upper LCD 22 is in the stereoscopic display mode. The above is the description of the internal configuration of the game apparatus 10.

  In addition, an angular velocity sensor 46 is connected to the information processing unit 31. The angular velocity sensor 46 detects an angular velocity around each axis (x axis, y axis, z axis). The game apparatus 10 can calculate the attitude of the game apparatus 10 in real space based on the angular velocities sequentially detected by the angular velocity sensor 46. Specifically, the game device 10 can calculate the rotation angle of the game device 10 around each axis by integrating the angular velocity around each axis detected by the angular velocity sensor 46 with time.

(Overview of shooting process)
Next, an outline of a shooting process (image process) executed in the game apparatus 10 according to the present embodiment will be described. FIG. 5 is a diagram illustrating an example of an image displayed on the upper LCD 22 when the outer imaging unit 23 captures an image of the marker 61 arranged in advance in the real space when the imaging process according to the present embodiment is performed. is there.

  As shown in FIG. 5, a marker 61 is arranged in the real space in the imaging direction of the outer imaging unit 23 of the game apparatus 10. The marker 61 is rectangular paper, and an arrow is drawn at the center thereof. The long side of the marker 61 is parallel to the direction of the arrow drawn at the center. The information processing unit 31 (CPU 311) of the game apparatus 10 detects the marker 61 included in the image by performing image processing such as pattern matching on the real image captured by the outer imaging unit 23, for example. Can do. When the marker 61 is detected in the actual image captured by the outer imaging unit 23, the virtual character is superimposed on the actual image captured by the outer imaging unit 23 and displayed on the upper LCD 22. When the marker 61 is not detected in the actual image captured by the outer imaging unit 23, a message indicating that the marker 61 is not detected is displayed on the upper LCD 22, and the virtual character is not displayed.

  Specifically, when the marker 61 is detected, the upper LCD 22 adds virtual characters 51 to 53 to a real image captured by the outer imaging unit 23 (an image capturing a real space including the marker 61 and the background), A cursor 55 and a display bar 56 are superimposed and displayed. The virtual characters 51 to 53 are characters existing in the virtual space, and are virtual objects that imitate people. More specifically, a state in which the virtual character 52 is on the marker 61 is displayed on the upper LCD 22. A virtual character 51 is displayed on the left side (left side on the screen) of the virtual character 52, and a virtual character 53 is displayed on the right side (right side on the screen) of the virtual character 52. The cursor 55 is for showing the currently selected virtual character. The display bar 56 is displayed to inform the user of the operation method of the game apparatus 10.

  The image displayed on the upper LCD 22 is a stereoscopically viewable image. That is, the actual image captured by the outer imaging unit 23 includes the left-eye actual image captured by the outer imaging unit (left) 23a and the right-eye actual image captured by the outer imaging unit (right) 23b. The virtual characters 51 to 53 are captured by a virtual stereo camera (left virtual camera and right virtual camera) existing in the virtual space, and are superimposed on the actual image. Specifically, a left-eye superimposed image is generated by superimposing an image obtained by capturing a virtual character with a left virtual camera on a left-eye real image captured by the outer imaging unit (left) 23a. In addition, an image obtained by capturing a virtual character with the right virtual camera is superimposed on the actual image for the right eye captured by the outer imaging unit (right) 23b, thereby generating a superimposed image for the right eye. Then, these two superimposed images are displayed on the upper LCD 22 so that the left-eye superimposed image and the right-eye superimposed image are visually recognized through the parallax barrier. Thereby, the user can visually recognize the image three-dimensionally.

  The virtual characters 51 to 53 are objects arranged in the virtual space. FIG. 6 is a diagram showing the definition of the coordinate system in the virtual space. The virtual space is defined by an XYZ coordinate system (marker coordinate system) with the center of the marker 61 as the origin. In the marker coordinate system, the Z axis is set in the same direction as the arrow of the marker 61, the X axis is set to the right (right direction) with respect to the arrow direction, and the Y axis is set to be vertically upward (upward) with respect to the marker 61. As described above, the virtual space coordinate system is defined with reference to the marker 61 arranged in the real space, whereby the real space and the virtual space can be associated with each other. Virtual characters 51 to 53 are arranged in the virtual space thus defined. For example, the virtual character 52 is arranged at the origin (0, 0, 0) of the marker coordinate system, and the direction (posture) is set to a direction obtained by rotating the X axis around the Y axis by 45 degrees. The virtual characters 51 and 53 are arranged at predetermined positions on the XZ plane, and their orientations are set to be the same as the virtual character 52.

  FIG. 7 is a diagram illustrating a state in which the virtual character 52 is arranged in the virtual space. As shown in FIG. 7, a virtual character 52 is placed at the origin of the marker coordinate system, and the virtual character 52 is captured by the left virtual camera 58a and the right virtual camera 58b placed in the virtual space. The left virtual camera 58a is for capturing an image of the virtual space visually recognized by the user's left eye, and the right virtual camera 58b is for capturing an image of the virtual space visually recognized by the user's right eye. is there. Specifically, the position and orientation of the left virtual camera 58a in the marker coordinate system coincide with the position and orientation of the outer imaging unit (left) 23a in real space. Further, the position and orientation of the right virtual camera 58b in the marker coordinate system coincide with the position and orientation of the outer imaging unit (right) 23b in the real space.

  As described above, when the virtual space is defined based on the marker 61 and the virtual characters 51 to 53 are arranged in the virtual space, when the position and the imaging direction of the outer imaging unit 23 are changed, the virtual LCD 51 is displayed on the upper LCD 22. The virtual characters 51 to 53 also change. FIG. 8 is a diagram illustrating an example of an image displayed on the upper LCD 22 when the marker 61 is imaged from the opposite direction to FIG. 5. In FIG. 8, when the position and orientation of the game apparatus 10 (outside imaging unit 23) are fixed, and the marker 61 shown in FIG. 5 is rotated 180 degrees around an axis passing through the center and perpendicular to the marker 61. The image displayed on the upper LCD 22 is shown. Alternatively, in FIG. 8, when the marker 61 shown in FIG. 5 is fixed and the game apparatus 10 is rotated 180 degrees around an axis that passes through the center of the marker 61 and is perpendicular to the marker 61, it is displayed on the upper LCD 22. An image is shown.

  As shown in FIG. 8, when the positional relationship (relative positional relationship; distance or posture) between the game apparatus 10 (outside imaging unit 23) and the marker 61 in the real space changes, the virtual character 51 displayed on the upper LCD 22 is changed. ~ 53 also changes. Specifically, the front of the virtual characters 51 to 53 is displayed in the positional relationship between the game apparatus 10 and the marker 61 shown in FIG. 5, but the positional relationship between the game apparatus 10 and the marker 61 shown in FIG. In the opposite direction), the rear surfaces of the virtual characters 51 to 53 are displayed. That is, when the position and the imaging direction of the outer imaging unit 23 are changed, the positions and imaging directions of the left and right virtual cameras are also changed according to the change. Thereby, the virtual characters 51 to 53 appear as if they exist on or around the marker 61 existing in the real space. Note that changing the display of the virtual character according to the positional relationship between the marker 61 and the game apparatus 10 can be realized using a known augmented reality technology.

  Next, the operation of the virtual character by the user will be described. First, an operation using the analog stick 15 will be described. FIG. 9 is a diagram illustrating how the virtual character 52 moves in response to an operation on the analog stick 15.

  As shown in FIG. 9, when the user slides the analog stick 15 in a predetermined direction (instructing the predetermined direction using the analog stick 15), the virtual character 52 selected by the cursor 55 moves in that direction. To do. For example, when the user uses the analog stick 15 to indicate the upward direction (when the analog stick 15 is slid upward), the virtual character 52 moves so as to walk on the XZ plane, and the upward direction of the screen of the upper LCD 22 Move to. Further, for example, when the user instructs the left direction using the analog stick 15, the virtual character 52 moves to walk on the XZ plane and moves to the left direction on the screen of the upper LCD 22. In this case, the virtual character 52 moves on the XZ plane of the virtual space. The moving direction in the virtual space does not necessarily match the moving direction on the screen of the upper LCD 22. That is, the moving direction on the screen of the upper LCD 22 is the moving direction on the display when the virtual space is imaged by the virtual camera and displayed on the upper LCD 22. In this way, the display position of the virtual character 52 changes in the direction designated by the user with the analog stick 15.

  Next, letting the virtual character draw attention to the game apparatus 10 (virtual camera) will be described. FIG. 10 is a diagram illustrating an example of an image displayed on the upper LCD 22 when a virtual character is focused on the direction of the virtual camera. In FIG. 10, when the marker 61 and the game apparatus 10 (outside imaging unit 23) are arranged in such a positional relationship that the right side surfaces of the virtual characters 51 to 53 are displayed, the virtual characters 51 to 53 are operated by the user. Shows a state in which attention is paid to the virtual cameras (58a and 58b). As described above, since the positions and orientations of the virtual camera and the outer imaging unit 23 coincide with each other, the attention of the virtual character to the virtual camera means the attention of the virtual character to the outer imaging unit 23. Specifically, when the right side of the virtual characters 51 to 53 is displayed as shown in FIG. 10, for example, when the user presses the x button 14D, all the virtual characters 51 to 53 are directed in the direction of the game apparatus 10. Turn to face. When the virtual characters 51 to 53 pay attention to the virtual camera, the line of sight of the virtual characters 51 to 53 is fixed to the virtual camera, that is, the outer imaging unit 23. Therefore, even if the user moves the game apparatus 10, the virtual characters 51 to 53 Always point in the direction of the virtual camera. That is, once the user presses the x button 14D, the camera attention mode is set, and the virtual characters 51 to 53 always pay attention to the virtual camera. Note that when the user presses the x button 14D again in the camera attention mode, the camera attention mode is canceled, and the face of each virtual character faces in a direction according to the direction of the body. Thus, the user can make the virtual character pay attention to the direction of the game apparatus 10 by an operation (input) on the operation means (input means).

  Note that the pose of the virtual character is changed by another operation. For example, when the user presses the a button 14B or the b button 14C, the selected virtual character changes from standing to sitting, or the virtual character raises or lowers the hand. In this way, the user can change the pose of each virtual character. Further, the selection change of the virtual character is performed by pressing the y button 14E. For example, when the y button 14E is pressed once, the cursor 55 moves above the virtual character 53 and the virtual character 53 is selected.

  Further, the facial expression of the virtual character is changed by another operation. For example, when the user presses the left / right button of the cross button 14A, the selected virtual character gets angry or laughs. In this way, the user can change the facial expression of each virtual character.

  As described above, the user can change the position, pose, and facial expression of each virtual character or cause the virtual character to focus on the camera by operating the buttons while viewing the upper LCD 22. While the user operates a button or moves the game apparatus 10 in the real space, the image displayed on the upper LCD 22 changes in real time. Thus, the user can freely determine the composition of the photograph to be taken to some extent by changing the position, pose, facial expression and orientation of each virtual character, the positional relationship between the game apparatus 10 and the marker 61, and the like. . Thereafter, the user can take (save) the image displayed on the upper LCD 22 by pressing a predetermined button. For example, when the L button 14G or the R button 14H is pressed, the image displayed when the button is pressed is displayed on the upper LCD 22 as a still image, and the still image is stored in the storage buffer (main memory 32). In the buffer area provided in the Thereafter, when a predetermined button is further pressed, the still image is stored in a nonvolatile storage means (data storage internal memory 35 or external memory 33).

  As described above, the user can take a picture of the virtual object by operating the virtual object as if the virtual object existed in the real space.

(Details of shooting process)
Next, with reference to FIG. 11 to FIG. 16, details of the photographing process according to the present embodiment will be described. First, main data stored in the main memory 32 and the VRAM 313 (hereinafter, these may be collectively referred to as a RAM) during the photographing process will be described. FIG. 11 is a diagram showing a RAM memory map of the game apparatus 10. As illustrated in FIG. 11, the RAM includes a shooting processing program 71, a real image for left eye 72L, a real image for right eye 72R, virtual character information 73, left virtual camera information 74L, right virtual camera information 74R, and a left virtual object image. 75L, right virtual object image 75R, operation information 76, save flag 77, left and right still image 78, camera attention flag 79, and the like are stored.

  The imaging processing program 71 is a program for causing the information processing unit 31 (CPU 311) to execute imaging processing shown in a flowchart to be described later.

  The left-eye real image 72L is an image obtained by imaging the real space, which is captured by the outer imaging unit (left) 23a.

  The right-eye real image 72R is an image obtained by imaging the real space, which is captured by the outer imaging unit (right) 23b.

  The virtual character information 73 is information related to the virtual characters 51 to 53. Specifically, the virtual character information 73 includes three-dimensional model data (polygon data) representing the shape of the virtual character, texture data representing the pattern of the virtual character, information on the position and orientation of the virtual character in the virtual space, and Contains information indicating whether it is currently selected. Each of the virtual characters 51 to 53 has these three-dimensional model data, texture data, information on the position and orientation, and information indicating whether or not it is currently selected. Each virtual character has information about the 3D model data, position, and posture for each part. For example, the virtual character 51 has 3D model data of the body (portion below the neck), information on the position and posture of the body, 3D model data of the head, and information on the position and posture of the head. ing. The position of the head and the position of the body have a predetermined relationship. Thus, by having the said information for every site | part, the direction of a head and a body can be varied. For example, when the body of the virtual character 51 is located at the origin in the virtual space and the body is oriented in the positive direction of the X axis, the face of the virtual character 51 is positioned at a predetermined position in the positive direction of the Y axis from the origin. The face can be directed in the positive direction of the Z axis.

  The left virtual camera information 74L is information representing the position and orientation of the left virtual camera 58a in the virtual space. Specifically, the left virtual camera information 74L is a matrix calculated based on the position and orientation of the marker 61 in the left-eye real image.

  The right virtual camera information 74R is information representing the position and orientation of the right virtual camera 58b in the virtual space. Specifically, the right virtual camera information 74R is a matrix calculated based on the position and orientation of the marker 61 in the real image for the right eye.

  The left virtual object image 75L is an image obtained by capturing a virtual character with the left virtual camera 58a.

  The right virtual object image 75R is an image obtained by capturing a virtual character with the right virtual camera 58b.

  The operation information 76 is information indicating an operation performed on the operation means (the operation buttons 14A to 14E, 14G, 14H, and the analog stick 15). When an operation on the operation means is performed, a signal indicating that the operation has been performed is transmitted to the CPU 311 and the operation information 76 is updated.

  The save flag 77 is a flag indicating that a captured image is saved, and is set to ON when, for example, a shooting instruction button (L button 14G or R button 14H) is pressed.

  The left and right still images 78 are images that are stored in response to pressing of the shooting instruction button (L button 14G or R button 14H), and include a left still image and a right still image.

  The camera attention flag 79 is a flag indicating whether or not the virtual character is in a mode in which attention is paid to the virtual camera (camera attention mode). When the camera attention mode is set, the camera attention flag 79 is set to ON. When the camera attention mode is not set, the camera attention flag 79 is set to OFF.

(Description of main flow)
Next, details of the photographing process according to the present embodiment will be described with reference to FIGS. FIG. 12 is a main flowchart showing details of the photographing process according to the present embodiment. When the power of the game apparatus 10 is turned on, the information processing section 31 (CPU 311) of the game apparatus 10 executes a startup program stored in a ROM (not shown), thereby initializing each unit such as the main memory 32. Is done. Next, a photographing processing program stored in a non-volatile memory (external memory 44 or the like; a computer-readable storage medium) is read into the RAM (specifically, the main memory 32), and the CPU 311 of the information processing unit 31 performs the processing. Program execution starts. The processing shown in the flowchart of FIG. 12 is performed by the information processing unit 31 (CPU 311 or GPU 312) after the above processing is completed. In FIG. 12, description of processing not directly related to the present invention is omitted. Further, the processing loop of step S1 to step S14 shown in FIG. 12 is repeatedly executed for each frame (for example, 1/30 second or 1/60 second, referred to as frame time).

  First, in step S1, the information processing section 31 determines whether or not shooting is in progress. Specifically, the information processing unit 31 refers to the RAM and determines whether or not the storage flag 77 is ON. If the determination result is negative, the information processing section 31 next executes a process of step S2. On the other hand, when the determination result is affirmative, the information processing section 31 next executes a process of step S7.

  In step S2, the information processing unit 31 acquires an actual camera image. Specifically, the information processing unit 31 transmits a command for capturing an image to the outer imaging unit 23. In response to the command, the outer imaging unit (left) 23a captures the real image 72L for the left eye, and the outer imaging unit (right) 23b captures the actual image for the right eye 72R. Then, the information processing section 31 acquires two captured images (the left-eye actual image 72L and the right-eye actual image 72R) and stores them in the RAM. Next, the information processing section 31 performs the process at step S3.

  In step S3, the information processing section 31 executes marker recognition processing. The marker recognition process is a process for recognizing the marker 61 included in the two images using the left-eye actual image 72L and the right-eye actual image 72R acquired in step S2. In the marker recognition process, virtual cameras (the left virtual camera 58a and the right virtual camera 58b) are set in the virtual space according to the marker recognition result. Details of the marker recognition process will be described later. After the marker recognition process, the process of step S4 is performed.

  In step S4, the information processing section 31 determines whether or not a marker has been recognized. The process of step S4 is a process of determining whether or not the marker 61 is normally recognized in the marker recognition process of step S3. Specifically, the information processing section 31 refers to the RAM and determines whether or not a detection result flag (described later) is ON. If the determination result is affirmative, the information processing section 31 next executes a process of step S5. On the other hand, when the determination result is negative, the information processing section 31 next executes a process of step S14.

  In step S5, the information processing section 31 determines whether or not the shooting button has been pressed. Specifically, the information processing section 31 refers to the operation information 76 stored in the RAM and determines whether or not the L button 14G or the R button 14H has been pressed. If the determination result is negative, the information processing section 31 next executes a process of step S6. On the other hand, when the determination result is affirmative, the information processing section 31 next executes a process of step S7.

  In step S <b> 6, the information processing section 31 refers to the operation information 76 and determines whether or not the analog stick 15 has been operated. If the determination result is affirmative, the information processing section 31 next executes a process of step S8. On the other hand, when the determination result is negative, the information processing section 31 next executes a process of step S9.

  On the other hand, in step S7, the information processing section 31 performs a storage process. The saving process is a process for saving (photographing) the image displayed on the upper LCD 22. Details of the storage process will be described later. After the process of step S7, the information processing section 31 next executes the process of step S14.

  In step S8, the information processing section 31 performs a character movement process. In the character movement process, the information processing section 31 moves the virtual character in the virtual space in response to an operation on the analog stick 15. Details of the character movement process will be described later. After the character movement process, the process of step S9 is performed.

  In step S9, the information processing section 31 determines whether or not the camera attention mode is set. Specifically, the information processing section 31 refers to the operation information 76 and determines whether or not the x button 14D is pressed or whether or not the camera attention flag 79 is ON. If the determination result is affirmative (when the x button 14D is pressed or the camera attention flag 79 is ON), the information processing section 31 sets the camera attention flag 79 to ON, and then proceeds to step S10. The attention process shown (described later) is executed. If the camera attention flag 79 is ON and the x button 14D is pressed, the information processing section 31 sets the camera attention flag 79 to OFF and executes non-attention processing (not shown). Then, the process of step S11 is executed. Although detailed description of the non-attention processing is omitted, in the non-attention processing, the head posture of each virtual character changed in the attention processing described later is restored. On the other hand, when the determination result is negative in step S9 (when the x button 14D is not pressed and the camera attention flag 79 is OFF), the information processing section 31 next executes the process of step S11.

  In step S <b> 10, the information processing section 31 performs attention processing. The attention process is a process for causing each virtual character to pay attention to the virtual camera, and is a process for directing the face of each virtual character toward the virtual camera. Details of the attention process will be described later. After the attention process, the process of step S11 is performed.

  In step S11, the information processing section 31 determines whether or not a predetermined button operation has been performed. Specifically, the information processing section 31 refers to the operation information 76 and determines whether any of the left and right buttons of the cross button 14A, the a button 14B, the b button 14C, and the y button 14E is pressed. Determine. If the determination result is affirmative, the information processing section 31 next executes a process of step S12. On the other hand, if the determination result is negative, the information processing section 31 next executes a process of step S13.

  In step S <b> 12, the information processing section 31 performs a process according to the button operation. Specifically, the information processing section 31 changes the facial expression of the selected virtual character when the left / right button of the cross button 14A is pressed. The expression of the virtual character is changed by the texture applied to the face of the virtual character. Therefore, the information processing section 31 changes the currently applied texture to another texture (stored in advance) in response to the pressing of the left / right button of the cross button 14A. Further, when the a button 14B or the b button 14C is pressed, the information processing section 31 changes the pose of the selected virtual character. Each virtual character is prepared in advance with a plurality of poses (a sitting state, a standing state, a hand-raised state, etc.), and the information processing unit 31 indicates that the a button 14B or the b button 14C has been pressed. Accordingly, one pose is selected from the plurality of poses. Further, when the y button 14E is pressed, the information processing unit 31 selects a virtual character different from the currently selected virtual character. For example, the virtual characters are selected in order from right to left on the display, or the virtual characters are selected in the order in which the virtual characters arranged in the virtual space are determined before the shooting process. Next, the information processing section 31 executes the process of step S13.

  In step S13, the information processing section 31 performs a virtual object image generation process. Specifically, the information processing section 31 generates the left virtual object image 75L by capturing the virtual space with the left virtual camera 58a, and captures the right virtual object image 75R by capturing the virtual space with the right virtual camera 58b. Generate. Then, the information processing section 31 stores the captured left virtual object image 75L and right virtual object image 75R in the RAM. Next, the information processing section 31 executes the process of step S14.

  In step S14, the information processing section 31 performs an output process. By executing the output process, a stereoscopically viewable image is displayed on the upper LCD 22. Specifically, when the process of step S13 is performed immediately before, the information processing unit 31 generates a superimposed image in which the real camera image acquired in step S2 and the virtual object image generated in step S13 are superimposed. . Specifically, the information processing section 31 generates a left-eye superimposed image by superimposing the left-eye real image 72L and the left virtual object image 75L, and superimposes the right-eye real image 72R and the right virtual object image 75R. A right-eye superimposed image is generated. Then, the information processing section 31 outputs the generated two superimposed images to the upper LCD 22. When the display settings of the cursor 55 and the display bar 56 are made, in addition to the virtual object image, an image showing the cursor 55 and the display bar 56 is superimposed on the left and right real camera images and displayed on the upper LCD 22. The On the other hand, when the cursor 55 and the display bar 56 are set not to be displayed, the cursor 55 and the display bar 56 are not displayed on the upper LCD 22.

  If the determination result in step S4 is negative, the information processing section 31 outputs the left-eye actual image 72L and the right-eye actual image 72R acquired in step S2 to the upper LCD 22. Further, when the save flag 77 is ON (when Step S7 is executed immediately before), the information processing section 31 displays the left and right still images (left and right still images 78) stored in the RAM at Step S7 on the upper LCD 22. Output. The information processing section 31 executes the process of step S1 again after the process of step S14. This is the end of the description of the flowchart shown in FIG.

  Although omitted in FIG. 12, when the still image (left and right still image 78) is stored in the RAM when the save flag 77 is ON, the still image is displayed on the upper LCD 22, and the user selects this still image. When the user confirms the content of the image and presses a predetermined button, the still image (left and right still image 78) stored in the RAM is a non-volatile storage means (data storage external memory 45, data storage internal memory 35, etc.). Saved in.

  Next, details of each process described above will be described.

(Explanation of marker recognition processing)
FIG. 13 is a flowchart showing details of the marker recognition process (step S3). First, in step S21, the information processing section 31 determines whether a marker has been detected. Here, it is determined by using the left-eye actual image 72L and the right-eye actual image 72R acquired in step S2 whether or not any of the two images includes a marker. Specifically, the information processing unit 31 performs image processing such as pattern matching on each image, and determines whether or not a marker is included. When the marker is not included in either one of the images, the determination result is negative, and the process of step S25 is executed next. On the other hand, if any image includes a marker, the determination result is affirmative, and then the process of step S22 is executed.

  In step S22, the information processing section 31 calculates the positional relationship between the real camera and the marker. Specifically, the information processing section 31 determines the outer imaging section based on the position and size of the marker included in the left-eye actual image 72L, the shape, the direction of the marker arrow, and the like. (Left) The positional relationship between 23a and the marker 61 existing in the real space is calculated. Here, the positional relationship between the outer imaging unit (left) 23a and the marker 61 is the other three-dimensional position and orientation when one of the marker 61 and the outer imaging unit (left) 23a is used as a reference. . That is, the positional relationship is a relative position and posture of the outer imaging unit (left) 23a with respect to the marker 61. Similarly, the information processing section 31 determines whether the marker included in the right-eye actual image 72R is based on the position, size, shape, and arrow direction of the marker in the right-eye actual image 72R. ) The positional relationship between 23b and the marker 61 existing in the real space is calculated.

  More specifically, in step S22, a marker coordinate system is set based on the marker recognition result, and the positional relationship between the marker 61 and the outer imaging unit (left) 23a is calculated. FIG. 17 is a diagram illustrating a positional relationship between the marker 61 and the outer imaging unit (left) 23a. As shown in FIG. 17, a marker coordinate system is set for the marker 61. Specifically, the origin of the marker coordinate system is set at the center of the marker 61. In addition, the Z axis of the marker coordinate system is set parallel to the direction of the arrow of the marker 61 (set parallel to the long side of the marker 61), and the X axis of the marker coordinate system is perpendicular to the direction of the arrow. Thus, it is set to the right with reference to the direction of the arrow (set parallel to the short side of the marker 61). Further, the Y axis is set vertically upward with respect to the marker 61 (set to the upward normal direction of the rectangular marker 61). The marker coordinate system is a coordinate system that defines a virtual space, and associates the real space with the virtual space. That is, the origin in the marker coordinate system is the origin of the virtual space and the center of the marker 61 in the real space.

  Then, a matrix representing the relative position and orientation of the outer imaging unit (left) 23a with respect to the marker 61 is calculated. As shown in FIG. 17, the matrix representing the position and orientation of the outer imaging unit (left) 23a is based on the coordinates expressed in the marker coordinate system in the virtual space, with the position and orientation of the outer imaging unit (left) 23a as a reference. It is the coordinate transformation matrix for transform | converting into the coordinate represented by the outer side imaging part (left) coordinate system. In step S23, a matrix representing the position and orientation of the outer imaging unit (left) 23a is calculated based on the markers included in the left-eye actual image 72L, and the outer imaging unit is calculated based on the markers included in the right-eye actual image 72R. (Right) A matrix representing the position and orientation of 23b is calculated.

  The outer imaging unit (left) 23a and the outer imaging unit (right) 23b are arranged so that their imaging directions are parallel and do not rotate with respect to the imaging direction. That is, the attitude of the outer imaging unit (left) 23a and the attitude of the outer imaging unit (right) 23b always match. The outer imaging unit (left) 23a and the outer imaging unit (right) 23b are arranged at a predetermined interval. For this reason, for example, when the position and orientation of the outer imaging unit (left) 23a are calculated based on the left-eye actual image 72L, the position of the outer imaging unit (right) 23b can be obtained without using the right-eye actual image 72R. And the posture can be calculated. After step S22, the information processing section 31 next executes the process of step S23.

  In step S23, the information processing section 31 determines the positions and orientations of the left and right virtual cameras. Here, the positions and orientations of the left and right virtual cameras coincide with the position and orientation of the outer imaging unit 23. That is, the position and orientation of the left virtual camera 58a in the virtual space are set to match the position and orientation of the outer imaging unit (left) 23a. Further, the position and orientation of the right virtual camera 58b in the virtual space are set so as to coincide with the position and orientation of the outer imaging section (right) 23b. Specifically, the position and orientation of the left virtual camera 58a are represented as a matrix (left view matrix), and the matrix representing the position and orientation of the outer imaging unit (left) 23a calculated in step S22 is the left virtual camera. Information 74L is stored in the RAM. Similarly, the position and orientation of the right virtual camera 58b are represented as a matrix (right view matrix), and the matrix representing the position and orientation of the outer imaging unit (right) 23b calculated in step S22 is the right virtual camera information 74R. Is stored in the RAM. As described above, since the posture of the outer imaging unit (left) 23a and the posture of the outer imaging unit (right) 23b always match, the posture of the left virtual camera 58a and the posture of the right virtual camera 58b also match. After step S23, the information processing section 31 next executes the process of step S24.

  In step S24, the information processing section 31 sets the detection result flag to ON and stores it in the RAM. After step S24, the information processing section 31 ends the marker recognition process shown in FIG.

  On the other hand, in step S25, the information processing section 31 sets the detection result flag to OFF and stores it in the RAM. Here, since no marker is detected in either the left-eye actual image 72L or the right-eye actual image 72R, the detection result flag indicating the marker detection result is set to OFF. After step S25, the information processing section 31 ends the marker recognition process shown in FIG.

(Description of saving process)
Next, details of the storage process (step S7) will be described. FIG. 14 is a flowchart showing details of the storage process (step S7).

  First, in step S31, the information processing section 31 refers to the RAM and determines whether or not the storage flag 77 is ON. If the determination result is negative, the information processing section 31 next executes a process of step S32. On the other hand, if the determination result is affirmative, the information processing section 31 next executes a process of step S34.

  In step S32, the information processing section 31 sets the cursor or the like to non-display. Here, the cursor 55 and the display bar 56 are set to non-display. Next, the information processing section 31 performs the process at step S33.

  In step S33, the information processing section 31 sets the save flag 77 to ON, and ends the save process.

  On the other hand, when it is determined in step S31 that the save flag is set to ON, the information processing section 31 executes the process of step S34. In step S34, the information processing section 31 stores the displayed image (that is, the left and right images) in the RAM as the left and right still images 78. Note that if the still image is already stored in the RAM, the process of step S34 is not performed. The image stored here is an image displayed on the upper LCD 22 in the previous frame, and is a left-eye superimposed image and a right-eye superimposed image on which the cursor or the like is not displayed. After executing the process of step S34, the information processing section 31 ends the storage process.

(Description of move processing)
Next, the details of the virtual character movement process (step S8) will be described. FIG. 15 is a flowchart showing details of the character movement process (step S8).

  First, in step S41, the information processing section 31 acquires a view matrix. The view matrix acquired here is the virtual camera in the virtual space (XYZ coordinate system) when a virtual camera (the virtual camera is not actually set in the virtual space) is set between the left and right virtual cameras. It is a matrix showing a position and a posture. Specifically, the information processing section 31 calculates the midpoint (the coordinate value of the marker coordinate system) between the left virtual camera 58a and the right virtual camera 58b based on the left virtual camera information 74L and the right virtual camera information 74R. . Then, the information processing section 31 acquires the posture of the left virtual camera 58a (which matches the posture of the right virtual camera 58b) from the left view matrix, and views a matrix indicating the calculated midpoint and the acquired posture. Calculate as a matrix. Next, the information processing section 31 performs the process at step S42.

  In step S42, the information processing section 31 calculates the camera direction based on the view matrix. The camera direction calculated here indicates an imaging direction of a virtual camera intermediate between the left and right virtual cameras (58a and 58b). FIG. 18 is a diagram illustrating the camera direction calculated in step S42. As shown in FIG. 18, the camera direction is the same as the imaging direction of the left and right virtual cameras, and is represented as a vector. Next, the information processing section 31 performs the process at step S43.

  In step S <b> 43, the information processing section 31 calculates the moving direction of the virtual character based on the camera direction and the operation information of the analog stick 15. Here, the moving direction of the virtual character in the marker coordinate system is calculated. Specifically, first, an XZ plane projection vector obtained by projecting a vector indicating the camera direction onto the XZ plane of the virtual space (marker coordinate system) is calculated. FIG. 19 is a diagram illustrating an XZ plane projection vector obtained by projecting the camera direction vector onto the XZ plane. Further, the information processing unit 31 calculates the input direction of the analog stick 15 based on the operation information of the analog stick 15. FIG. 20 is a diagram illustrating the input direction of the analog stick 15. As shown in FIG. 20, the input direction of the analog stick 15 is a direction designated by the analog stick 15 (direction in which the analog stick 15 is slid), and is represented as a vector on the xy plane (see FIG. 1). . The information processing section 31 refers to the operation information 76 and calculates a vector indicating the input direction of the analog stick 15 as shown in FIG.

  Next, the information processing section 31 calculates the moving direction of the virtual character based on the XZ plane projection vector and the input direction vector of the analog stick 15. Specifically, the movement direction is calculated by rotating the XZ plane projection vector around the Y axis in accordance with the input to the analog stick 15. More specifically, the information processing section 31 calculates an angle θ between the input direction vector and the y axis as shown in FIG. 20, and the XZ plane projection vector is rotated by the angle θ around the Y axis (see FIG. 19). The direction of rotation is calculated as the moving direction. In this way, the information processing section 31 calculates the moving direction of the virtual character.

  For example, when the upward direction (upward direction (y-axis positive direction) shown in FIG. 1) is instructed using the analog stick 15, the XZ plane projection vector is not rotated (rotation angle is 0 degree), and the moving direction is , In the same direction as the XZ plane projection vector. When the downward direction (downward direction (y-axis negative direction) shown in FIG. 1) is designated using the analog stick 15, the XZ plane projection vector is rotated by 180 degrees around the Y-axis, and the moving direction is XZ. The direction is opposite to the plane projection vector. When the analog stick 15 is used to specify the right direction (the right direction (x-axis positive direction) shown in FIG. 1), the XZ plane projection vector is rotated 90 degrees around the Y axis, and the moving direction is rotated. It faces right with respect to the previous XZ plane projection vector.

  When the camera direction calculated in step S42 is a direction parallel to the Y axis, that is, when the imaging direction of the virtual camera is a direction parallel to the Y axis, the moving direction of the virtual character is the above camera direction. Not calculated using. The camera direction is parallel to the Y axis (Y axis negative direction) when the marker 61 is imaged by the outer imaging unit 23 from directly above. When the camera direction is parallel to the Y axis, the XZ plane projection vector obtained by projecting the camera direction onto the XZ plane is a point vector having no direction as shown in FIG. FIG. 21 is a diagram illustrating an XZ plane projection vector when the imaging direction of the virtual camera is a direction parallel to the Y axis. For this reason, the moving direction of the virtual character cannot be calculated using the camera direction as described above. Therefore, in this case, instead of the camera direction (imaging direction of the virtual camera), the upward direction of the virtual camera (Yc axis direction shown in FIGS. 19 and 21) is calculated. FIG. 22 is a diagram illustrating the upward direction of the virtual camera and the input direction of the analog stick 15 when the camera direction is parallel to the Y axis. As shown in FIG. 22, the moving direction of the virtual character is calculated based on the calculated upward direction of the virtual camera and the input direction of the analog stick 15. That is, the upward vector of the virtual camera is projected onto the XZ plane, and the projected vector is angled around the Y axis of the marker coordinate system (the angle formed by the upward direction of the virtual camera and the input direction of the analog stick 15). The direction rotated at is calculated as the moving direction of the virtual character.

  The movement direction thus calculated is the movement direction in the virtual space, that is, the movement direction in the marker coordinate system. When the virtual character moves in the calculated movement direction in the virtual space, the virtual character moves in the same direction as the direction (slid direction) instructed using the analog stick 15 when displayed on the upper LCD 22. . That is, the moving direction of the virtual character on the display coincides with the direction in which the analog stick 15 is slid. Next, the information processing section 31 performs the process at step S44.

  In step S44, the information processing section 31 moves the virtual character in the movement direction calculated in step S43. Specifically, the information processing unit 31 moves the virtual character by updating the position of the selected virtual character in the virtual space among the virtual character information stored in the RAM. That is, the information processing section 31 adds the vector of the moving direction calculated in step S43 (the vector has a predetermined size) to the position of the selected virtual character in the virtual space, thereby The character position is updated and stored in the RAM. Next, the information processing section 31 performs the process at step S45.

  In step S45, the information processing section 31 determines whether or not the virtual character has moved out of the predetermined range. Specifically, the information processing section 31 determines whether or not the position of the virtual character moved in step S44 exceeds a predetermined range centered on the origin of the marker coordinate system. If the determination result is affirmative, the information processing section 31 next executes a process of step S46. On the other hand, if the determination result is negative, the information processing section 31 ends the movement process shown in FIG.

  In step S46, the information processing section 31 corrects the position of the virtual character within the predetermined range. Here, it is processing for preventing the virtual character from being too far away from the origin of the marker coordinate system. When the virtual character is too far from the origin of the marker coordinate system and exceeds the imaging range of the virtual camera, the user attempts to image a position away from the marker 61 using the outer imaging unit 23 in order to see the character. Then, the marker 61 is not included in the imaging range of the outer imaging unit 23, the marker coordinate system cannot be defined, and the user cannot see the virtual character. In order to prevent this, in step S46, the position of the virtual character moved in step S44 is corrected within the predetermined range. Thereafter, the information processing section 31 ends the movement process shown in FIG.

(Explanation of attention processing)
Next, details of the attention process (step S10) will be described. FIG. 16 is a flowchart showing details of attention processing (step S10). First, in step S51, the information processing section 31 acquires the position of the virtual character's body. Specifically, the information processing section 31 refers to the virtual character information 73, and among the plurality of virtual characters, the position of the body of the virtual character that has not yet undergone attention processing (steps S51 to S56) To get. As described above, the virtual character information 73 includes position data and position information of each part along with shape data and the like for each virtual character. The information processing unit 31 refers to the virtual character information 73 and performs virtual processing. Get the body position of the character. Next, the information processing section 31 performs the process at step S52.

  In step S52, the information processing section 31 calculates the visual line direction of the virtual character based on the position of the virtual camera and the position of the head of the virtual character. FIG. 23 is a diagram illustrating the viewing direction of the virtual character. Here, as shown in FIG. 23, the “virtual camera position” is an intermediate position between the position of the left virtual camera 58a and the position of the right virtual camera 58b. Specifically, the information processing unit 31 determines that “the position of the virtual camera is based on the view matrix (a matrix indicating the midpoint of the left and right virtual cameras and the same posture as the left and right virtual cameras) acquired in step S41. "(The midpoint of the left and right virtual cameras). In addition, the information processing unit 31 calculates the position of the head of the virtual character based on the position of the body of the virtual character acquired in step S51. The body and head of the virtual character have a predetermined positional relationship, and the information processing unit 31 adds the predetermined vector to the body position of the virtual character to add the position of the head of the virtual character. Is calculated. Then, the information processing section 31 calculates a vector from the position of the head of the virtual character toward the position of the virtual camera, and stores it in the RAM as the visual line direction of the virtual character. Next, the information processing section 31 performs the process at step S53. Since the positional relationship between the body and head of the virtual character differs for each pose of the virtual character, the position of the head of the virtual character is calculated according to the current pose of the virtual character.

  In step S53, the information processing section 31 calculates a rotation angle around each axis (XY axis) of the head based on the line-of-sight direction calculated in step S52. Here, the rotation angles around the X axis and the Y axis necessary for making the head (face) of the virtual character face the line of sight are calculated. Specifically, the information processing unit 31 refers to the virtual character information 73 and acquires a matrix (current body posture matrix) indicating the posture of the virtual character's body. Next, the information processing unit 31 calculates a head posture matrix (head posture matrix in the visual line direction) when the head of the virtual character is directed in the visual line direction. Then, the information processing unit 31 calculates a rotation matrix indicating the rotation of the head based on the current body posture matrix and the head posture matrix in the line-of-sight direction, and based on the rotation matrix, The rotation angle around the axes (X axis and Y axis) is calculated. Next, the information processing section 31 performs the process at step S54.

  In step S54, the information processing section 31 determines whether or not the rotation angle around each axis calculated in step S53 is outside a predetermined range. Specifically, the information processing section 31 determines whether or not the rotation angle around the X axis (vertical direction) is outside a predetermined range (for example, a range of −60 degrees to 60 degrees), and around the Y axis (horizontal direction). It is determined whether the rotation angle is outside a predetermined range (for example, a range of -80 degrees to 80 degrees). If either one of the rotation angle around the X axis and the rotation angle around the Y axis is outside the predetermined range (that is, the determination result is affirmative), the information processing section 31 next executes the process of step S55. On the other hand, if neither the rotation angle around the X axis nor the rotation angle around the Y axis is outside the predetermined range (that is, the determination result is negative), the information processing section 31 next executes the process of step S56. .

  In step S55, the information processing section 31 corrects the rotation angle around each axis within the predetermined range. In step S55, when the rotation angle around each axis is outside the predetermined range, the rotation angle around each axis is corrected to the upper limit value or the lower limit value so as to be within the predetermined range. For example, when the rotation angle around the X axis exceeds the upper limit value (60 degrees) of the predetermined range, the information processing section 31 corrects the rotation angle around the X axis to the upper limit value (60 degrees). Thus, by correcting the rotation angle of the head, for example, the head rotates 180 degrees horizontally with respect to the body, or the head rotates 150 degrees vertically with respect to the body. As described above, the head can be prevented from rotating unnaturally. Next, the information processing section 31 executes the process of step S56.

  In step S56, the information processing section 31 determines a head posture matrix. Specifically, when the rotation angle of the head is corrected in the immediately preceding step S55, the information processing section 31 calculates the posture matrix of the head when the head is rotated at the corrected rotation angle. To do. If the rotation angle of the head is not corrected in step S55, the information processing section 31 determines the head posture matrix in the line-of-sight direction calculated in step S53 as a new head posture matrix. The character information 73 is updated. Next, the information processing section 31 executes the process of step S57.

  In step S57, the information processing section 31 determines whether or not attention processing (processing shown in steps S51 to S56) has been executed for all virtual characters. If the determination result is negative, the information processing section 31 performs the process of step S51 again. When the determination result is affirmative, the information processing section 31 ends the attention process illustrated in FIG.

Note that the above-described processing is merely an example, and may be used, for example, in the order of the processing or for each operation.

Buttons, threshold values (values used in S45 and S54) used for each determination, and the like may be any.

  As described above, in the present embodiment, the real space and the virtual space are associated with each other, and the virtual object is arranged in the virtual space. Then, by taking an image of the real space with the outer imaging unit 23 and taking an image of the virtual object with the virtual camera, it is possible to take a photograph that can give the user a sense that the virtual object exists in the real space. it can.

  In the present embodiment, the user can move a virtual character existing in the virtual space, change the pose of the virtual character, or cause the camera to focus on the virtual character. Thereby, it is possible to give the user a feeling that the virtual character actually exists in the real space. For example, the posture of the virtual character's face (head) is changed so that the face of the virtual character faces the virtual camera (outside imaging unit 23). Thereby, it is possible to give a feeling that the virtual character exists in the real space. Furthermore, in this embodiment, since a virtual character is imaged by the left and right virtual cameras, a three-dimensional image is obtained. Then, the posture of the head of the virtual character is set so as to face the center of the left and right virtual cameras. For this reason, it is possible to give the user a feeling that the virtual character exists in the real space.

  Further, when the virtual character is focused on the camera, the angle at which the head of the virtual character rotates is limited to a predetermined range (laterally). Specifically, the movable range of the head of the virtual character is limited to a range of −80 degrees to 80 degrees in the horizontal direction and a range of −60 degrees to 60 degrees in the vertical direction. For example, when a virtual character is imaged from directly behind and the virtual character is focused on the camera, the face of the virtual character rotates in the oblique direction (80 degrees in the horizontal direction with respect to the body direction) without facing directly behind Facing the direction). Thus, by limiting the rotation angle of the head, it is possible to obtain a feeling that the virtual character exists in the real space and faces the direction of the camera. That is, it is not normally considered that the human neck turns 180 degrees backward, and the motion of the virtual character is a natural motion close to such a real human motion. Therefore, it is possible to give the user a feeling that the virtual character actually exists in the real space.

(Modification)
In the present embodiment, a virtual character imitating a person as a virtual object is arranged in the virtual space. In another embodiment, the virtual object may be any object, for example, an object imitating an animal, an object imitating a plant, an object of a robot, or the like.

  In this embodiment, the virtual character is moved or the pose is changed by a button operation. In other embodiments, patterns of positions and postures of a plurality of virtual characters may be stored in advance. For example, a pattern in which five virtual characters are arranged in a straight line and each has a different pose, or a pattern in which five virtual characters are arranged to form a pentagon and each has the same pose May be stored in advance. And according to operation by a user, you may change the position and attitude | position of each virtual character into the pattern memorize | stored previously.

  In this embodiment, the virtual character is moved, the expression or pose is changed, or attention is given in accordance with the button operation. However, the button operation is merely an example, and which button is pressed. In this case, what kind of operation is performed may be arbitrarily set. In another embodiment, a plurality of icons for operation are displayed on the lower LCD 12, and the icons are selected by touch operation on the screen (touch panel 13), and the position, posture, expression, etc. of the virtual character are changed. Alternatively, the virtual character may be noted by a touch operation. In other embodiments, the acceleration sensor 39 and the angular velocity sensor 46 may function as input means. Specifically, based on the detection results of the acceleration sensor 39 and the angular velocity sensor 46, a predetermined gesture operation on the game apparatus 10 (for example, an operation of moving the game apparatus 10 up and down or tilting the game apparatus 10). It may be determined whether or not the virtual character has been performed, and the virtual character may be moved or noticed according to the determination result.

  In the present embodiment, the position and orientation of the virtual character are changed by the user's operation (although the movement process and the attention process are performed), but in other embodiments, the position or orientation of the virtual character is It may be changed by an operation (either the movement process or the attention process may be executed).

  In this embodiment, in the attention process, the posture of the face (head) is changed so that the face of the virtual character faces the virtual camera. In other embodiments, the entire body may be directed toward the virtual camera. In other embodiments, not the face but only the eyes may be directed toward the virtual camera, or arms and legs may be directed toward the virtual camera. That is, in the attention process, one part of the body of the virtual character may be directed toward the virtual camera.

  In this embodiment, the face of the virtual character is directed to the center of the left and right virtual cameras in the attention process. However, in another embodiment, the face of the virtual character may be directed to one of the left and right virtual cameras. .

  In the present embodiment, a stereoscopically viewable image (stereoscopic image) is captured and stored. That is, in this embodiment, real space is imaged by two cameras (the outer imaging unit (left) 23a and the outer imaging unit (right) 23b), and the virtual space is captured by two virtual cameras (the left virtual camera 58a and the right virtual camera). 58b) and imaged and saved. In other embodiments, a planar image may be captured and stored. That is, in another embodiment, real space may be imaged with one real camera, and virtual space may be imaged with one virtual camera.

  In the present embodiment, the positional relationship (relative position and orientation) between the outer imaging unit 23 and the marker 61 is calculated by imaging the marker 61 with the outer imaging unit 23. A virtual space (marker coordinate system) was defined based on the calculated positional relationship. In other embodiments, not only the marker 61 but also other objects may be recognized from the captured image, and the positional relationship may be calculated. For example, a predetermined object (for example, a chair or table that exists in the real space) may be detected by image recognition such as pattern matching, and the positional relationship is calculated to define the virtual space. Also good. That is, in another embodiment, even if a specific target (the marker or the predetermined object) existing in the real space is recognized from the captured image, the positional relationship between the specific target and the outer imaging unit 23 is calculated. Good. Further, the specific object may not be recognized based on the image captured by the real camera, but may be recognized by other recognition means (for example, an ultrasonic sensor or the like).

  In other embodiments, the position and orientation of the actual camera may be detected by a GPS, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, or the like. Then, based on the detected position and orientation, the position and orientation of the virtual camera in the virtual space may be set, and an image captured by the real camera and an image captured by the virtual camera may be superimposed. For example, a virtual object may be arranged at a predetermined position in the virtual space corresponding to the real space, and the position of the real space corresponding to the predetermined position in the virtual space may be photographed with a real camera from a predetermined direction. In this case, the position and posture of the real camera can be detected by GPS or posture detection means (acceleration sensor, angular velocity sensor, geomagnetic sensor, etc.), and the virtual camera is placed in the virtual space so as to match the position and posture of the real camera. Can be set. In this way, by superimposing the image captured by the real camera and the image captured by the virtual camera, it is possible to capture a photograph in which the virtual object exists at that position in the real space.

  In this embodiment, augmented reality is realized by using the video see-through method. That is, in the present embodiment, the image captured by the outer imaging unit 23 and the image captured by the virtual camera (left and right virtual cameras) are superimposed to generate a superimposed image, and the superimposed image is displayed on the upper LCD 22. It was done. In another embodiment, augmented reality may be realized by an optical see-through method. For example, a user wears a head-mound display equipped with a camera for detecting a marker arranged in the real space, and the user can visually recognize the real space through a display unit corresponding to a lens portion of glasses. This display unit is made of a material that can be directly guided to the eyes of the user through the real space. Furthermore, a virtual object image generated by a computer can be displayed on the display unit.

  In another embodiment, the above-described shooting processing method is not limited to a game device, and is applied to an arbitrary electronic device such as a PDA (Personal Digital Assistant), a high-function mobile phone, a camera (a camera as a device), or the like. May be.

  In this embodiment, an LCD capable of displaying a stereoscopic image with the naked eye is used as the display device. In other embodiments, the present invention is applicable even when stereoscopic display is performed using glasses such as a time-division method, a deflection method, and an anaglyph method (red and blue glasses method).

  Further, in another embodiment, a plurality of information processing apparatuses connected to be communicable by wire or wireless, for example, share and process each process, and are configured as an imaging processing system that realizes the above-described imaging processing method. May be. For example, the outer imaging unit 23 may be separated from the information processing apparatus, and the outer imaging unit 23 may be connected to the information processing apparatus by wire or wirelessly. Alternatively, the position and orientation of the outer imaging unit 23 may be detected by detection means separated from the information processing apparatus, and the detection result may be transmitted to the information processing apparatus.

  Moreover, in the said embodiment, the process by the flowchart mentioned above was performed because the information processing part 31 of the game device 10 performed a predetermined | prescribed program. In another embodiment, part or all of the above processing may be performed by a dedicated circuit included in the game apparatus 10.

  The shooting processing program is not limited to the memory, but may be stored in a computer-readable storage medium such as an optical disk or a magnetic disk and provided to the game apparatus 10. Further, for example, the shooting process program may be stored in a RAM of a server on a network, the game apparatus 10 may be connected to the network, and the shooting process program may be provided to the game apparatus 10.

10 Game device 11 Lower housing 12 Lower LCD
13 Touch Panel 14 Operation Buttons 15 Analog Stick 16 LED
21 Upper housing 22 Upper LCD
23 Outside imaging unit 23a Outside imaging unit (left)
23b Outside imaging unit (right)
24 Inner imaging unit 25 3D adjustment switch 26 3D indicator 28 Touch pen 31 Information processing unit 311 CPU
312 GPU
32 Main memory 51, 52, 53 Virtual character 55 Cursor 56 Display bar 58a Left virtual camera 58b Right virtual camera 61 Marker

Claims (16)

An image processing program executed in a computer of an image processing apparatus that generates an image obtained by superimposing a virtual object image on an image captured by a real camera, the computer comprising:
Real image acquisition means for acquiring a real image of a subject imaged by the real camera;
Position and orientation information acquisition means for acquiring position and orientation information corresponding to the position and orientation of the real camera in real space;
Virtual camera setting means for setting the position and orientation of the virtual camera based on the position and orientation information acquired by the position and orientation information acquisition means in the virtual space;
Virtual object changing means for changing at least one of the position and orientation of at least a part of the virtual object arranged in the virtual space in response to an input to the input means by the user;
Virtual object image generation means for generating a virtual object image by imaging with the virtual camera at least one of the position and orientation of the virtual object changing means by the virtual object changing means;
A superimposed image generating unit that generates a superimposed image by superimposing the real image acquired by the real image acquiring unit and the virtual object image generated by the virtual object image generating unit; and
An image processing program that functions as a storage unit that stores the superimposed image in a storage unit in response to a photographing instruction from a user.   The position / orientation information acquisition unit detects a specific object included in the actual image acquired by the actual image acquisition unit, and based on the detection result, the position of the actual camera relative to the specific object and The image processing program according to claim 1, wherein the posture is acquired. The image processing apparatus is connectable to a display means;
The superimposed image generation unit repeatedly generates a superimposed image obtained by superimposing the real image acquired by the real image acquisition unit and the virtual object image generated by the virtual object image generation unit until the shooting instruction is performed. And
The image processing program according to claim 1 or 2, further causing a display control unit to display the superimposed image generated by the superimposed image generation unit on the display unit until the photographing instruction is performed. The virtual object includes a first part and a second part,
The virtual object changing unit includes a part changing unit that changes a posture of the first part so that the first part of the virtual object faces the virtual camera in response to an input to the input unit by a user. The image processing program according to any one of claims 1 to 3.   In the case where the posture of the first part is changed so that the first part faces the virtual camera, the posture of the first part with respect to the second part is out of the first range. 5. The image processing program according to claim 4, wherein the posture of the first part is set to an upper limit or a lower limit of the first range.   When the user inputs to the input unit, the part changing unit is configured so that the first part of the virtual object faces the virtual camera until the imaging instruction is performed. The image processing program according to claim 4, wherein the posture is repeatedly changed. The real camera is a stereo camera,
The real image acquisition means acquires a real image for the left eye captured by the real camera for the left eye of the stereo camera and a real image for the right eye captured by the real camera for the right eye of the stereo camera,
The position and orientation information acquisition means acquires position and orientation information corresponding to the position and orientation of the real camera for the left eye and the real camera for the right eye of the stereo camera,
The virtual camera setting unit sets the position and posture of the left virtual camera according to the position and orientation information of the real camera for the left eye acquired by the position and orientation information acquisition unit, and is acquired by the position and orientation information acquisition unit. Set the position and orientation of the right virtual camera according to the position and orientation information of the real camera for the right eye,
The part changing means changes the posture of the first part so that the first part of the virtual object is directed to an intermediate position between the left virtual camera and the right virtual camera,
The virtual object image generating means captures the virtual object whose posture of the first part has been changed by the part changing means with the left virtual camera and the right virtual camera, so that a virtual object image for the left eye and a right eye Virtual object image for
The superimposed image generating means includes a left-eye superimposed image obtained by superimposing the left-eye real image and the left-eye virtual object image, and a right-eye superimposed image obtained by superimposing the right-eye real image and the right-eye virtual object image. The image processing program according to claim 4, wherein the image processing program generates a superimposed image.   The image processing program according to claim 1, wherein the virtual object changing unit includes a moving unit that moves the virtual object in response to an input to the input unit by a user.   The virtual object changing unit moves the virtual object so that the direction of the input and the moving direction of the virtual object on the display unit are the same in response to an input to the input unit by a user. The image processing program according to claim 3, comprising means.   When the moving means exceeds the second range in the virtual space when the virtual object is moved in response to an input to the input means by a user, the position of the virtual object is corrected to the second range. The image processing program according to claim 8.   The image processing program according to claim 1, wherein the virtual object changing unit includes a pose changing unit that changes a pose of the virtual object in accordance with an input to the input unit by a user. Causing the computer to further function as a selection means for selecting one virtual object from a plurality of virtual objects arranged in the virtual space;
12. The virtual object changing unit according to claim 1, wherein the virtual object changing unit changes at least one of a position and a posture of at least a part of the virtual object selected by the selection unit in response to an input to the input unit by a user. An image processing program according to any one of the above. Further causing the computer to function as selection means for selecting one virtual object from a plurality of virtual objects arranged in the virtual space,
The superimposed image generating unit generates a superimposed image in which a cursor indicating that the virtual object has been selected by the selecting unit is further superimposed,
The virtual object changing means changes at least one of a position and a posture of at least a part of the virtual object selected by the selection means in response to an input to the input means by a user,
The image processing program according to claim 3, wherein the storage unit stores the superimposed image in the storage unit while hiding the cursor in accordance with a photographing instruction from a user. An image processing apparatus that generates an image obtained by superimposing a virtual object image on an image captured by a real camera,
Real image acquisition means for acquiring a real image of a subject imaged by the real camera;
Position and orientation information acquisition means for acquiring position and orientation information according to the position and orientation of the real camera in real space;
Virtual camera setting means for setting the position and orientation of the virtual camera based on the position and orientation information acquired by the position and orientation information acquisition means in the virtual space;
Virtual object changing means for changing at least one of the position and orientation of at least a part of the virtual object arranged in the virtual space in response to an input to the input means by the user;
Virtual object image generation means for generating a virtual object image by imaging the virtual object whose position and orientation have been changed by the virtual object changing means with the virtual camera;
Superimposed image generation means for generating a superimposed image obtained by superimposing the real image acquired by the real image acquisition means and the virtual object image generated by the virtual object image generation means;
An image processing apparatus comprising: a storage unit that stores the superimposed image in a storage unit in response to a shooting instruction from a user. An image processing system for generating an image obtained by superimposing a virtual object image on an image captured by a real camera,
Real image acquisition means for acquiring a real image of a subject imaged by the real camera;
Position and orientation information acquisition means for acquiring position and orientation information according to the position and orientation of the real camera in real space;
Virtual camera setting means for setting the position and orientation of the virtual camera based on the position and orientation information acquired by the position and orientation information acquisition means in the virtual space;
Virtual object changing means for changing at least one of the position and orientation of at least a part of the virtual object arranged in the virtual space in response to an input to the input means by the user;
Virtual object image generation means for generating a virtual object image by imaging the virtual object whose position and orientation have been changed by the virtual object changing means with the virtual camera;
Superimposed image generation means for generating a superimposed image obtained by superimposing the real image acquired by the real image acquisition means and the virtual object image generated by the virtual object image generation means;
An image processing system comprising: a storage unit that stores the superimposed image in a storage unit in response to a shooting instruction from a user. An image processing method for generating an image in which a virtual object image is superimposed on an image captured by a real camera,
A real image acquisition step of acquiring a real image of a subject imaged by the real camera;
A position and orientation information acquisition step for acquiring position and orientation information corresponding to the position and orientation of the real camera in real space;
In a virtual space, a virtual camera setting step for setting the position and orientation of the virtual camera based on the position and orientation information acquired in the position and orientation information acquisition step;
A virtual object changing step of changing at least one of a position and a posture of at least a part of the virtual object arranged in the virtual space in response to an input to the input means by the user;
A virtual object image generation step of generating a virtual object image by imaging the virtual object whose position and orientation are changed in the virtual object changing step with the virtual camera;
A superimposed image generating step of generating a superimposed image obtained by superimposing the real image acquired in the real image acquiring step and the virtual object image generated in the virtual object image generating step;
An image processing method comprising: a storage step of storing the superimposed image in a storage unit in response to a photographing instruction from a user.

Images