JP4265814B1 - Posture calculation device, posture calculation program, game device, and game program - Google Patents

Abstract

An attitude of an input device is accurately calculated using a gyro sensor.
An attitude calculation apparatus acquires data from an input device including at least a gyro sensor, an acceleration sensor, and an imaging unit, and calculates the attitude of the input device. First, the attitude calculation device calculates the attitude of the input device based on the angular velocity detected by the gyro sensor. Next, the posture is corrected based on acceleration data detected by the acceleration sensor. Further, the posture is corrected based on a predetermined image to be imaged captured by the imaging means.
[Selection] Figure 14

Description

  The present invention relates to an attitude calculation device or an attitude calculation program, and more specifically to an attitude calculation device or an attitude calculation program for calculating the attitude of an input device.

Conventionally, techniques for calculating the attitude of an input device using an acceleration sensor and a gyro sensor have been considered. For example, Patent Document 1 describes a game device that uses an input control device including an acceleration sensor and a gyro sensor. This game device controls a sword possessed by a game character in accordance with the movement of an input control device. Specifically, data on the motion of swinging the sword is created based on the output of the acceleration sensor, and data on the posture of the sword is created based on the output of the gyro sensor.
JP 2000-308756 A

  When the attitude is calculated using a gyro sensor as in Patent Document 1, an error occurs between the calculated attitude and the actual attitude of the input control device. When the input control device moves slowly, the input controller's angular velocity is not detected by the gyro sensor. Conversely, when the input controller moves excessively, the input controller's angular velocity exceeds the detectable range of the gyro sensor. This is because there is a case where it may occur. An error may also occur when the angular velocity changes suddenly during a period shorter than the output interval of the angular velocity data. Then, since the angular velocity error is accumulated with time in the posture calculated from the angular velocity, the posture error becomes large. In Patent Document 1, since the posture error calculated from the gyro sensor is not assumed, the posture may not be accurately calculated.

  Therefore, an object of the present invention is to provide an attitude calculation device or an attitude calculation program that can accurately calculate the attitude of an input device using a gyro sensor.

  The present invention employs the following configuration in order to solve the above problems. Note that the reference numerals in parentheses and supplementary explanations in this section show the correspondence with the embodiments described later in order to help understanding of the present invention, and do not limit the present invention.

The present invention acquires data (operation data) from an input device (8) provided with at least a gyro sensor (55, 56), an acceleration sensor (37), and an image pickup means (image pickup device 40). This is a posture calculation device (game device 3) that calculates the posture of the player. The posture calculation device includes posture calculation means (CPU 10 that executes step S4; hereinafter, only the step number will be described), first correction means (S5), and second correction means (S6). The posture calculation means calculates a first posture (first posture data 68) of the input device based on an angular velocity (angular velocity data 63) detected by the gyro sensor. The first correction means sets the posture of the input device when the direction of acceleration indicated by the acceleration data (64) detected by the acceleration sensor is vertically downward as the second posture (angle θ2), and determines the magnitude of the acceleration. Is corrected so that the first posture approaches the second posture so that the first posture (angle θ1) becomes closer to the second posture as the magnitude of gravity acceleration becomes closer to the magnitude (S18, S21). ). The second correction unit sets the posture of the input device calculated from the orientation and / or position of the imaging target in the image captured by the imaging unit as a third posture (third posture data 76), and Correction is performed to bring the first posture closer to the third posture at a predetermined rate (S37). Furthermore, the first correction means performs the correction only when the difference between the magnitude of acceleration and the magnitude of gravitational acceleration is smaller than a predetermined reference (S15). The second correction unit relates to a roll direction when the imaging direction of the imaging unit is used as a reference in the third posture based on a direction of a predetermined imaging target in an image captured by the imaging unit. While calculating the posture (roll posture component data 73), based on the position of the imaging target in the image, out of the third posture, the pitch with respect to the imaging direction of the imaging means and / or Alternatively, the posture (yaw posture component data 74) regarding the yaw direction (in the embodiment, only the yaw direction) is calculated.

  According to the above, the first posture calculated using the gyro sensor is corrected based on the acceleration data and the image to be imaged. Thus, the posture error calculated from the gyro sensor can be corrected, so that the posture of the input device can be accurately calculated using the gyro sensor. In addition, according to the above, the first posture can be easily corrected using the second posture by bringing the first posture closer to the second posture determined from the acceleration detected by the acceleration sensor. it can. Similarly, the first posture can be easily corrected using the third posture by bringing the first posture closer to the third posture determined from the image captured by the imaging unit.

  Furthermore, according to the above, the closer the magnitude of the acceleration detected by the acceleration sensor is to the magnitude of the gravitational acceleration, the stronger the second attitude is reflected in the corrected attitude. Here, the detection result of the acceleration sensor is presumed that the closer the magnitude of the acceleration is to the gravitational acceleration, the more accurately the direction of the gravitational acceleration is indicated, and the more accurate the second posture is. Is done. According to the above invention, when the second posture is accurate, the first posture is corrected to be closer to the second posture, and when the second posture is not very accurate, the first posture is corrected. Since the posture is not corrected much, the corrected posture can be calculated more accurately.

  Further, according to the above, when the difference between the magnitude of the acceleration detected by the acceleration sensor and the magnitude of the gravitational acceleration is equal to or greater than a predetermined reference, the first correction unit does not perform correction. That is, when it is estimated that the detection result of the acceleration sensor does not accurately indicate the direction of gravitational acceleration, the correction using the (inaccurate) second posture is not performed. The posture can be calculated more accurately.

  The correction using the captured image cannot be performed unless the imaging unit captures the imaging target, and therefore the second correction process cannot be performed and the second correction process can be performed. There is. Here, if the second correction unit performs correction so that the corrected first posture matches the third posture, the second correction processing is performed from a state where the second correction processing cannot be performed. When the state shifts to a state where it can be performed, the first posture may be rapidly changed by the second correction process. On the other hand, in the above-described invention, the first posture is brought close to the third posture at a predetermined rate, so that the first posture can be prevented from changing suddenly even in the above case. . Therefore, it is possible to prevent the user from feeling uncomfortable with the operation due to a sudden change in the first posture, and the operability of the input device can be improved.

  Further, according to the above, since the posture in the roll direction and the pitch and / or the yaw direction is calculated based on the orientation of the imaging target in the image captured by the imaging means, the posture in each rotation direction is accurately calculated. can do. Therefore, it is possible to calculate the third posture more accurately, and in turn, it is possible to more accurately calculate the corrected first posture.

  The second correction unit determines whether the imaging unit is oriented in a direction in which the predetermined imaging target can be imaged based on the first posture, and determines the direction in which the imaging unit can capture the predetermined imaging target. You may correct | amend only when it is suitable.

  Here, there is a possibility that the imaging unit may erroneously detect an object that is not an imaging target (assuming that it is an imaging target) when the predetermined imaging target is not directed in a direction in which imaging can be performed. At this time, there is a possibility that the correction is not correctly performed as a result of calculating the inaccurate third posture. On the other hand, according to the above invention, since the correction is not performed in the above case, it is possible to prevent the first posture from being corrected using the inaccurate third posture, As a result, the first posture can be calculated more accurately.

  The second correction unit may correct the first posture after correction by the first correction unit.

  According to the above, after the correction using the first posture determined from the detection result of the acceleration sensor is performed first, the correction using the second posture determined from the captured image is performed. That is, in the final correction result, the correction using the second posture is reflected with priority. In general, the second posture is considered to be more accurate than the first posture. Therefore, the correction using the second posture is prioritized and reflected in the correction result, thereby providing a more accurate posture. Can be calculated.

  The present invention may also be provided as a game device that performs game processing using the first posture corrected by the posture calculation device according to the above invention as the posture of the input device.

  According to the above, the game player can perform the game operation by using the accurate first posture corrected based on the acceleration data and the image to be captured as the game input. The operability of game operations can be improved.

  In addition, the present invention may be implemented in the form of an attitude calculation program or a game program that causes a computer of an information processing apparatus to function as each of the above means.

  According to the present invention, the posture calculated using the gyro sensor is corrected based on the acceleration data and / or the image to be imaged. Thus, the posture error calculated from the gyro sensor can be corrected, so that the posture of the input device can be accurately calculated using the gyro sensor.

[Overall configuration of game system]
With reference to FIG. 1, a game system 1 including a game apparatus, which is an example of an attitude calculation apparatus according to an embodiment of the present invention, will be described. FIG. 1 is an external view of the game system 1. Hereinafter, the game apparatus and the game program of the present embodiment will be described using a stationary game apparatus as an example. In FIG. 1, the game system 1 includes a television receiver (hereinafter simply referred to as “TV”) 2, a game device 3, an optical disk 4, an input device 8, and a marker unit 6. In the present system, game processing is executed by the game device 3 based on a game operation using the input device 8.

  An optical disk 4 that is an example of an information storage medium that can be used interchangeably with the game apparatus 3 is detachably inserted into the game apparatus 3. The optical disc 4 stores a game program to be executed on the game apparatus 3. An insertion slot for the optical disk 4 is provided on the front surface of the game apparatus 3. The game apparatus 3 executes a game process by reading and executing a game program stored in the optical disc 4 inserted into the insertion slot.

  A television 2 which is an example of a display device is connected to the game apparatus 3 via a connection cord. The television 2 displays a game image obtained as a result of the game process executed in the game device 3. In addition, a marker unit 6 is installed around the screen of the television 2 (upper side of the screen in FIG. 1). The marker unit 6 includes two markers 6R and 6L at both ends thereof. The marker 6R (same for the marker 6L) is specifically one or more infrared LEDs, and outputs infrared light toward the front of the television 2. The marker unit 6 is connected to the game apparatus 3, and the game apparatus 3 can control lighting of each infrared LED included in the marker unit 6.

  The input device 8 gives operation data indicating the content of the operation performed on the own device to the game device 3. In the present embodiment, the input device 8 includes a controller 5 and a gyro sensor unit 7. Although details will be described later, the input device 8 has a configuration in which a gyro sensor unit 7 is detachably connected to the controller 5. The controller 5 and the game apparatus 3 are connected by wireless communication. In the present embodiment, for example, Bluetooth (registered trademark) technology is used for wireless communication between the controller 5 and the game apparatus 3. In other embodiments, the controller 5 and the game apparatus 3 may be connected by wire.

[Internal configuration of game device 3]
Next, the internal configuration of the game apparatus 3 will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration of the game apparatus 3. The game apparatus 3 includes a CPU 10, a system LSI 11, an external main memory 12, a ROM / RTC 13, a disk drive 14, an AV-IC 15 and the like.

  The CPU 10 executes a game process by executing a game program stored on the optical disc 4, and functions as a game processor. The CPU 10 is connected to the system LSI 11. In addition to the CPU 10, an external main memory 12, a ROM / RTC 13, a disk drive 14, and an AV-IC 15 are connected to the system LSI 11. The system LSI 11 performs processing such as control of data transfer between components connected thereto, generation of an image to be displayed, and acquisition of data from an external device. The internal configuration of the system LSI will be described later. The volatile external main memory 12 stores a program such as a game program read from the optical disc 4 or a game program read from the flash memory 17, or stores various data. Used as a work area and buffer area. The ROM / RTC 13 includes a ROM (so-called boot ROM) in which a program for starting the game apparatus 3 is incorporated, and a clock circuit (RTC: Real Time Clock) that counts time. The disk drive 14 reads program data, texture data, and the like from the optical disk 4 and writes the read data to an internal main memory 11e or an external main memory 12 described later.

  The system LSI 11 includes an input / output processor (I / O processor) 11a, a GPU (Graphics Processor Unit) 11b, a DSP (Digital Signal Processor) 11c, a VRA 1d, and an internal main memory 11e. Although not shown, these components 11a to 11e are connected to each other by an internal bus.

  The GPU 11b forms part of a drawing unit and generates an image according to a graphics command (drawing command) from the CPU 10. The VRAM 11d stores data (data such as polygon data and texture data) necessary for the GPU 11b to execute the graphics command. When an image is generated, the GPU 11b creates image data using data stored in the VRAM 11d.

  The DSP 11c functions as an audio processor, and generates sound data using sound data and sound waveform (tone color) data stored in the internal main memory 11e and the external main memory 12.

  The image data and audio data generated as described above are read out by the AV-IC 15. The AV-IC 15 outputs the read image data to the television 2 via the AV connector 16, and outputs the read audio data to the speaker 2 a built in the television 2. As a result, an image is displayed on the television 2 and a sound is output from the speaker 2a.

  The input / output processor 11a performs transmission / reception of data to / from components connected to the input / output processor 11a and downloads data from an external device. The input / output processor 11a is connected to the flash memory 17, the wireless communication module 18, the wireless controller module 19, the expansion connector 20, and the memory card connector 21. An antenna 22 is connected to the wireless communication module 18, and an antenna 23 is connected to the wireless controller module 19.

  The input / output processor 11a is connected to the network via the wireless communication module 18 and the antenna 22, and can communicate with other game devices and various servers connected to the network. The input / output processor 11a periodically accesses the flash memory 17 to detect the presence / absence of data that needs to be transmitted to the network. If there is such data, the input / output processor 11a communicates with the network via the wireless communication module 18 and the antenna 22. Send. Further, the input / output processor 11a receives data transmitted from other game devices and data downloaded from the download server via the network, the antenna 22 and the wireless communication module 18, and receives the received data in the flash memory 17. Remember. By executing the game program, the CPU 10 reads out the data stored in the flash memory 17 and uses it in the game program. In the flash memory 17, in addition to data transmitted and received between the game apparatus 3 and other game apparatuses and various servers, save data (game result data or intermediate data) of the game played using the game apparatus 3 May be stored.

  The input / output processor 11a receives operation data transmitted from the controller 5 via the antenna 23 and the wireless controller module 19, and stores (temporarily stores) the data in the buffer area of the internal main memory 11e or the external main memory 12.

  Further, an expansion connector 20 and a memory card connector 21 are connected to the input / output processor 11a. The expansion connector 20 is a connector for an interface such as USB or SCSI, and connects a medium such as an external storage medium, a peripheral device such as another controller, or a wired communication connector. By connecting, communication with the network can be performed instead of the wireless communication module 18. The memory card connector 21 is a connector for connecting an external storage medium such as a memory card. For example, the input / output processor 11a can access an external storage medium via the expansion connector 20 or the memory card connector 21 to store data in the external storage medium or read data from the external storage medium.

  The game apparatus 3 is provided with a power button 24, a reset button 25, and an eject button 26. The power button 24 and the reset button 25 are connected to the system LSI 11. When the power button 24 is turned on, power is supplied to each component of the game apparatus 3 via an AC adapter (not shown). When the reset button 25 is pressed, the system LSI 11 restarts the boot program for the game apparatus 3. The eject button 26 is connected to the disk drive 14. When the eject button 26 is pressed, the optical disk 4 is ejected from the disk drive 14.

[Configuration of Input Device 8]
Next, the input device 8 will be described with reference to FIGS. FIG. 3 is a perspective view showing an external configuration of the input device 8. FIG. 4 is a perspective view showing an external configuration of the controller 5. 3 is a perspective view of the controller 5 as seen from the upper rear side, and FIG. 4 is a perspective view of the controller 5 as seen from the lower front side.

  3 and 4, the controller 5 includes a housing 31 formed by plastic molding, for example. The housing 31 has a substantially rectangular parallelepiped shape whose longitudinal direction is the front-rear direction (the Z-axis direction shown in FIG. 3), and is a size that can be gripped with one hand of an adult or a child as a whole. The player can perform a game operation by pressing a button provided on the controller 5 and moving the controller 5 itself to change its position and posture.

  The housing 31 is provided with a plurality of operation buttons. As shown in FIG. 3, a cross button 32a, a first button 32b, a second button 32c, an A button 32d, a minus button 32e, a home button 32f, a plus button 32g, and a power button 32h are provided on the upper surface of the housing 31. It is done. In the present specification, the upper surface of the housing 31 on which these buttons 32a to 32h are provided may be referred to as a “button surface”. On the other hand, as shown in FIG. 4, a recess is formed on the lower surface of the housing 31, and a B button 32i is provided on the rear inclined surface of the recess. A function corresponding to the game program executed by the game apparatus 3 is appropriately assigned to each of the operation buttons 32a to 32i. The power button 32h is for remotely turning on / off the main body of the game apparatus 3. The home button 32 f and the power button 32 h are embedded in the upper surface of the housing 31. This can prevent the player from pressing the home button 32f or the power button 32h by mistake.

  A connector 33 is provided on the rear surface of the housing 31. The connector 33 is used to connect another device (for example, the gyro sensor unit 7 or another controller) to the controller 5. Further, locking holes 33a are provided on both sides of the connector 33 on the rear surface of the housing 31 in order to prevent the other devices from being easily detached.

  A plurality (four in FIG. 3) of LEDs 34 a to 34 d are provided behind the upper surface of the housing 31. Here, the controller type (number) is assigned to the controller 5 to distinguish it from other main controllers. The LEDs 34a to 34d are used for the purpose of notifying the player of the controller type currently set in the controller 5 and notifying the player of the remaining battery level of the controller 5. Specifically, when a game operation is performed using the controller 5, any one of the plurality of LEDs 34a to 34d is turned on according to the controller type.

  Further, the controller 5 has an imaging information calculation unit 35 (FIG. 6), and a light incident surface 35a of the imaging information calculation unit 35 is provided on the front surface of the housing 31 as shown in FIG. The light incident surface 35a is made of a material that transmits at least infrared light from the markers 6R and 6L.

  Between the first button 32b and the home button 32f on the upper surface of the housing 31, a sound release hole 31a for releasing sound from the speaker 49 (FIG. 5) built in the controller 5 is formed.

  Next, the internal structure of the controller 5 will be described with reference to FIGS. 5 and 6 are diagrams showing the internal structure of the controller 5. FIG. FIG. 5 is a perspective view showing a state in which the upper housing (a part of the housing 31) of the controller 5 is removed. FIG. 6 is a perspective view showing a state in which the lower casing (a part of the housing 31) of the controller 5 is removed. The perspective view shown in FIG. 6 is a perspective view of the substrate 30 shown in FIG.

  In FIG. 5, a substrate 30 is fixed inside the housing 31, and operation buttons 32 a to 32 h, LEDs 34 a to 34 d, an acceleration sensor 37, an antenna 45, and a speaker 49 are provided on the upper main surface of the substrate 30. Etc. are provided. These are connected to a microcomputer (microcomputer) 42 (see FIG. 6) by wiring (not shown) formed on the substrate 30 and the like. In the present embodiment, the acceleration sensor 37 is disposed at a position shifted from the center of the controller 5 with respect to the X-axis direction. This makes it easier to calculate the movement of the controller 5 when the controller 5 is rotated about the Z axis. The acceleration sensor 37 is disposed in front of the center of the controller 5 in the longitudinal direction (Z-axis direction). Further, the controller 5 functions as a wireless controller by the wireless module 44 (FIG. 6) and the antenna 45.

  On the other hand, in FIG. 6, an imaging information calculation unit 35 is provided at the front edge on the lower main surface of the substrate 30. The imaging information calculation unit 35 includes an infrared filter 38, a lens 39, an imaging element 40, and an image processing circuit 41 in order from the front of the controller 5. These members 38 to 41 are respectively attached to the lower main surface of the substrate 30.

  Further, the microcomputer 42 and the vibrator 48 are provided on the lower main surface of the substrate 30. The vibrator 48 is, for example, a vibration motor or a solenoid, and is connected to the microcomputer 42 by wiring formed on the substrate 30 or the like. The controller 48 is vibrated by the operation of the vibrator 48 according to the instruction of the microcomputer 42. As a result, a so-called vibration-compatible game in which the vibration is transmitted to the hand of the player holding the controller 5 can be realized. In the present embodiment, the vibrator 48 is disposed slightly forward of the housing 31. That is, by arranging the vibrator 48 on the end side of the center of the controller 5, the entire controller 5 can be vibrated greatly by the vibration of the vibrator 48. The connector 33 is attached to the rear edge on the lower main surface of the substrate 30. 5 and 6, the controller 5 includes a crystal resonator that generates a basic clock of the microcomputer 42, an amplifier that outputs an audio signal to the speaker 49, and the like.

  Further, the gyro sensor unit 7 includes gyro sensors (gyro sensors 55 and 56 shown in FIG. 7) that detect angular velocities around three axes. The gyro sensor unit 7 is detachably attached to the connector 33 of the controller 5. A plug (plug 53 shown in FIG. 7) that can be connected to the connector 33 is provided at the front end of the gyro sensor unit 7 (end on the Z-axis positive direction side shown in FIG. 3). Further, hooks (not shown) are provided on both sides of the plug 53. In a state where the gyro sensor unit 7 is attached to the controller 5, the plug 53 is connected to the connector 33 and the hook is locked in the locking hole 33 a of the controller 5. Thereby, the controller 5 and the gyro sensor unit 7 are firmly fixed. The gyro sensor unit 7 has a button 51 on a side surface (surface in the X-axis direction shown in FIG. 3). The button 51 is configured such that when the button 51 is pressed, the hook is released from the locked state with respect to the locking hole 33a. Therefore, the gyro sensor unit 7 can be detached from the controller 5 by removing the plug 53 from the connector 33 while pressing the button 51.

  A connector having the same shape as the connector 33 is provided at the rear end of the gyro sensor unit 7. Therefore, other devices that can be attached to the controller 5 (connector 33 thereof) can also be attached to the connector of the gyro sensor unit 7. In FIG. 3, a cover 52 is detachably attached to the connector.

  The shapes of the controller 5 and the gyro sensor unit 7 shown in FIGS. 3 to 6, the shapes of the operation buttons, the number of acceleration sensors and vibrators, and the installation positions are merely examples, and other shapes, numbers, Even in the installation position, the present invention can be realized. In the present embodiment, the imaging direction by the imaging unit is the positive Z-axis direction, but the imaging direction may be any direction. That is, the position of the imaging information calculation unit 35 in the controller 5 (the light incident surface 35a of the imaging information calculation unit 35) does not have to be the front surface of the housing 31, and other surfaces can be used as long as light can be taken in from the outside of the housing 31. May be provided.

  FIG. 7 is a block diagram showing a configuration of the input device 8 (the controller 5 and the gyro sensor unit 7). The controller 5 includes an operation unit 32 (operation buttons 32a to 32i), a connector 33, an imaging information calculation unit 35, a communication unit 36, and an acceleration sensor 37. The controller 5 transmits data indicating the details of the operation performed on the own device to the game apparatus 3 as operation data.

  The operation unit 32 includes the operation buttons 32a to 32i described above, and the operation button data indicating the input state to each operation button 32a to 32i (whether or not each operation button 32a to 32i is pressed) is transmitted to the microcomputer of the communication unit 36. Output to 42.

  The imaging information calculation unit 35 is a system for analyzing the image data captured by the imaging unit, discriminating a region having a high luminance in the image data, and calculating a center of gravity position, a size, and the like of the region. Since the imaging information calculation unit 35 has a sampling period of, for example, about 200 frames / second at the maximum, it can track and analyze even a relatively fast movement of the controller 5.

  The imaging information calculation unit 35 includes an infrared filter 38, a lens 39, an imaging element 40, and an image processing circuit 41. The infrared filter 38 passes only infrared rays from the light incident from the front of the controller 5. The lens 39 collects the infrared light transmitted through the infrared filter 38 and makes it incident on the image sensor 40. The image sensor 40 is a solid-state image sensor such as a CMOS sensor or a CCD sensor, for example, and receives the infrared light collected by the lens 39 and outputs an image signal. Here, the markers 6 </ b> R and 6 </ b> L of the marker unit 6 disposed in the vicinity of the display screen of the television 2 are configured by infrared LEDs that output infrared light toward the front of the television 2. Therefore, by providing the infrared filter 38, the image sensor 40 receives only the infrared light that has passed through the infrared filter 38 and generates image data, so that the images of the markers 6R and 6L can be captured more accurately. Hereinafter, an image captured by the image sensor 40 is referred to as a captured image. Image data generated by the image sensor 40 is processed by the image processing circuit 41. The image processing circuit 41 calculates the position of the imaging target (markers 6R and 6L) in the captured image. The image processing circuit 41 outputs coordinates indicating the calculated position to the microcomputer 42 of the communication unit 36. The coordinate data is transmitted to the game apparatus 3 as operation data by the microcomputer 42. Hereinafter, the coordinates are referred to as “marker coordinates”. Since the marker coordinates change corresponding to the direction (tilt angle) and position of the controller 5 itself, the game apparatus 3 can calculate the direction and position of the controller 5 using the marker coordinates.

  In other embodiments, the controller 5 may not include the image processing circuit 41, and the captured image itself may be transmitted from the controller 5 to the game apparatus 3. At this time, the game apparatus 3 may have a circuit or a program having the same function as the image processing circuit 41, and may calculate the marker coordinates.

  The acceleration sensor 37 detects the acceleration (including gravity acceleration) of the controller 5, that is, detects the force (including gravity) applied to the controller 5. The acceleration sensor 37 detects the value of the acceleration (linear acceleration) in the linear direction along the sensing axis direction among the accelerations applied to the detection unit of the acceleration sensor 37. For example, in the case of a multi-axis acceleration sensor having two or more axes, the component acceleration along each axis is detected as the acceleration applied to the detection unit of the acceleration sensor. For example, the triaxial or biaxial acceleration sensor may be of the type available from Analog Devices, Inc. or ST Microelectronics NV. The acceleration sensor 37 is, for example, a capacitance type acceleration sensor, but other types of acceleration sensors may be used.

  In the present embodiment, the acceleration sensor 37 has a vertical direction (Y-axis direction shown in FIG. 3), a horizontal direction (X-axis direction shown in FIG. 3), and a front-back direction (Z-axis direction shown in FIG. 3) with reference to the controller 5. ) Linear acceleration is detected in each of the three axis directions. Since the acceleration sensor 37 detects acceleration in the linear direction along each axis, the output from the acceleration sensor 37 represents the linear acceleration value of each of the three axes. That is, the detected acceleration is represented as a three-dimensional vector (ax, ay, az) in an XYZ coordinate system (controller coordinate system) set with reference to the input device 8 (controller 5). Hereinafter, a vector having the respective acceleration values related to the three axes detected by the acceleration sensor 37 as components is referred to as an acceleration vector.

  Data indicating the acceleration detected by the acceleration sensor 37 (acceleration data) is output to the communication unit 36. The acceleration detected by the acceleration sensor 37 changes in accordance with the direction (tilt angle) and movement of the controller 5 itself, so that the game apparatus 3 can calculate the direction and movement of the controller 5 using the acceleration data. it can. In the present embodiment, the game apparatus 3 determines the attitude of the controller 5 based on the acceleration data.

  Data (acceleration data) indicating the acceleration (acceleration vector) detected by the acceleration sensor 37 is output to the communication unit 36. In the present embodiment, the acceleration sensor 37 is used as a sensor that outputs data for determining the tilt angle of the controller 5.

  In addition, based on the acceleration signal output from the acceleration sensor 37, a computer such as a processor (for example, the CPU 10) of the game apparatus 3 or a processor (for example, the microcomputer 42) of the controller 5 performs processing, whereby further information regarding the controller 5 is obtained. Those skilled in the art will be able to easily understand from the description of the present specification that can be estimated or calculated (determined). For example, when processing on the computer side is executed on the assumption that the controller 5 on which the acceleration sensor 37 is mounted is stationary (that is, the processing is executed assuming that the acceleration detected by the acceleration sensor is only gravitational acceleration). When the controller 5 is actually stationary, it can be determined whether or not the attitude of the controller 5 is inclined with respect to the direction of gravity based on the detected acceleration. Specifically, whether or not the controller 5 is inclined with respect to the reference depending on whether or not 1G (gravity acceleration) is applied, based on the state in which the detection axis of the acceleration sensor 37 is directed vertically downward. It is possible to know how much it is inclined with respect to the reference according to its size. Further, in the case of the multi-axis acceleration sensor 37, it is possible to know in detail how much the controller 5 is inclined with respect to the direction of gravity by further processing the acceleration signal of each axis. . In this case, the processor may calculate the tilt angle of the controller 5 based on the output from the acceleration sensor 37, or may calculate the tilt direction of the controller 5 without calculating the tilt angle. Good. Thus, by using the acceleration sensor 37 in combination with the processor, the tilt angle or posture of the controller 5 can be determined.

  On the other hand, when it is assumed that the controller 5 is in a dynamic state (a state in which the controller 5 is moved), the acceleration sensor 37 detects an acceleration corresponding to the movement of the controller 5 in addition to the gravitational acceleration. Therefore, the movement direction of the controller 5 can be known by removing the gravitational acceleration component from the detected acceleration by a predetermined process. Even if it is assumed that the controller 5 is in a dynamic state, the direction of gravity is obtained by removing the acceleration component corresponding to the movement of the acceleration sensor from the detected acceleration by a predetermined process. It is possible to know the inclination of the controller 5 with respect to. In another embodiment, the acceleration sensor 37 is a built-in process for performing a predetermined process on the acceleration signal before outputting the acceleration signal detected by the built-in acceleration detection means to the microcomputer 42. An apparatus or other type of dedicated processing apparatus may be provided. A built-in or dedicated processing device converts the acceleration signal into a tilt angle (or other preferred parameter) if, for example, the acceleration sensor 37 is used to detect static acceleration (eg, gravitational acceleration). It may be a thing.

  The communication unit 36 includes a microcomputer 42, a memory 43, a wireless module 44, and an antenna 45. The microcomputer 42 controls the wireless module 44 that wirelessly transmits data acquired by the microcomputer 42 to the game apparatus 3 while using the memory 43 as a storage area when performing processing. The microcomputer 42 is connected to the connector 33. Data transmitted from the gyro sensor unit 7 is input to the microcomputer 42 via the connector 33. Hereinafter, the configuration of the gyro sensor unit 7 will be described.

  The gyro sensor unit 7 includes a plug 53, a microcomputer 54, a 2-axis gyro sensor 55, and a 1-axis gyro sensor 56. As described above, the gyro sensor unit 7 detects angular velocities around the three axes (in this embodiment, the XYZ axes), and transmits data (angular velocity data) indicating the detected angular velocities to the controller 5.

  The biaxial gyro sensor 55 detects an angular velocity around the X axis and an angular velocity (per unit time) around the Y axis. The single axis gyro sensor 56 detects an angular velocity (per unit time) around the Z axis. In this specification, the rotation directions around the XYZ axes are referred to as a roll direction, a pitch direction, and a yaw direction, respectively, with reference to the imaging direction (Z axis positive direction) of the controller 5. That is, the biaxial gyro sensor 55 detects angular velocities in the roll direction (rotation direction around the X axis) and the pitch direction (rotation direction around the Y axis), and the single axis gyro sensor 56 detects the yaw direction (around the Z axis). Detect the angular velocity in the rotation direction.

  In this embodiment, the 2-axis gyro sensor 55 and the 1-axis gyro sensor 56 are used to detect the angular velocity around the three axes. However, in other embodiments, the angular velocity around the three axes is Any number and combination of gyro sensors may be used as long as they can be detected.

  Further, in the present embodiment, for the purpose of facilitating the calculation in the posture calculation process described later, the three axes where the gyro sensors 55 and 56 detect the angular velocity are the three axes (XYZ) where the acceleration sensor 37 detects the acceleration. Axis). However, in other embodiments, the three axes for detecting the angular velocities by the gyro sensors 56 and 57 and the three axes for detecting the acceleration by the acceleration sensor 37 may not coincide with each other.

  Data indicating the angular velocity detected by each of the gyro sensors 56 and 57 is output to the microcomputer 54. Accordingly, the microcomputer 54 receives data indicating the angular velocity around the three axes of the XYZ axes. The microcomputer 54 transmits data indicating the angular velocities around the three axes as angular velocity data to the controller 5 via the plug 53. Although transmission from the microcomputer 54 to the controller 5 is sequentially performed every predetermined cycle, since the game processing is generally performed in units of 1/60 seconds (one frame time), this time Transmission is preferably performed in the following cycle.

  Returning to the description of the controller 5, the data output from the operation unit 32, the imaging information calculation unit 35, and the acceleration sensor 37 to the microcomputer 42 and the data transmitted from the gyro sensor unit 7 to the microcomputer 42 are temporarily stored. Stored in the memory 43. These data are transmitted to the game apparatus 3 as the operation data. That is, the microcomputer 42 outputs the operation data stored in the memory 43 to the wireless module 44 when the transmission timing to the wireless controller module 19 of the game apparatus 3 arrives. The wireless module 44 modulates a carrier wave of a predetermined frequency with operation data using, for example, Bluetooth (registered trademark) technology, and radiates a weak radio signal from the antenna 45. That is, the operation data is modulated by the wireless module 44 into a weak radio signal and transmitted from the controller 5. The weak radio signal is received by the wireless controller module 19 on the game apparatus 3 side. By demodulating and decoding the received weak radio signal, the game apparatus 3 can acquire operation data. And CPU10 of the game device 3 performs a game process based on the acquired operation data and a game program. Note that the wireless transmission from the communication unit 36 to the wireless controller module 19 is sequentially performed at predetermined intervals, but the game processing is generally performed in units of 1/60 seconds (one frame time). Therefore, it is preferable to perform transmission at a period equal to or shorter than this time. The communication unit 36 of the controller 5 outputs each operation data to the wireless controller module 19 of the game apparatus 3 at a rate of once every 1/200 seconds, for example.

  By using the controller 5, the player can perform an operation of tilting the controller 5 to an arbitrary tilt angle in addition to the conventional general game operation of pressing each operation button. In addition, according to the controller 5, the player can also perform an operation of instructing an arbitrary position on the screen by the controller 5 and an operation of moving the controller 5 itself.

[Outline of posture calculation processing]
Next, with reference to FIGS. 8 to 12, an outline of a posture calculation process for calculating the posture of the input device 8 executed in the game apparatus 3 will be described. In the present embodiment, the game apparatus 3 acquires data (operation data) from the input device 8 including the gyro sensors 55 and 56, the acceleration sensor 37, and the imaging means (imaging element 40). Calculate the posture. In the present embodiment, the input device 8 is configured to include both the acceleration sensor 37 and the image sensor 40. However, in other embodiments, the input device 8 includes only one of the acceleration sensor 37 and the image sensor 40. It may be a configuration.

  The game apparatus 3 includes (1) posture calculation means, (2) first correction means, and (3) second correction means. In the present embodiment, these means are realized by causing a game program (attitude calculation program) executed by the computer (CPU 10) of the game apparatus 3 to function as the means. In other embodiments, part or all of the above means may be realized as a dedicated circuit included in the game apparatus 3.

(1) Attitude calculation means The attitude calculation means calculates the attitude of the input device 8 based on the angular velocity detected by the gyro sensors 55 and 56 (step S4 described later). Any method may be used to calculate the posture from the angular velocity. For example, there is a method of sequentially adding the angular velocity (per unit time) to the initial posture. That is, the current posture can be calculated by integrating the angular velocities sequentially output from the gyro sensors 55 and 56 and calculating the amount of change in posture from the initial state from the integration result. In the following, the posture of the input device 8 calculated from the angular velocity by the posture calculation means is referred to as a “first posture”. However, the posture after the first posture is corrected is also referred to as the first posture.

  Here, the first posture calculated using the angular velocities detected by the gyro sensors 55 and 56 has an error between the actual posture of the input device 8 due to erroneous detection of the gyro sensors 55 and 56. May occur. Therefore, in the present embodiment, the game apparatus 3 corrects the first posture using the acceleration detected by the acceleration sensor 37. Further, the first posture is corrected using an image (captured image) captured by the image sensor 40.

(2) First Correction Unit The first correction unit corrects the first posture based on acceleration data detected by the acceleration sensor 37 (step S5 described later). In the present embodiment, the first correction unit performs correction to bring the first posture closer to the second posture. Here, the second posture refers to a posture determined from the acceleration data, specifically, a posture of the input device 8 when it is assumed that the direction of acceleration indicated by the acceleration data is vertically downward. That is, the posture is calculated on the assumption that the acceleration indicated by the acceleration data is gravitational acceleration. Hereinafter, a correction process (first correction process) by the first correction unit will be described with reference to FIGS.

  FIG. 8 is a diagram illustrating correction of the first posture using the second posture. In this embodiment, the posture in the three-dimensional space is actually processed. However, in FIGS. 8 to 10, the posture in the two-dimensional plane will be described for easy understanding of the drawings. A vector G shown in FIG. 8A indicates a vertically downward direction defined in a spatial coordinate system based on a predetermined position in a space where the input device 8 exists, that is, a gravity direction. Also, the vector v1 shown in FIG. 8A is a spatial coordinate of the downward vector of the input device 8 when the controller 5 is in the first posture (that is, the negative Y-axis direction shown in FIGS. 3 to 5). It shows the orientation in the system. When the posture of the input device 8 is in the basic state, the vector indicating the posture matches the vector G. Therefore, the vector v1 corresponds to the first posture in the spatial coordinate system. The first posture can also be expressed as a rotation made by the vector v1 with respect to the vector G, and is shown as an angle θ1 in the two-dimensional FIG. Since the first posture is calculated based on the angular velocity, the vector v1 is calculated by adding the angular velocity to the previous posture and rotating the vector v1. The second posture is calculated based on the acceleration data. A vector v2 shown in FIG. 8A indicates the direction of acceleration indicated by the acceleration data (the direction of acceleration in the view coordinate system). However, the acceleration data is an acceleration applied to the input device 8, and what can be acquired is a vector in a coordinate system based on the input device 8. FIG. 8B shows the relationship between the acceleration sensor axis and the acceleration vector. As shown in FIG. 8B, when the angle formed by the acceleration vector v0 acquired from the acceleration sensor and the Y-axis negative direction of the sensor is θ2, the vector v1 in the spatial coordinate system of FIG. A vector v2 obtained by adding the rotation θ2 to the acceleration vector in the spatial coordinate system. Since the second posture is “the posture of the input device 8 when the acceleration direction indicated by the acceleration data is regarded as being vertically downward” as described above, the rotation of the angle θ2 from the vector v2 to the vector v1 is performed. Becomes the second posture. If the second posture is represented by a downward vector of the input device 8 in the spatial coordinate system as a vector v1, it can be represented as a vector v2 'obtained by rotating the vector G by θ2. In the case of a three-dimensional posture, it can be expressed by a three-dimensional rotation matrix or the like. When the first posture is correctly calculated from the angular velocity and the acceleration data accurately indicates the direction of gravity, the direction of the vector v2 indicating the direction of acceleration is the vertical downward direction of the spatial coordinate system, that is, gravity. Match the direction. That is, when the first posture is not correctly calculated from the angular velocity, or when the acceleration data does not indicate the correct gravity direction, the vector v2 indicating the direction of acceleration and the gravity as shown in FIG. The direction vector G is different. In a situation where the direction indicated by the acceleration data is assumed to coincide with the direction of gravity, such as in a stationary state, the vector v2 is more accurate as data corresponding to the attitude of the input device 8 than the vector v1. It is considered a thing. Even when the input device is not stationary, taking into account the accuracy of the average posture within a certain period of time, the acceleration vector will be close to the direction of gravity on average, so that the error may increase with time. It can be considered more reliable than the posture calculated from the accumulated angular velocity. On the other hand, if the correct posture has been calculated at the previous calculation timing, the posture at the next calculation timing is calculated more accurately using the angular velocity than the acceleration. it is conceivable that. In other words, the posture calculation based on the angular velocity has a smaller error for each timing than the calculation based on the acceleration, but the error increases with time, while the posture calculation based on the acceleration may have a larger error for each timing. Since it can be calculated at each timing, it has a feature that no error is accumulated. Therefore, the first correction unit performs correction in consideration of both the first posture and the second posture.

  The correction by the first correction means is correction that brings the first posture closer to the second posture. That is, the first correction unit performs correction to bring the angle θ1 close to the angle θ2. This may be expressed as a correction that brings the vector v1 closer to the vector v2 '. However, in the calculation process, if the vector v2 is known, the correction is possible even if the vector v2 'itself is not calculated. In the present embodiment, the correction is performed using a vector v3 indicating the correction amount. FIG. 9 is a diagram illustrating a vector v3 indicating the correction amount. A vector v3 illustrated in FIG. 9 is a vector indicating a correction amount for correcting the first posture. Specifically, an angle Δθ formed by the vector v3 with respect to the vector v2 is a correction amount. Although details will be described later, the vector v3 is set between the vector G and the vector v2 (see FIG. 9). By rotating the vector v1 by Δθ, the vector v1 approaches the vector v2 ′ described above.

  The first correction process is performed by rotating the first posture (vector v1) by the correction amount. FIG. 10 is a diagram illustrating a vector indicating the first posture after the correction by the first correction process. As shown in FIG. 10, the corrected first posture (vector v1 ') is obtained by rotating the first posture before correction (vector v1) by an angle Δθ. As a result, the angle θ1 ′ representing the first posture after correction is between the angle θ1 and the angle θ2, and it can be seen that the correction for bringing the angle θ1 closer to the angle θ2 was performed.

  In the present embodiment, the first correction unit performs correction to bring the first posture closer to the second posture, and does not match the corrected first posture with the second posture. This is to prevent the corrected first posture from changing suddenly even when the acceleration data changes suddenly due to erroneous detection or intense operation. However, in other embodiments, the first correction unit may perform correction so that the corrected first posture matches the second posture. Although details will be described later, in the present embodiment, the rate at which the first correction unit brings the first posture closer to the second posture is the magnitude of the acceleration indicated by the acceleration data (more specifically, It is determined so as to change according to the difference between the magnitude and the gravitational acceleration. However, in other embodiments, the ratio may be a predetermined fixed value.

(3) Second Correction Unit The second correction unit corrects the first posture based on a predetermined image to be captured that is captured by the imaging unit (step S6 described later). Here, in the present embodiment, the predetermined imaging target is the marker unit 6 (infrared LED). In the present embodiment, the second correction unit performs correction to bring the first posture closer to the third posture. The third posture is a posture calculated from the image to be imaged, and specifically, the posture of the input device 8 calculated from the orientation and / or position of the imaged object in the image. Hereinafter, the correction processing (second correction processing) by the second correction means will be described with reference to FIGS. 11 and 12.

  FIG. 11 is a diagram illustrating the correction of the first posture using the third posture. In this embodiment, the posture in the three-dimensional space is actually processed. However, in FIG. 11 and FIG. 12, the posture in the two-dimensional plane will be described for easy understanding of the drawings. A vector v1 shown in FIG. 11 indicates the first posture in the spatial coordinate system. A vector v4 shown in FIG. 11 indicates the third posture in the spatial coordinate system. Since the position and orientation of the marker unit 6 are determined in advance, the orientation of the input device 8 can be relatively calculated based on the orientation and position of the marker in the image. Assuming that the third posture represents the correct posture, when the first posture is correctly calculated from the angular velocity, the vector v1 indicating the first posture indicates the third posture. It matches the vector v4. That is, when the first posture is not correctly calculated from the angular velocity, the vector v1 indicating the first posture and the vector v4 indicating the third posture are different as shown in FIG.

  The second correction process is performed by bringing the first posture (vector v1) closer to the third posture (vector v4) at a predetermined rate. FIG. 12 is a diagram illustrating the first posture after correction by the second correction processing. As shown in FIG. 12, the first posture after correction (vector v1 ′) brings the first posture before correction (vector v1) closer to the third posture (vector v4) at a predetermined rate. Obtained by.

  Here, depending on the orientation and position of the input device 8, the imaging unit may not be able to capture the marker unit 6. In this case, the second correction unit cannot perform the second correction process. Therefore, if the second correction unit performs correction so that the corrected first posture matches the third posture, the second correction processing is performed from a state where the second correction processing cannot be performed. There is a possibility that the first posture may change abruptly when transitioning to a state where it is possible to do so. In this way, if the first posture suddenly changes without conforming to the player's intention, the player will feel uncomfortable with the operation (even if the corrected posture is accurate). In order to prevent such a sudden change, in the present embodiment, correction is performed so that the first posture approaches the third posture at a predetermined rate. As a result, it is possible to prevent the first posture from changing suddenly, thereby preventing the player from feeling uncomfortable with the operation. However, in other embodiments, such as when the input device 8 can be assumed to be used in a posture in which the imaging unit can always image the marker unit 6, the second correction unit is the corrected first unit. You may correct | amend so that this attitude | position may correspond with a 3rd attitude | position.

  In the present embodiment, the game apparatus 3 executes both the first correction process and the second correction process, but in other embodiments, one of the first correction process and the second correction process. May be executed only. In the present embodiment, the game apparatus 3 performs the first correction process first and the second correction process later, but either the first correction process or the second correction process is performed first. You may do it.

  As described above, according to the present embodiment, the attitude of the input device 8 calculated from the angular velocities detected by the gyro sensors 55 and 56 is corrected using the acceleration detected by the acceleration sensor 37, and imaging is performed. Correction is performed using the image captured by the means. Thereby, the error of the attitude calculated from the gyro sensor can be reduced, and the attitude of the input device 8 can be calculated more accurately.

  In addition, from the detection result of the acceleration sensor 37, the rotation around the gravity direction (rotation in the yaw direction) cannot be detected, and therefore the correction by the first correction unit cannot be performed in the yaw direction. However, the correction using the detection result of the acceleration sensor 37 has a feature that the posture of the input device 8 can be any posture (because acceleration can always be detected). On the other hand, if the marker unit 6 is not present in the imaging direction of the input device 8, the marker coordinates are not detected, and therefore the correction by the second correction unit cannot be performed depending on the attitude of the input device 8. However, the correction using the captured image has a feature that the posture (especially the posture in the roll direction) can be accurately calculated. In the present embodiment, the posture of the input device 8 can be calculated more accurately by performing two types of corrections having different features as described above.

[Details of processing in game device 3]
Next, details of processing executed in the game apparatus 3 will be described. First, main data used in the processing in the game apparatus 3 will be described with reference to FIG. FIG. 13 is a diagram illustrating main data stored in the main memory (the external main memory 12 or the internal main memory 11e) of the game apparatus 3. As shown in FIG. 13, a game program 60, operation data 62, and game processing data 67 are stored in the main memory of the game apparatus 3. In addition to the data shown in FIG. 13, the main memory stores data necessary for game processing, such as image data of various objects appearing in the game and data indicating various parameters of the objects.

  A part or all of the game program 60 is read from the optical disc 4 and stored in the main memory at an appropriate timing after the game apparatus 3 is turned on. The game program 60 includes a posture calculation program 61. The attitude calculation program 61 is a program for executing an attitude calculation process for calculating the attitude of the input device 8.

  The operation data 62 is operation data transmitted from the controller 5 to the game apparatus 3. As described above, since the operation data is transmitted from the controller 5 to the game apparatus 3 at a rate of once every 1/200 seconds, the operation data 62 stored in the main memory is updated at this rate. In the present embodiment, only the latest (last acquired) operation data may be stored in the main memory.

  The operation data 62 includes angular velocity data 63, acceleration data 64, marker coordinate data 65, and operation button data 66. The angular velocity data 63 is data indicating angular velocities detected by the gyro sensors 55 and 56 of the gyro sensor unit 7. Here, the angular velocity data 63 indicates the respective angular velocities around the three axes of XYZ shown in FIG. The acceleration data 64 is data indicating the acceleration (acceleration vector) detected by the acceleration sensor 37. Here, the acceleration data 64 represents a three-dimensional acceleration vector Va1 having as components the accelerations in the directions of the three axes of XYZ shown in FIG. In the present embodiment, the magnitude of the acceleration vector Va1 detected by the acceleration sensor 37 while the controller 5 is stationary is “1”. That is, the magnitude of gravitational acceleration detected by the acceleration sensor 37 is “1”.

  The marker coordinate data 65 is data indicating coordinates calculated by the image processing circuit 41 of the imaging information calculation unit 35, that is, the marker coordinates. The marker coordinates are expressed in a two-dimensional coordinate system (x′y ′ coordinate system shown in FIG. 17) for representing a position on a plane corresponding to the captured image. Note that, when the two markers 6R and 6L are imaged by the imaging element 40, two marker coordinates are calculated. On the other hand, if either one of the markers 6R and 6L is not located within the imageable range of the image sensor 40, only one marker is imaged by the image sensor 40, and only one marker coordinate is calculated. Further, when both of the markers 6R and 6L are not located within the image capturing range of the image sensor 40, the marker is not imaged by the image sensor 40, and the marker coordinates are not calculated. Therefore, the marker coordinate data 65 may indicate two marker coordinates, may indicate one marker coordinate, or may indicate that there is no marker coordinate.

  The operation button data 66 is data indicating an input state for each of the operation buttons 32a to 32i.

  The game process data 67 is data used in a game process (FIG. 14) described later. The game processing data 67 includes first posture data 68, acceleration magnitude data 69, correction degree data 70, correction amount vector data 71, correction matrix data 72, roll posture component data 73, yaw posture component data 74, pitch posture component. Data 75 and third attitude data 76 are included. In addition to the data shown in FIG. 13, the game processing data 67 includes various data (data indicating game parameters, etc.) used in the game processing.

上記行列Ｍ１は、所定の基準姿勢から現在の入力装置８の姿勢への回転を表す回転行列である。以下では、第１の姿勢を示す行列Ｍ１を、「第１姿勢行列Ｍ１」と呼ぶ。なお、第１姿勢行列Ｍ１により表される第１の姿勢は、入力装置８が存在する空間の所定位置を基準としたｘｙｚ座標系（上記空間座標系）における姿勢である。ここでは、ｘｙｚ座標系は、入力装置８がマーカ部６の正面に位置することを前提とし、入力装置８の位置からマーカ部６を向く方向をｚ軸正方向とし、鉛直上向き（重力方向の逆方向）をｙ軸正方向とし、入力装置８の位置からマーカ部６を向いた場合の左方向をｘ軸正方向とした座標系であるとする。ここでは、上記所定の基準姿勢は、マーカ部６の正面に位置する入力装置８の撮像方向がマーカ部６の中央を向き、かつ、コントローラ５のボタン面が鉛直上向きとなる姿勢（すなわち、入力装置８を基準としたＸ軸、Ｙ軸、Ｚ軸が、それぞれｘ軸、ｙ軸、ｚ軸の向きと一致する姿勢）であるとする。なお、本実施形態では、行列を用いて第１の姿勢を表現することとしたが、他の実施形態においては、第１の姿勢は、３次のベクトルまたは３つの角度によって表現されてもよい。 The first attitude data 68 is data indicating the first attitude calculated using the angular velocity data 63. In the present embodiment, the first posture is expressed by a 3 × 3 matrix M1 shown in the following equation (1).

The matrix M1 is a rotation matrix representing the rotation from a predetermined reference posture to the current input device 8 posture. Hereinafter, the matrix M1 indicating the first posture is referred to as a “first posture matrix M1”. Note that the first posture represented by the first posture matrix M1 is a posture in the xyz coordinate system (the spatial coordinate system) with reference to a predetermined position in the space where the input device 8 exists. Here, the xyz coordinate system is based on the premise that the input device 8 is located in front of the marker unit 6, and the direction from the input device 8 toward the marker unit 6 is the z-axis positive direction, and is vertically upward (in the gravitational direction). It is assumed that the coordinate system is such that the reverse direction) is the y-axis positive direction and the left direction when facing the marker unit 6 from the position of the input device 8 is the x-axis positive direction. Here, the predetermined reference posture is a posture in which the imaging direction of the input device 8 located in front of the marker unit 6 faces the center of the marker unit 6 and the button surface of the controller 5 is vertically upward (that is, input) Assume that the X axis, the Y axis, and the Z axis with respect to the device 8 are postures that coincide with the directions of the x axis, the y axis, and the z axis, respectively. In the present embodiment, the first posture is expressed using a matrix. However, in other embodiments, the first posture may be expressed by a cubic vector or three angles. .

  The acceleration magnitude data 69 is data indicating the magnitude (length) L of the acceleration vector Va1 indicated by the acceleration data 64.

  The correction degree data 70 is data indicating the degree of correction of the first attitude using the second attitude (correction degree A). The correction degree A is a value that can take a range of 0 ≦ A ≦ C1 (C1 is a predetermined constant of 0 <C1 ≦ 1). Although details will be described later, the corrected first posture becomes closer to the second posture as the correction degree A is larger.

  The correction amount vector data 71 is data indicating a vector indicating a correction amount for correcting the first posture (vector v3 shown in FIG. 9; hereinafter referred to as a correction amount vector). The correction amount vector Vg is calculated based on the vector Va2 representing the acceleration vector Va1 in the xyz coordinate system and the correction degree A.

  The correction matrix data 72 is data indicating a rotation matrix (referred to as a correction matrix) Ma used for correcting the first posture. That is, in the first correction process, the first posture is corrected by multiplying the first posture matrix M1 representing the first posture by the correction matrix Ma. The correction matrix Ma is calculated based on the vector Va2 and the correction amount vector Vg.

  The roll posture component data 73 is data indicating a posture component (roll posture component) M3r related to the roll direction among the posture components included in the third posture calculated from the image to be captured. The yaw posture component data 74 is data indicating the posture component (yaw posture component) M3y related to the yaw direction among the posture components included in the third posture, and the pitch posture component data 75 is the third posture information. Among the posture components included in the posture, this is data indicating a posture component (pitch posture component) M3p related to the pitch direction. Note that the roll direction, yaw direction, and pitch direction referred to here are rotation directions when the imaging direction (Z-axis positive direction) of the input device 8 is used as a reference. In the present embodiment, each of the posture components M3r, M3y, and M3p is represented by a 3 × 3 matrix as in the first posture.

  The third attitude data 76 is data indicating a third attitude calculated from the image to be imaged. In the present embodiment, the third posture is represented by a 3 × 3 matrix M3, like the first posture. Hereinafter, the matrix M3 indicating the third posture is referred to as a “third posture matrix M3”. In the present embodiment, since the marker coordinate data is transmitted as operation data from the input device 8, the third posture matrix M <b> 3 is calculated based on the marker coordinate data 65. Specifically, the third posture matrix M3 is obtained by combining the posture components M3r, M3y, and M3p.

  Next, details of processing performed in the game apparatus 3 will be described with reference to FIGS. FIG. 14 is a main flowchart showing the flow of processing executed in the game apparatus 3. When the power of the game apparatus 3 is turned on, the CPU 10 of the game apparatus 3 executes a startup program stored in a boot ROM (not shown), whereby each unit such as the main memory is initialized. Then, the game program stored in the optical disc 4 is read into the main memory, and the CPU 10 starts executing the game program. The flowchart shown in FIG. 14 is a flowchart showing processing performed after the above processing is completed.

  First, in step S1, the CPU 10 executes an initialization process relating to the game. In this initialization process, the values of various parameters used in the game process are initialized, a virtual game space is constructed, and player objects and other objects are placed at initial positions in the game space. Following step S1, the process of step S2 is executed.

  In step S2, the CPU 10 executes an initial posture setting process. Specifically, a predetermined value is set as the initial posture of the input device 8 in response to the player performing a predetermined operation (for example, an operation of pressing the A button 32d). Here, the reference posture is a posture in which the Z axis is parallel to the vertical direction and the imaging direction of the input device 8 faces the center of the marker unit 6 (the center of the markers 6R and 6L). It is desirable to perform the predetermined operation while holding the input device 8 so that the initial posture becomes the reference posture, but if the input device is close to a stationary state and the marker portion can be imaged, It is possible to calculate the initial posture. When a predetermined operation is performed, the CPU 10 stores data indicating a matrix representing the initial posture in the main memory as first posture data. After the above step S2, the processing loop of steps S3 to S8 is repeatedly executed while the game is executed. Note that one processing loop is executed once per frame time (for example, 1/60 seconds).

  In this embodiment, the initial posture setting process (step S2) is executed only once before the game starts (before the processing loop of steps S3 to S8 is executed). In the embodiment, the initial posture setting process may be executed at an arbitrary timing during the game. That is, the CPU 10 may execute the initial posture setting process in response to the player performing the predetermined operation during the game.

  In step S3, the CPU 10 acquires operation data. That is, operation data transmitted from the controller 5 is received via the wireless controller module 19. Then, angular velocity data, acceleration data, marker coordinate data, and operation button data included in the received operation data are stored in the main memory. Following step S3, the process of step S4 is executed.

  In step S4, the CPU 10 calculates the first posture based on the angular velocity data 63 stored in the main memory. Any method may be used to calculate the attitude of the input device 8 from the angular velocity. However, in the present embodiment, the first attitude is the previous first attitude (calculated in the previous processing loop). (First attitude) and the current angular velocity (the angular velocity acquired in the current processing loop). Specifically, the CPU 10 sets the posture obtained by rotating the previous first posture by unit time at the current angular velocity as the first posture. The previous first posture is indicated by the first posture data 68 stored in the main memory, and the current angular velocity is indicated by the angular velocity data 63 stored in the main memory. Data indicating the posture (3 × 3 matrix) calculated in step S4 is newly stored in the main memory as the first posture data 68. Following step S4, the process of step S5 is executed.

  In step S5, the CPU 10 executes the first correction process described above. The first correction process is a process of correcting the first posture using acceleration data. Hereinafter, the details of the first correction process will be described with reference to FIG.

  FIG. 15 is a flowchart showing the flow of the first correction process (step S5) shown in FIG. In the first correction process, first, in step S <b> 11, the CPU 10 calculates the magnitude L of the acceleration detected by the acceleration sensor 37. That is, the acceleration data 64 stored in the main memory is read, and the magnitude L is calculated for the acceleration vector Va1 indicated by the acceleration data 64. Data indicating the calculated magnitude L is stored as acceleration magnitude data 69 in the main memory. Following step S11, the process of step S12 is executed.

  In step S12, the CPU 10 determines whether or not the magnitude of the acceleration detected by the acceleration sensor 37 is zero. That is, the acceleration magnitude data 69 stored in the main memory is read, and it is determined whether or not the magnitude L indicated by the acceleration magnitude data 69 is zero. If the determination result of step S12 is negative, the process of step S13 is executed. On the other hand, when the determination result of step S12 is affirmative, the processes of subsequent steps S13 to S21 are skipped, and the CPU 10 ends the first correction process. Thus, in this embodiment, when the magnitude of the acceleration detected by the acceleration sensor 37 is 0, correction using the acceleration is not performed. This means that the direction of gravity cannot be calculated from the detection result of the acceleration sensor 37 when the magnitude of the acceleration is 0, and the subsequent steps S13 to S21 are performed if the magnitude of the acceleration vector is 0. The reason is that it is difficult.

  In step S <b> 13, the CPU 10 normalizes the acceleration vector Va <b> 1 detected by the acceleration sensor 37. That is, the acceleration data 64 stored in the main memory is read, and the acceleration vector Va1 indicated by the acceleration data 64 is corrected so that the magnitude is 1. The CPU 10 stores data indicating the normalized acceleration vector Va1 in the main memory. Following step S13, the process of step S14 is executed.

In step S <b> 14, the CPU 10 calculates the correction degree A indicating the degree of correction of the first posture in the first correction process. The correction degree A is calculated based on the magnitude L of the acceleration vector Va1 before normalization. Specifically, the CPU 10 reads acceleration magnitude data 69 stored in the main memory. Then, using the magnitude L indicated by the acceleration magnitude data 69, the correction degree A is calculated according to the following equation (2).
A = | L−1 | (2)
Data indicating the correction degree A calculated by the above equation (2) is stored as correction degree data 70 in the main memory. The correction degree A calculated by the above equation (2) is a value that is not a final value and is being calculated, and the final correction degree A is obtained by converting the value in the subsequent step S16. A value is obtained. Following step S14, the process of step S15 is executed.

  In step S15, the CPU 10 determines whether or not the correction degree A calculated in step S14 is smaller than a predetermined value R. The predetermined value R is determined in advance and is set to 0.4, for example. Here, as described above, in the present embodiment, the magnitude of the gravitational acceleration detected by the acceleration sensor 37 is “1”, and the correction degree A is the magnitude L of the acceleration vector Va1 and “1”. (The above formula (2)). Therefore, the case where the correction degree A is equal to or greater than the predetermined value R is a case where the magnitude L of the acceleration vector Va1 is separated from the magnitude of the gravitational acceleration by the predetermined value R or more. If the determination result of step S15 is affirmative, the process of step S16 is executed. On the other hand, when the determination result of step S15 is negative, the processes of subsequent steps S16 to S21 are skipped, and the CPU 10 ends the first correction process.

  As described above, in the present embodiment, only when the difference between the acceleration magnitude L detected by the acceleration sensor 37 and the gravitational acceleration magnitude (= 1) is smaller than a predetermined reference (predetermined value R). Correction is performed, and correction is not performed when the difference between the magnitude L and the magnitude of gravitational acceleration is equal to or greater than a predetermined reference. Here, in the state where the input device 8 is moved, in addition to the gravitational acceleration, the acceleration due to inertia caused by the movement of the input device 8 is detected by the acceleration sensor 37, so the magnitude of the detected acceleration vector Va1. L becomes a value different from “1”, and when the input device 8 is moved violently, the size L becomes a value far from “1”. Therefore, the case where the difference between the magnitude L and the magnitude of the gravitational acceleration is equal to or greater than a predetermined reference is estimated to be a case where the input device 8 is moved violently. On the other hand, when the input device 8 is moved violently, the acceleration vector Va1 detected by the acceleration sensor 37 contains many components other than gravitational acceleration (the acceleration component due to inertia), and therefore the acceleration vector Va1. The value of is presumed to be unreliable as a value indicating the direction of gravity. Therefore, the determination process of step S15 determines whether or not the input device 8 is moved violently, in other words, whether or not the value of the acceleration vector Va1 is reliable as a value indicating the direction of gravity. It is processing. In the present embodiment, if the value of the acceleration vector Va1 is not reliable as a value indicating the direction of gravity by the determination process in step S15, no correction is performed, and the value of the acceleration vector Va1 is a value indicating the direction of gravity. Corrections are made only when reliable. As a result, it is possible to prevent the first posture from being correctly corrected as a result of performing the correction on the first posture using the acceleration vector Va1 that is not reliable as a value indicating the direction of gravity.

In step S16, the CPU 10 converts the value of the correction degree A. In the present embodiment, the correction degree A is converted so that the correction degree A becomes closer to 1 as the magnitude L of the detected acceleration vector Va1 is closer to the magnitude of gravitational acceleration. Specifically, the CPU 10 reads the correction degree data 70 stored in the main memory, and converts the correction degree A indicated by the correction degree data 70 according to the following equations (3) to (5).
A2 = 1- (A1 / R) (3)
A3 = A2 × A2 (4)
A4 = A3 × C1 (5)
In the above formulas (3) to (5), the variable A1 is the correction degree before conversion (value indicated by the correction degree data 70 currently stored in the main memory), and the variable A4 is finally converted in step S16. Is the degree of correction. The above expression (3) is an expression for obtaining the corrected degree of correction A2 so that the degree of correction A1 before the conversion becomes closer to 1 as the magnitude of the gravitational acceleration (= 1) is closer. The above equation (4) is an equation for obtaining the corrected degree of correction A3 so that a larger weight is assigned as the correction degree A2 before conversion is closer to 1. The above equation (5) is an equation for adjusting the magnitude of the correction amount. That is, the correction amount of the constant C1 increases as the value of the constant C1 increases. The constant C1 is determined in advance and is set in a range of 0 <C1 ≦ 1 (for example, 0.03). Data indicating the correction degree A4 obtained by the conversion according to the above equations (3) to (5) is stored as new correction degree data 70 in the main memory. Following step S16, the process of step S17 is executed.

  In the present embodiment, the conversions according to the above formulas (3) to (5) are performed. However, in other embodiments, some or all of the conversions in the above formulas (3) to (5) are omitted. May be. However, when the conversion of the above equation (3) is omitted, it is necessary to replace the acceleration vector Va2 and the gravity direction vector (0, -1, 0) in equation (7) used in step S18 described later.

In step S17, the CPU 10 converts the acceleration vector Va1 expressed in the XYZ coordinate system into a value Va2 in the xyz coordinate system. The acceleration vector Va2 in the xyz coordinate system is calculated by converting the normalized acceleration vector Va1 using the first posture matrix M1 representing the first posture obtained in the previous frame. That is, the CPU 10 reads the data of the acceleration vector Va1 (normalized) stored in the main memory and the first attitude data 68 in step S13. Then, using the acceleration vector Va1 and the first posture matrix M1 indicated by the first posture data 68, an acceleration vector Va2 in the xyz coordinate system is calculated. More specifically, normalized acceleration vector Va1 = (nx, ny, nz), each element of the first posture matrix M1 is a variable shown in the above equation (1), and acceleration expressed in the xyz coordinate system If the vector Va2 = (vx, vy, vz), the acceleration vector Va2 is calculated according to the following equation (6).
vx = Xx * nx + Yx * ny + Zx * nz
vy = Xy * nx + Yy * ny + Zy * nz
vz = Xz * nx + Yz * ny + Zz * nz (6)
As shown in the above equation (6), the acceleration vector Va2 can be obtained by rotating the acceleration vector Va1 by the first posture matrix M1 that is a rotation matrix. The acceleration vector Va2 calculated in step S17 is stored in the main memory. Following step S17, the process of step S18 is executed.

In step S18, the CPU 10 calculates the correction amount vector Vg using the acceleration vector Va2 expressed in the xyz coordinate system and the correction degree A. The correction amount vector Vg is calculated using the degree of correction after the conversion in step S16 and a vector (0, -1, 0) indicating the vertical downward direction (gravity direction) of the xyz coordinate system. Specifically, the CPU 10 reads out the correction degree data 70 stored in the main memory, and uses the correction degree A indicated by the correction degree data 70, according to the following equation (7), the correction amount vector Vg = ( gx, gy, gz) is calculated.
gx = (0−vx) × A + vx
gy = (− 1−vy) × A + vy
gz = (0−vz) × A + vz (7)
As shown in the above equation (7), the correction amount vector Vg is a line segment connecting the end point of the acceleration vector Va2 to the end point of the gravity direction vector (0, -1, 0) as A: (1-A). It is a vector whose end point is a point to be internally divided. Accordingly, the correction amount vector Vg is closer to the gravity direction vector as the correction degree A is larger. The CPU 10 stores data indicating the correction amount vector Vg calculated by the above equation (7) as correction amount vector data 71 in the main memory. Following step S18, the process of step S19 is executed.

  In step S19, the CPU 10 normalizes the correction amount vector Vg calculated in step S18. That is, the correction amount vector data 71 stored in the main memory is read, and the vector indicated by the correction amount vector data 71 is normalized. Then, the data indicating the normalized vector is stored as new correction amount vector data 71 in the main memory. The correction amount vector Vg calculated in step S19 corresponds to the vector v3 shown in FIG. Following step S19, the process of step S20 is executed.

  In step S20, the CPU 10 calculates a correction matrix Ma for correcting the first posture. The correction matrix Ma is calculated based on the acceleration vector Va2 expressed in the xyz coordinate system and the correction amount vector Vg normalized in step S19. Specifically, the CPU 10 reads the acceleration vector Va2 and the correction amount vector data 71 stored in the main memory in step S17. Then, a rotation matrix for rotating the acceleration vector Va2 so as to coincide with the correction amount vector Vg is calculated, and the calculated rotation matrix is set as a correction matrix Ma. That is, the correction matrix Ma is a rotation matrix that performs the rotation of the angle Δθ shown in FIG. Data indicating the correction matrix Ma calculated in step S20 is stored in the main memory as correction matrix data 72. Following step S20, the process of step S21 is executed.

  In step S21, the CPU 10 corrects the first posture matrix M1 indicating the first posture with the correction matrix Ma. Specifically, the CPU 10 reads the first attitude data 68 and the correction matrix data 72 stored in the main memory. Then, the first attitude matrix M1 indicated by the first attitude data 68 is converted by the correction matrix Ma indicated by the correction matrix data 72 (the product of the first attitude matrix M1 and the correction matrix Ma is calculated). The converted first posture matrix M1 indicates the corrected first posture. That is, the process of step S21 corresponds to the process of rotating the vector v1 shown in FIG. 10 by the angle Δθ. The CPU 10 stores data indicating the converted first attitude matrix M1 as new first attitude data 68 in the main memory. After step S21, the CPU 10 ends the first correction process.

  As described above, in the first correction process, the correction amount vector Vg is calculated between the acceleration vector detected by the acceleration sensor 37 and the gravity direction vector (vector G shown in FIG. 8) (steps S18 and S19). ), The first posture is corrected by the correction amount (correction matrix Ma. Angle Δθ shown in FIG. 9) represented by the correction amount vector Vg (step S21). As a result, the first attitude calculated from the gyro sensors 55 and 56 (vector v1 or angle θ1 shown in FIG. 8) is corrected to approach the second attitude (angle θ2 shown in FIG. 8) determined by the acceleration sensor 37. It can be performed. By performing such correction, the first posture can be corrected so as to have a more accurate value.

  In the first correction process, as the acceleration vector Va1 is reliable (the smaller the difference between the magnitude L of the acceleration vector Va1 and the magnitude of gravitational acceleration), the value of the correction degree A increases. The first posture is corrected so as to be closer to the second posture. That is, the more reliable the acceleration vector Va1, the larger the correction amount, and the second posture is strongly reflected in the corrected first posture. As described above, according to the present embodiment, the correction amount in the first correction process is determined according to the reliability of the acceleration sensor vector Va1, and therefore the correction amount can be appropriately determined according to the reliability. The attitude of the input device 8 can be calculated more accurately.

  In the present embodiment, the correction amount vector Vg calculated in step S18 is a point that internally divides the line segment connecting the end point of the acceleration vector Va2 to the end point of the gravity direction vector into A: (1-A). Is the vector that approaches the gravity direction vector as the value of the correction degree A increases. In another embodiment, depending on the method of calculating the correction degree A, the correction amount vector Vg is a point that internally divides the end point of the gravity direction vector and the end point of the acceleration vector Va2 into (1-A): A. The end point vector may be a vector that approaches the gravity direction vector as the correction degree A is smaller. At this time, in step S20, a rotation matrix for rotating the correction amount vector Vg so as to coincide with the direction of gravity may be calculated, and the calculated rotation matrix may be used as the correction matrix Ma. This also makes it possible to perform correction as in the present embodiment.

  Returning to the description of FIG. 14, in step S <b> 6 subsequent to step S <b> 5, the CPU 10 executes the second correction process described above. The second correction process is a process of correcting the first posture using the marker coordinate data. Hereinafter, the details of the second correction process will be described with reference to FIG.

  FIG. 16 is a flowchart showing the flow of the second correction process (step S6) shown in FIG. In the first correction process, first, in step S <b> 31, the CPU 10 determines whether or not the marker unit 6 is imaged by the imaging unit (imaging device 40) of the input device 8. The determination in step S31 can be made by referring to the marker coordinate data 65 stored in the main memory. Here, when the marker coordinate data 65 indicates two marker coordinates, it is determined that the marker unit 6 is captured, and when the marker coordinate data 65 indicates only one marker coordinate, or there is no marker coordinate. When it shows, it determines with the marker part 6 not being imaged. If the determination result of step S31 is affirmative, the processes of subsequent steps S32 to S37 are executed. On the other hand, when the determination result of step S31 is negative, the processes of the subsequent steps S32 to S37 are skipped, and the CPU 10 ends the second correction process. As described above, when the marker unit 6 is not imaged by the image sensor 40, the attitude of the input device 8 cannot be calculated using data obtained from the image sensor 40. In this case, the second correction is performed. No correction is performed in the process.

  In step S32, the CPU 10 calculates a roll posture component M3r based on the marker coordinate data. The roll posture component M3r is calculated based on the orientation of the marker unit 6 in the captured image, that is, based on the slope of the line connecting the two marker coordinates indicated by the marker coordinate data 65. Hereinafter, an example of a method for calculating the roll posture component M3r will be described with reference to FIG.

  FIG. 17 is a diagram illustrating two-dimensional coordinates corresponding to a captured image. As shown in FIG. 17, in the present embodiment, a two-dimensional coordinate system (x′y ′ coordinate system) for representing a position in a captured image has a range of the captured image of −1 ≦ x ′ ≦ 1, − 1 ≦ y ′ ≦ 1. The x′y ′ coordinate system captures images when the input device 8 is in a reference posture (an orientation in which the imaging direction of the input device 8 faces the center of the marker unit 6 and the button surface of the controller 5 is vertically upward). In the image, it is assumed that the vertically downward direction is the positive direction of the y ′ axis and the right direction is the positive direction of the x ′ axis. Also, point P1 and point P2 shown in FIG. 17 indicate the positions of the marker coordinates, and point P3 is a midpoint between points P1 and P2. A vector v10 shown in FIG. 17 is a vector having a point P1 as a start point and a point P2 as an end point.

上式（８）によって算出された行列を示すデータは、ロール姿勢成分データ７３としてメインメモリに記憶される。ステップＳ３２の次にステップＳ３３の処理が実行される。 In order to calculate the roll posture component M3r, the CPU 10 first reads the marker coordinate data 65, and calculates the vector v10 from the two marker coordinates indicated by the marker coordinate data 65. Further, a vector (hx, hy) obtained by normalizing the vector v10 is calculated. The vector (hx, hy) is directed in the positive x-axis direction when the input device 8 is in the reference posture, and the direction changes according to the rotation of the input device 8 in the roll direction. Since the vector (hx, hy) corresponds to the posture in the roll direction, the roll posture component M3r can be calculated based on the vector (hx, hy). Specifically, the CPU 10 calculates the roll posture component M3r according to the following equation (8).

Data indicating the matrix calculated by the above equation (8) is stored in the main memory as roll posture component data 73. Following step S32, the process of step S33 is executed.

  In step S33, the CPU 10 calculates a yaw posture component M3y based on the marker coordinate data. The yaw posture component M3y is calculated based on the orientation and position of the marker unit 6 in the captured image. Hereinafter, an example of a method for calculating the yaw posture component M3y will be described with reference to FIG.

  First, the CPU 10 reads the marker coordinate data 65 and calculates the midpoint between the two marker coordinates indicated by the marker coordinate data 65. In the present embodiment, the position of the middle point is used as the position of the marker unit 6. Further, the CPU 10 rotates the calculated midpoint coordinates by the rotation angle related to the roll direction of the input device 8 (in the direction opposite to the rotation direction of the input device 8) around the origin of the x′y ′ coordinate system. Calculated coordinates (px, py) are calculated. In other words, the coordinates of the midpoint are rotated around the origin so that the vector (hx, hy) faces the positive x-axis direction. If the input device 8 is at the same position as the marker unit 6 in the horizontal direction (x-axis direction) (that is, the front position of the marker unit 6), the coordinates (px, py) can be used to calculate the attitude related to the yaw direction.

Next, the CPU 10 based on the coordinates (px, py) after the rotation of the midpoint and the angle (limit angle) θy ′ in the yaw direction when the marker unit 6 is located at the end in the x′-axis direction. Then, the rotation angle θy related to the yaw direction is calculated. Here, the limit angle θy ′ and the x-coordinate value px ′ after the rotation of the midpoint when the limit angle θy1 is reached can be obtained in advance. Therefore, the rotation angle θy related to the yaw direction can be calculated using the fact that the ratio between px and px ′ is equal to the ratio between θy and θy ′. Specifically, the rotation angle θy related to the yaw direction can be calculated by the following equation (9).
θy = px × θy ′ / px ′ (9)
When the horizontal length of the marker unit 6 is ignored, the limit angle θy ′ can be set to ½ of the angle of view of the controller 5 and the value of px ′ can be set to “1”.

上式（１０）によって算出された行列を示すデータが、ヨー姿勢成分データ７４としてメインメモリに記憶される。以上のステップＳ３３の次にステップＳ３４の処理が実行される。 Finally, the CPU 10 calculates a rotation matrix for performing the rotation of the angle θy calculated by the above equation (9) as the yaw attitude component M3y. Specifically, the yaw posture component M3y is calculated by the following equation (10).

Data indicating the matrix calculated by the above equation (10) is stored in the main memory as yaw attitude component data 74. Following step S33, the process of step S34 is executed.

  In step S34, the CPU 10 combines the roll posture component M3r and the yaw posture component M3y. That is, the roll attitude component data 73 and the yaw attitude component data 74 are read from the main memory, and the roll attitude component M3r and the yaw attitude component M3y indicated by the data 73 and 74 are integrated. Following step S34, the process of step S35 is executed.

  In step S35, the CPU 10 calculates a pitch posture component M3p based on the first posture. Although different from the processing of the present embodiment, the pitch attitude component M3p can also be calculated based on the y coordinate value of the coordinates (px, py) in the same manner as the yaw attitude component M3y. . However, in the method of calculating the attitude in the yaw direction (pitch direction) using the coordinates (px, py), the input device 8 is at the same position as the marker unit 6 in the horizontal direction (vertical direction in the case of the pitch direction). This method is based on the assumption. In the game system 1 of the present embodiment, it is considered that the player operates the input device 8 at a position substantially in front of the marker unit 6 (TV 2) in the horizontal direction. It is possible to calculate the attitude in the yaw direction by the method of step S33 on the assumption that “they are at the same position”. On the other hand, it is conceivable that the player stands and operates the input device 8 or sits down and operates the input device 8, and the position of the marker unit 6 is also arranged on the upper side of the screen of the television 2. It is also possible to arrange them in Therefore, in the game system 1 of the present embodiment, it is not necessarily assumed that “the input device 8 is at the same position as the marker unit 6 with respect to the vertical direction”. Therefore, the pitch direction using the coordinates (px, py) is not necessarily assumed. The posture may not be calculated.

上式（１２）によって算出された行列を示すデータが、ピッチ姿勢成分データ７５としてメインメモリに記憶される。以上のステップＳ３５の次にステップＳ３６の処理が実行される。 Therefore, in the present embodiment, the first posture is used as it is for the pitch posture component M3p (thus, in the second correction process, the pitch direction is not corrected). Specifically, the CPU 10 reads the first attitude data 68 from the main memory. Then, using each element of the first posture matrix M1 indicated by the first posture data 68, the rotation angle θp with respect to the pitch direction is calculated according to the following equation (11).
cos (θp) = (Zx × Zx + Zz × Zz) ^1/2
sin (θp) = Zy (11)
Variables Zx, Zy, and Zz in the above equation (11) are elements of the first posture matrix M1 shown in the above equation (1). The first posture matrix M1 used here is the first posture matrix M1 after the first correction processing is performed in the current processing loop. Further, the CPU 10 calculates a matrix of the pitch posture component M3p according to the following equation (12) using cos (θp) and sin (θp) calculated by the above equation (11).

Data indicating the matrix calculated by the above equation (12) is stored in the main memory as pitch attitude component data 75. Following step S35, the process of step S36 is executed.

  In step S36, the CPU 10 calculates a third posture based on the posture components in the roll direction, the yaw direction, and the pitch direction. The third attitude is obtained by further synthesizing the pitch attitude component M3p with the synthesis result of the roll attitude component M3r and the yaw attitude component M3y. Specifically, the CPU 10 reads the pitch posture component data 75 from the main memory, and integrates the pitch posture component M3p indicated by the pitch posture component data 75 in the matrix calculated in step S34. Data indicating the calculated matrix is stored in the main memory as third attitude data 76. Following step S36, the process of step S37 is executed.

In step S37, the CPU 10 corrects the first attitude using the third attitude. The correction in step S37 is performed by bringing the first posture matrix M1 closer to the third posture matrix M3 at a predetermined ratio (the following constant C2). The CPU 10 reads the first attitude data 68 and the third attitude data 76 from the main memory. Then, correction is performed according to the following equation (13) using the first posture matrix M1 indicated by the first posture data 68 and the third posture matrix M3 indicated by the third posture data 76.
M1 = (M3-M1 ′) × C2 + M1 ′ (13)
In the above equation (13), the variable M1 ′ is the first posture matrix before correction. The constant C2 is preset in the range of 0 <C2 ≦ 1, and is set to 0.1, for example. Data indicating the corrected first posture matrix M1 calculated by the above equation (13) is stored in the main memory as new first posture data 68. After step S37, the CPU 10 ends the second correction process.

  As described above, in the second correction process, the third posture is calculated from the captured image (marker coordinates), and the first posture is corrected so as to approach the third posture. By this correction, the first posture can be corrected so as to have a more accurate value. In the present embodiment, the third posture is calculated from the captured image only for the posture in the roll direction and the yaw direction. However, as described above, the third posture is calculated from the captured image also in the pitch direction. In other embodiments, the third orientation may be calculated from the captured image in the roll direction, the yaw direction, and the pitch direction. In the second correction process, the third posture may be calculated in at least one of the roll direction, the yaw direction, and the pitch direction.

  Returning to the description of FIG. 14, in step S <b> 7 following step S <b> 6, the CPU 10 executes a game process using the corrected first posture. This game process may be any process as long as the process reflects the first attitude matrix M1 representing the corrected first attitude as an input value in the game result. For example, it may be a process of controlling and displaying an object in the virtual game space so as to have the posture indicated by the first posture matrix M1, or the posture indicated by the first posture matrix M1 and a predetermined posture. It may be a process of controlling and displaying the object so as to move at a speed according to the angle. Following step S7, the process of step S8 is executed.

  In step S8, the CPU 10 determines whether or not to end the game. The determination in step S8 is made based on, for example, whether or not the game has been cleared, whether or not the game is over, and whether or not the player has given an instruction to stop the game. If the determination result of step S8 is negative, the process of step S3 is executed again. Thereafter, the processing loop of steps S3 to S8 is repeatedly executed until it is determined in step S8 that the game is to be ended. On the other hand, if the determination result of step S8 is affirmative, the CPU 10 ends the game process shown in FIG. This is the end of the description of the game process.

  As described above, in the present embodiment, the first posture of the input device 8 is calculated from the angular velocities detected by the gyro sensors 55 and 56 (step S4), and the first posture is subjected to the first correction process (S5). And it corrected by the 2nd correction process (S6). Since the game process is executed using the corrected first attitude (step S7), the CPU 10 can execute the game process based on the accurate attitude of the input device 8. According to this, for example, the posture of the input device 8 can be accurately reflected on the posture of the object in the game space, so that the operability of the game operation can be improved.

[Modification]
In the above embodiment, the case where a three-dimensional posture is calculated using a gyro sensor that detects angular velocities around three axes has been described as an example. However, as illustrated in FIGS. The present invention can also be applied when calculating a posture (rotation angle) on a two-dimensional plane. The posture on the two-dimensional plane may be calculated by detecting an angular velocity about two axes with a two-axis gyro sensor, or calculated by detecting an angular velocity about a predetermined axis with a one-axis gyro sensor. May be.

  In another embodiment, the second correction process may be executed only when it is estimated that the input device 8 is imaging the marker unit 6. Specifically, before executing the second correction process, the CPU 10 determines whether or not the input device 8 (imaging unit) faces a direction in which the marker unit 6 can be imaged. This determination can be performed using the first posture or the second posture. For example, the first posture (or the second posture) may be determined as to whether the imaging direction of the input device 8 is the same or opposite to the direction from the input device 8 to the marker unit 6. In addition, the first posture used for the above determination may be the first posture in which the first and second correction processes are performed in the previous processing loop, or may be calculated in the current processing loop. The first posture in which one correction process is performed may be used.

  As a result of the above determination, when it is determined that the input device 8 faces the direction in which the marker unit 6 can be imaged, the CPU 10 performs the second correction process and faces the direction in which the marker unit 6 can be imaged. If it is determined that there is not, the second correction process is skipped. Note that an object that is not the marker unit 6 (for example, an electric lamp in a room or sunlight outside a window) may be erroneously detected as the marker unit 6, and the third posture is obtained using the marker coordinates obtained by the erroneous detection. When is calculated, correct correction cannot be performed even if the second correction process is performed using the third posture. On the other hand, by performing the above determination process, it is possible to prevent the second correction process from being performed using the third posture calculated from the marker coordinates obtained by the erroneous detection. As a result, the second correction process can be performed more accurately.

  As described above, the present invention is used, for example, as a game device or a game program for performing game processing according to the posture of the input device, for example, for accurately calculating the posture of the input device using a gyro sensor. It is possible.

External view of game system Functional block diagram of game device The perspective view which shows the external appearance structure of an input device Perspective view showing the external configuration of the controller Diagram showing the internal structure of the controller Diagram showing the internal structure of the controller Block diagram showing the configuration of the input device The figure which shows the vector which shows a 1st attitude | position and a 2nd attitude | position The figure which shows vector v3 which shows correction amount The figure which shows the vector which shows the 1st attitude | position after correction | amendment by a 1st correction process. The figure which shows the vector which shows a 1st attitude | position and a 3rd attitude | position The figure which shows the 1st attitude | position after correction | amendment by a 2nd correction process. The figure which shows the main data memorize | stored in the main memory of a game device Main flowchart showing the flow of processing executed in the game device Flowchart showing the flow of the first correction process (step S5) shown in FIG. Flowchart showing the flow of the second correction process (step S6) shown in FIG. The figure which shows the two-dimensional coordinate corresponding to a captured image

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Game system 2 Television 3 Game device 4 Optical disk 5 Controller 6 Marker part 7 Gyro sensor unit 8 Input device 10 CPU
11c GPU
11e Internal main memory 12 External main memory 63 Angular velocity data 64 Acceleration data 65 Marker coordinate data 68 First attitude data 76 Third attitude data

Claims (8)

An attitude calculation device that acquires data from an input device including at least a gyro sensor, an acceleration sensor, and an imaging unit, and calculates an attitude of the input device,
Attitude calculating means for calculating a first attitude of the input device based on an angular velocity detected by the gyro sensor;
The orientation of the input device when the direction of acceleration indicated by the acceleration data detected by the acceleration sensor is vertically downward is the second orientation, and the first acceleration is closer to the gravitational acceleration. First correction means for performing correction to bring the first posture closer to the second posture so that the posture becomes closer to the second posture;
The posture of the input device calculated from the orientation and / or position of a predetermined imaging target in an image captured by the imaging means is set as a third posture, and the first posture is set at a predetermined ratio to the third posture. Second correction means for correcting the posture to approach,
The first correction means performs the correction only when the difference between the magnitude of acceleration and the magnitude of gravitational acceleration is smaller than a predetermined reference,
The second correction means is based on the orientation of the imaging target in the image picked up by the imaging means, and the posture related to the roll direction when the imaging direction of the imaging means is used as a reference among the third postures. , And the attitude related to the pitch and / or yaw direction when the imaging direction of the imaging means is used as a reference among the third attitudes based on the position of the imaging target in the image. , Posture calculation device.   The second correction unit determines whether the imaging unit is in a direction in which the predetermined imaging target can be captured based on the first posture, and the imaging unit captures the predetermined imaging target. The posture calculation apparatus according to claim 1, wherein the correction is performed only when the direction is possible.   The posture calculation apparatus according to claim 1, wherein the second correction unit corrects the first posture corrected by the first correction unit.   A game device that performs a game process using the first posture corrected by the posture calculation device according to any one of claims 1 to 3 as a posture of the input device. An attitude calculation program executed in a computer of an attitude calculation device that acquires data from an input device including at least a gyro sensor, an acceleration sensor, and an imaging unit, and calculates an attitude of the input device,
Attitude calculating means for calculating a first attitude of the input device based on an angular velocity detected by the gyro sensor;
The orientation of the input device when the direction of acceleration indicated by the acceleration data detected by the acceleration sensor is vertically downward is the second orientation, and the first acceleration is closer to the gravitational acceleration. First correction means for performing correction to bring the first posture closer to the second posture so that the posture becomes closer to the second posture;
The posture of the input device calculated from the orientation and / or position of a predetermined imaging target in an image captured by the imaging means is set as a third posture, and the first posture is set at a predetermined ratio to the third posture. Causing the computer to function as second correction means for correcting the posture to approach,
The first correction means performs the correction only when the difference between the magnitude of acceleration and the magnitude of gravitational acceleration is smaller than a predetermined reference,
The second correction means is based on the orientation of the imaging target in the image picked up by the imaging means, and the posture related to the roll direction when the imaging direction of the imaging means is used as a reference among the third postures. , And the attitude related to the pitch and / or yaw direction when the imaging direction of the imaging means is used as a reference among the third attitudes based on the position of the imaging target in the image. , Posture calculation program.   The second correction unit determines whether the imaging unit is in a direction in which the predetermined imaging target can be captured based on the first posture, and the imaging unit captures the predetermined imaging target. The posture calculation program according to claim 5, wherein the correction is performed only when the direction is possible.   The attitude calculation program according to claim 5 or 6, wherein the second correction means corrects the first attitude corrected by the first correction means.   A game program for performing a game process using the first attitude corrected by the attitude calculation program according to claim 5 as an attitude of the input device.

Images