WO2007129432A1

WO2007129432A1 - Game device

Info

Publication number: WO2007129432A1
Application number: PCT/JP2007/000123
Authority: WO
Original assignee: Sony Computer Entertainment Inc.; Saito, Katsu
Priority date: 2006-05-02
Filing date: 2007-02-23
Publication date: 2007-11-15
Also published as: JP5089078B2; JP2007296248A

Abstract

A game device includes an input reception unit (400) for receiving an operation input inputted from a game controller and an object control unit (406) for controlling the operation of an object in a virtual space according to the operation input received by the input reception unit. An acquisition unit (402) acquires position information or posture information on the game controller in the real space. The object control unit (406) controls the operation of the object by using the position information or the posture information.

Description

Specification

Game device

Technical field

The present invention relates to a game device that executes a game application.

Background art

[0002] In recent years, code data is recognized by imaging a two-dimensional code with a camera and the code data is recognized.

There has been proposed a technique for executing a predetermined process associated with a stored data. Compared with one-dimensional barcodes, two-dimensional codes can be encoded with a larger amount of information, and various types of two-dimensional codes are now in practical use. Under such circumstances, there has been proposed a technique related to image recognition of a two-dimensional code (for example, see Patent Document 1).

There are various measurement systems that use a magnetic method, an ultrasonic method, or an optical method to acquire the three-dimensional position and orientation of an object in real space. Magnetic measurement systems are susceptible to environmental electronic devices and geomagnetism, and ultrasonic measurement systems are susceptible to environmental pressure and temperature. On the other hand, an optical measurement system uses a captured image captured by a camera and is not affected by the surrounding magnetic field, atmospheric pressure, or temperature. As a measurement system that uses an optical method, a plurality of LEDs provided in an objet can be blinked with a uniquely defined blink pattern, and a 3D position and posture can be captured using a 2D image sensor. Have been proposed (see, for example, Patent Document 2). In Patent Document 2, an LED that functions as a marker has identification information that can be uniquely identified for each LED.

Patent Document 1: Japanese Patent Laid-Open No. 2 0 06 _7 2 6 6 7

Patent Document 2: Japanese Patent Laid-Open No. 2 0 0 3 _ 2 5 4 7 1 6

Disclosure of the invention

Problems to be solved by the invention [0004] With recent technological advances, the hardware specifications of game systems have been dramatically improved. Recently, a game system has been realized in which a game device optically recognizes a user's movement and uses it as an input to a game when the user moves his body in front of a powerful camera by connecting a camera and a game device. ing. In such a game environment, user demands for game applications tend to diversify.

[0005] In order to satisfy the diversified demands of users, the present inventor uses a camera connected to a game device to extract information of a captured image taken by a game controller as input information of the game device. As a result, we found the possibility of realizing an unprecedented game application. We also found the possibility of realizing an unprecedented game application by installing an acceleration sensor in the game controller and using the measured information as input information for the game device. In addition to game systems, it is thought that effective applications can be realized in other environments, such as entertainment systems such as educational sites and business sites.

Means for solving the problem

[0006] In order to solve the above problems, a game device according to an aspect of the present invention is based on an input receiving unit that receives an operation input input from a game controller, and an operation input received by the input receiving unit. And an object control unit for controlling the movement of the object in the space. The game apparatus further includes an acquisition unit that acquires position information or posture information in the real space of the game controller, and the object control unit controls the movement of the object using the position information or the posture information. .

The invention's effect

According to the present invention, it is possible to provide a game device that executes a new application using new input information.

Brief Description of Drawings

FIG. 1 is a diagram showing a use environment of a game system according to an embodiment of the present invention. FIG. 2 is a diagram showing an external configuration of a controller.

FIG. 3 is a diagram showing an external configuration of the back side of the controller.

FIG. 4 is a diagram showing an internal configuration of a controller.

FIG. 5 is a diagram showing a configuration of a game device.

FIG. 6 is a diagram illustrating a configuration of an image processing unit.

[FIG. 7] (a) is a diagram showing a current frame image, and (b) is a diagram showing a difference image between the current frame image and the previous frame image.

[Fig. 8] (a) is a diagram showing a binarized image of the current frame image, and (b) is a diagram showing a logical product image.

FIG. 9 is a flowchart showing the processing procedure of LED image candidate detection processing.

FIG. 10 is a flowchart showing the processing procedure of LED extraction processing.

[Fig. 1 1] S 1 2 2 in FIG. 10 is a flow chart showing the processing procedure of the second LED detection process shown in FIG.

[FIG. 12] S 1 2 4 in FIG. 10 is a flowchart showing a processing procedure of the third LED detection processing shown.

FIG. 13 is a flow chart showing the procedure of the fourth LED detection process shown in S 1 2 6 of FIG.

FIG. 14 is a diagram for explaining an algorithm for the virtual object corresponding to the controller to realize a smooth follow-up operation in the game screen.

FIG. 15 is a diagram showing a controller having a two-dimensional code.

FIG. 16 is a diagram showing an example of a two-dimensional pattern constituting a two-dimensional code.

FIG. 17 is a diagram illustrating a configuration of an image processing unit.

FIG. 18 is a diagram showing an internal configuration of a controller according to an example.

FIG. 19 is a diagram showing a configuration of a game device according to an example.

FIG. 20 is a diagram showing a configuration of an application processing unit.

[Fig.21] An example of character movement is shown.

Explanation of symbols

1 ■ ■ ■ Game system, 2 ■ ■ ■ Imaging device, 3 ■ ■ ■ Image display device, 4 ■ ■ ■ Voice output device, 1 0 · ■ ■ Game device, 20 · ■ 'Controller, 30 0 ■ ■ ■ 3-axis acceleration sensor, 302 · ■ ■ Jia mouth sensor, 1 1 0 · ■ ■ Application processing unit, 400 ■ ■ ■ Input acceptance unit, 402 ■ ■ ■ Acquisition unit, 4 04 ■ ■ ■ Change detection unit, 406 · ■ ■ Object control unit, 408 · ■ 'Image signal generation unit.

BEST MODE FOR CARRYING OUT THE INVENTION

[0010] An embodiment of the present invention provides a game apparatus capable of acquiring position information and / or posture information in a real space of a game controller and executing a game application that can be executed based on the information. Therefore, in this specification, first, a technique for obtaining the position information and / or posture information of the game controller is shown. The following shows (1) a method using LED, (2) a method using a two-dimensional code, (3) a method using a measurement value of an acceleration sensor, etc., followed by position information and / or posture of a game controller. Indicates game applications that can be executed based on the information.

[0011] (1) Acquisition method of position information and / or posture information using LED

FIG. 1 shows a use environment of a game system according to an embodiment of the present invention. The game system 1 includes an imaging device 2, an image display device 3, an audio output device 4, a game device 10 and a controller 20. The imaging device 2, the image display device 3, the audio output device 4 and the controller 20 are connected to the game device 10.

[0012] The controller 20 is an operation input device for a user to perform an operation input, and the game device 10 processes a game application based on the operation input in the controller 20, and the processing result of the game application Is a processing device for generating an image signal indicating The technology shown in the present embodiment can be realized not only in a game application but also in an entertainment system provided with a processing device that executes other types of applications. In the following, a game system 1 that executes a game application will be described on behalf of the entertainment system.

[0013] The imaging device 2 includes a CCD image sensor or a CMOS image sensor. This is a video camera that captures real space with a predetermined period and generates a frame image for each period. For example, the imaging speed of the imaging device 2 may be set to 30 frames and may be matched with the frame rate of the image display device 3. The imaging device 2 is connected to the game device 10 via a USB (universal serial bus) or other interface.

The image display device 3 is a display that outputs an image signal, and is a game device

10 In response to the image signal generated in 0, the game screen is displayed. The sound output device 4 is a speaker that outputs sound, and receives a sound signal generated by the game device 10 and outputs game sound. The image display device 3 and the audio output device 4 constitute an output device in the game system 1.

[0015] Game device 10 and image display device 3 may be connected by wire or may be connected wirelessly. The game apparatus 10 and the image display apparatus 3 may be connected by an AV cable. In addition, a home network constructed with a network (L A N) cable or a wireless L A N may be constructed between the game apparatus 10 and the image display apparatus 3.

The controller 20 has a function of transmitting an operation input by a user to the game apparatus 10, and is configured as a wireless controller capable of wireless communication with the game apparatus 10 in this embodiment. The controller 20 and the game device 10 may establish a wireless connection using the Bluetooth (registered trademark) protocol. The game apparatus 10 can wirelessly connect with a plurality of controllers 20, that is, in the game system 1, a 1-to-N connection between the game apparatus 10 and the controller 20 can be realized. The game device 10 functions as a parent device or master, and the controller 20 functions as a child device or slave. The controller 20 is not limited to a wireless controller, and may be a wired controller connected to the game apparatus 10 via a cable.

[0017] The controller 20 is driven by a battery (not shown) and has a plurality of buttons and keys for performing an operation input for advancing the game. When the user operates the controller 20 key, the operation input becomes wireless. To the game device 10. The game apparatus 10 receives an operation input related to the game application from the controller 20, controls the progress of the game according to the operation input, and generates a game image signal and a game sound signal. The generated game image signal and game sound signal are output by the image display device 3 and the sound output device 4, respectively. The game apparatus 10 also has a function of transmitting a vibration control signal that vibrates the controller 20 to the controller 20 in accordance with the progress of the game application. The controller 20 has a vibrator and vibrates the vibrator when receiving a vibration control signal.

[0018] In the game system 1 of the present embodiment, the controller 20 includes a plurality of light emitting elements. The plurality of light emitting elements are LEDs of the same color, and serve as indicators that represent controller numbers set in the game application. The controller number in the game application is used by the user, for example, when selecting a game character at the start of the game, so it is necessary to notify the user of the controller number by some means. Therefore, in controller 20, for example, if the first LED is lit, the controller number is 1, and if the second LED is lit, the controller number is 2. Inform the user of the controller number. It is also possible to express the controller number by combining multiple LEDs.

When the controller 20 is a wired controller, it is possible to determine the controller number at the position where the cable connector extending from the controller 20 is inserted into the port of the game apparatus 10. However, when using a multi-port device with multiple ports externally attached to the game device 10, the user cannot know the controller number immediately, so use multiple LEDs. It is preferable to notify the user of the controller number.

[0020] In the game system 1, the LED of the controller 20 is used not only as an indicator of the controller number, but also as a game input that affects the progress of the game application. In this case, the LED control is Instead of lighting as an indicator, it can be switched to lighting as input to the game application. The imaging device 2 captures the LED of the controller 20, generates a frame image, and supplies it to the game device 10. The game apparatus 10 acquires the frame image, estimates the position and posture in the frame image of the controller 20 from the position of the LED image in the frame image, and acquires position information and posture information in the real space. The acquired position information and / or posture information is reflected in the processing of the game application. That is, the game apparatus 10 of this embodiment processes the game application using not only the operation input input by the controller 20 but also the acquired position information and posture information of the controller 20, and the processing result It has a function to generate an image signal indicating.

[0021] FIG. 2 shows an external configuration of the controller. The controller 20 is provided with a direction key 21, an analog stick 27, and four types of operation buttons 26. The direction key 21, the analog stick 27, and the operation button 26 are input units provided on the upper surface 30 of the casing. The four types of buttons 2 2 to 25 are marked with different figures in different colors to distinguish them. That is, ○ button 2 2 has a red circle, X button 2 3 has a blue bag, mouth button 24 has a purple square, and Δ button 25 has a green triangle. A plurality of LEDs are provided on the rear surface 29 of the controller 20.

[0022] The user grips the left gripping portion 28a with his left hand, grips the right gripping portion 28b with his right hand, and operates the controller 20. Direction keys 2 1, analog stick 2 7, and operation buttons 2 6 are provided on the top surface of the chassis 30 so that the user can operate with the left gripping part 2 8 a and right gripping part 2 8 b gripped. It is done.

[0023] The upper surface 30 of the housing is also provided with a button 31 with LED. The button with LED 3 1 is used as a button for displaying a menu screen on the image display device 3, for example. In addition, it has a function to indicate the arrival of e-mail to Reuser according to the light emission status of the LED and to indicate the charge status of the controller 20 battery. For example, red when charging, green when charging is complete, When the remaining capacity is low, turn on the LED so that it flashes red.

Since the user plays a game while looking at the game screen displayed on the image display device 3, the image display device 3 exists in front of the controller 20 indicated by an arrow A. For this reason, the housing rear surface 29 on which the LEDs are arranged normally faces the image display device 3. In the present embodiment, the imaging device 2 needs to capture an LED during execution of the game application. Therefore, the imaging device 2 is arranged so that the imaging range faces the same direction as the image display device 3, and the rear surface of the controller 20 housing It is preferable to face 29. In general, since a user often plays a game in front of the image display device 3, the imaging device 2 is arranged so that the direction of the optical axis thereof coincides with the front direction of the image display device 3. Specifically, the imaging device 2 is preferably arranged in the vicinity of the image display device 3 so that the imaging range includes a position where the user can visually recognize the display screen of the image display device 3. Thereby, the imaging device 2 can image the user's body and the controller 20.

FIG. 3 shows an external configuration of the back side of the controller. In FIG. 3, illustrations of the direction key 21 and the operation button 26 on the upper surface 30 of the housing of the controller 20 are omitted. The LED rear area 29 is provided on the rear surface 29 of the housing, and the first LE D40a, the second LE D40b, the third LE D40c, and the fourth LE D40d are disposed in the LED arrangement area 42. . Hereinafter, the first LEDD 40a, the second LEDD 40b, the third LEDD40c, and the fourth LEDD40d are collectively referred to as "LEDD40". The LED arrangement region 42 is formed in the central region of the housing back surface 29. A USB connector 46 is provided at the center of the LED arrangement region 42. A USB cable extending from the game apparatus 10 is connected to the USB connector 46 so that the controller 20 can be charged. If the USB cable is connected, the controller 20 can be used as a wired controller.

[0026] Operation buttons 48a, 48b, 48c, 4801 (1_ £ D placement area 42) are provided on the left and right sides of the housing rear 29 side. Operation buttons 48 a, 48 b, 4 8c and 48d are provided at positions where the user can operate with the tip of the index finger even when the user holds the left grip 28a and the right grip 28b. As a result, the LED 40 is not hidden by the index finger when operating the operation buttons 48a, 48b, 48c, and 48d.

The imaging device 2 acquires RGB luminance values at each pixel. In order to accurately detect the emitted LED 40, it is preferable that the contrast between the LED 40 and the area around the LED 40 is large. Therefore, the LED arrangement area 42 has a color with a lightness lower than that of the adjacent housing color, for example, a black color scheme. The LED arrangement region 42 is configured as a black translucent plate, and the LED 40 is provided behind it, that is, inside the housing. As a result, the LED 40 is not visible from the imaging device 2 unless it is lit, and an image is captured only when it is lit. The black translucent plate also has a function of diffusing light and changing the narrow directivity of LED 40 to a wide directivity. By providing the LED 40 in the LED arrangement area 42, the contrast with the emitted LED 40 can be increased, and the LED image can be effectively extracted from the frame image in the subsequent image processing.

[0028] The first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d are arranged in a predetermined two-dimensional pattern. For example, the first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d are disposed at positions that constitute the vertex of the rectangle. The game apparatus 10 knows this positional relationship in advance and uses it for the extraction process of the LED image. Numbers are stamped or printed in the vicinity of each LED 40, and the user can know the controller number by looking at the numbers in the vicinity of the lit LED 40.

FIG. 4 shows the internal configuration of the controller. The controller 20 includes a wireless communication module 58, a processing unit 60, and a 1_ 曰 040 and a vibrator 44. The wireless communication module 58 has a function of transmitting / receiving data to / from the wireless communication module of the game apparatus 10. The processing unit 60 executes a predetermined process in the controller 20.

[0030] The processing unit 60 includes a main control unit 50, an input receiving unit 52, a P WM control unit 54, And a drive unit 56. The main control unit 50 transmits and receives necessary data to and from the wireless communication module 58.

[0031] The input reception unit 52 includes a direction key 21, an operation button 26, and an analog stick.

Accepts input information from the input section such as 27 and sends it to the main control section 50. The main control unit 50 supplies the received input information to the wireless communication module 58, and the wireless communication module 58 transmits the input information to the game apparatus 10 at a predetermined timing. Further, when the wireless communication module 58 receives the vibration control signal from the game apparatus 10, the wireless communication module 58 supplies the vibration control signal to the main control unit 50, and the main control unit 50 vibrates the vibrator 44 based on the vibration control signal. Operate 56. The drive unit 56 may be configured as a switch for driving the vibrator 44, or may be configured as a PWM control unit that varies the duty ratio of the supply voltage.

In this embodiment, the \ ^ 1 \ 1 control unit 54 includes a first PWM control unit 54 a, a second PWM control unit 54 b, a third PWM control unit 54 c, and a fourth PWM control unit 54 d. Composed. First \ ^ 1 \ 1 control unit 543, second PWM control unit 54b, first

The 3 PWM control unit 54 c and the fourth PWM control unit 54 d are provided to control lighting of the first LED 40 a, the second LED 40 b, the third LED 40 c, and the fourth LED 40 d, respectively. The \ ^ 1 \ 1 control unit 54 controls the voltage applied to the LED 40 by PWM (pulse width modulation). For example, the PWM control unit 54 can adjust the luminance of the LED 40 by PWM controlling the applied voltage at a high frequency of several kilohertz. Moreover, the \ ^ 1 \ 1 control unit 54 can make the imaging device 2 recognize on / off of the lighting of the LED 40 by PWM control of the applied voltage at a low frequency of about several Hz to 10 OHz. .

In the game system 1 of the present embodiment, first, a controller 20 of a user who wishes to participate in the game system 1 establishes communication with the game apparatus 10. At this time, the identification information of the controller 20, for example, the Bluetooth address, is transferred to the game apparatus 10, and the subsequent communication is performed based on the Bluetooth address. If Bluetooth is not used as the communication protocol, a device address such as a MAC address may be used. And. After the connection is established, the user can participate in the game application.

At this time, information indicating the controller number is transmitted from the game apparatus 10 to the controller 20. Based on this number instruction information, the main control unit 50 lights only the LED 40 corresponding to the set controller number. This allows the Reuser to know the controller number of its own controller 20. Instead of the number instruction information, a control signal for instructing lighting of LED 40 corresponding to the controller number may be transmitted from the game apparatus 10. In this way, the main control unit 50 can light any LED 40.

[0035] In the game system 1 of the present embodiment, the game apparatus 10 includes the controller 20 in the real space based on the frame image captured with all the LEDs 40a to 40d lit. Estimate position and orientation. The estimated position and orientation are used as input for the game application to be executed. The game apparatus 10 also accepts operation inputs such as direction keys 21 and operation buttons 26 of the controller 20. Based on the input information, the game apparatus 10 generates game parameters for the game application, controls the progress of the game, and generates AV data for the game image and game sound. The AV data is output from the image display device 3 and the audio output device 4. The user moves the controller 20 in the real space while looking at the game screen, and further plays a game by operating the direction key 21 and the operation buttons 26. Hereinafter, processing in the game apparatus 10 will be described.

FIG. 5 shows a configuration of the game device. The game device 10 includes an input unit 100, an image processing unit 1 0 2, a position estimation unit 1 0 4, a blinking instruction unit 1 0 6, a wireless communication module 1 0 8, an application processing unit 1 1 0 and an output Part 1 1 2 is provided. The processing functions of the game apparatus 10 in the present embodiment are realized by a CPU, a memory, a program associated with the memory, and the like, and here, a configuration realized by their cooperation is depicted. The program may be built in the game device 10 or supplied from the outside in a form stored in a recording medium. There may be. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof. In the illustrated example, the game apparatus 10 has functions as a CPU power image processing unit 10 0 2, a position estimation unit 1 0 4, a blinking instruction unit 1 0 6, and an abrasion processing unit 1 1 0. Note that the game device 10 may have a plurality of CPUs due to the hardware configuration. In such a case, one CPU functions as an application processing unit 110 that executes a game application, and other CPU power image processing unit 10 0 2 that processes a captured image by the imaging device 2, position estimation Part 1 0 4, may function as blinking instruction part 1 0 6

The wireless communication module 10 8 establishes wireless communication with the wireless communication module 58 of the controller 20. In the synchronization establishment phase, connection inquiry, that is, “inquiry” processing is performed on terminal devices including the peripheral controller 20 in the wireless communication module 10 8 power. Specifically, the wireless communication module broadcasts IQ (inquiry) packets to terminal devices near 10 8 power. The wireless communication module 5 8 of the controller 20 that received the IQ bucket is the FHS packet containing the Bluetooth address and Bluetooth clock information (Frequency Hop Synchronize on) packet ¾r7 "—on the device 10 Reply _{Q In} the transmission / reception at this point in time, since the consent regarding the frequency hobbing pattern has not been established between the gaming device 10 and the controller 20, the fixed hopping pattern defined exclusively for the inquiry is used.

[0038] The wireless communication module 1 0 8 receives the FHS packet from the controller 2 0, grasps what kind of controller 2 0 exists, and then transmits the ID packet to the specific controller 2 0. To do. This is a “calling” process by the wireless communication module 10 8. When a response to the ID packet is returned from the specific controller 20, the wireless communication module 10 8 sends an FHS bucket to the controller 20, and informs the controller 20 of its address and clock. As a result, the game device 10 and the controller 20 are the same. Hopping patterns can be shared.

When “calling” is performed, a piconet is formed between the controller 20 and the game apparatus 10, and the “connected” state is entered. A piconet is a network that is temporarily formed between terminals when the Bluetooth terminals are brought close together. Up to 8 Bluetooth terminals can participate in a single piconet. Can do. The connected controller 20 is assigned a slave identifier, that is, a 3-bit address (1 to 7) given to the connected controller 20 from the wireless communication module 108. This slave identifier is called AM_ADDR (active member one address). When in the “connected” state, control packets for communication link setting are transmitted and received, which enables “data transfer”.

[0040] Although an example in which the controller 20 is called from the game device 10 has been described above, the game device 10 may be called from the controller 20. In this case, the controller 20 executes the calling process while the game apparatus 10 is executing the page scan. As a result, the controller 20 can call the game device 10 and enter the connected state. Thereafter, the roles of the master and the slave are switched, and the game device 10 becomes the master and the controller 20 becomes the slave. By realizing such a master-slave relationship, the controller 20 can participate in the game application.

[0041] When the controller 20 enters the game application when connected, the application processing unit 110 assigns the controller number of the controller 20 and generates number instruction information indicating the controller number. The wireless communication module 10 8 transmits the number instruction information to the controller 20. When controller 20 receives the number indication information, it turns on the corresponding LED 40. The wireless communication module 10 8 may transmit a control signal for instructing lighting of the LED 40 corresponding to the controller number to the controller 20 instead of the number instruction information.

[0042] When the game application is started, the blinking instruction unit 10 06 instructs the controller 20 to instruct a blinking pattern to blink at a predetermined cycle. Generate a signal. This blinking control signal is transmitted from the wireless communication module 10 8 to the controller 20.

The input unit 100 is configured as a USB interface, and acquires a frame image from the imaging device 2 at a predetermined imaging speed (for example, 30 frame seconds). The image processing unit 10 2 extracts an LED image from the frame image according to an LED extraction algorithm described later. The position estimation unit 10 4 acquires the position information of the LED image from the extracted LED image, and estimates the position and orientation of the controller 20. The position estimation unit 104 may function as a position acquisition unit because it acquires the position information of the LED image. It should be noted that the position of the controller 20 may be set at the center of gravity of the four LED images.

[0044] The position information and posture information of the controller 20 estimated from the positions and postures of the four LEDs 40 that are lit are used as input of the game application. Therefore, the position information and orientation information of the controller 20 estimated by the position estimation unit 104 are sequentially sent to the application processing unit 110 and reflected in the application processing. Input to the application processing unit 110 is also transmitted via the wireless communication module 10 08 when the user operates the direction key 21 of the controller 20 or the like. The application processing unit 110 progresses the game based on the position / posture information of the controller 20 and the operation input in the controller 20, and generates an image signal and an audio signal indicating the processing result of the game application. The image signal and the audio signal are sent from the output unit 1 1 2 to the image display device 3 and the audio output device 4, respectively, and output.

FIG. 6 shows a configuration of the image processing unit. The image processing unit 1 0 2 is a frame image acquisition unit 1 3 0, a current frame buffer 1 3 2, a previous frame buffer 1 3 4, a differential image generation unit 1 3 6, a binary image generation unit 1 3 8, a logic A calculation unit 1 4 0 and an LED image extraction unit 1 4 2 are provided.

The frame image acquisition unit 1 3 0 acquires frame image data from the input unit 1 100 and temporarily stores it in the current frame buffer 1 3 2. The previous frame buffer 1 3 4 stores the previous frame image data. Difference image generator 136 generates a differential image between the current frame data stored in the current frame buffer 1 32 and the previous frame data stored in the previous frame buffer 1334. The R pixel value of the previous frame data at the coordinates (x, y) in the frame image is P r, the G pixel value is P g, the B pixel value is P b, and the R pixel value of the current frame data is C r, G When the pixel value is Cg and the B pixel value is Cb, the difference image function F (x, y) is expressed by the following equation.

[0047] [Equation 1]

F (x, y) = y (Pr-Cr) ² + (Pg-Cg) ² + (Pb-Cb) ² -Th ■ ■ ■ (1)

Here, T h is a predetermined threshold value. If F (x, y) is greater than 0, the pixel value of the coordinate (x, y) is encoded as "1", and the pixel is white on the display. On the other hand, if F (x, y) is 0 or less, the pixel value is encoded as "0" and the pixel is black on the display. In this manner, the difference image generated by the difference image generation unit 136 is a binarized image. By generating the difference image, it is possible to remove the influence of objects that do not move between the previous frame image and the current frame image.

[0049] The binarized image generation unit 138 binarizes the current frame data using a predetermined threshold value, and generates a binarized image. The binarized image generation unit 1 38 encodes the pixel value of the pixel holding the luminance greater than the predetermined threshold value to “1”, and sets the pixel value of the pixel holding the luminance value below the predetermined threshold to “0”. Encode. By generating a binary image of the current frame image, it is possible to extract only bright objects in the current frame image.

[0050] The logical operation unit 140 generates a logical operation image obtained by performing a logical operation on the difference image generated by the difference image generation unit 136 and the binarized image generated by the binarized image generation unit 1 38. Generate. Specifically, the logical operation unit 140 generates a logical product image by performing a logical product operation on the pixel value of the difference image and the pixel value of the binarized image for each pixel. This is generated by mathematically ANDing the pixel values of corresponding pixels in the difference image and the binarized image. [0051] Fig. 7 (a) shows the current frame image. In the current frame image, the user grasps the controller 20 and plays a game, and the right side is illuminated. At this time, all LEDs 40 of the controller 20 are lit. All LEDs 40 emit light with the same brightness by the same lighting pattern.

Figure 7 (b) shows the difference image between the current frame image and the previous frame image. When the controller 20 is moved from the position of the previous frame image, a difference image is generated by Equation (1). In this example, the controller 20 and at least a part of the body moving together with the controller 20 are acquired as a white image (the pixel value is “1”). Note that such a white image is not always acquired, and FIG. 7 (b) is merely an example of the difference image generated when the mathematical expression (1) is satisfied. At this time, since the illumination does not move, it is not included in the difference image. If the user does not move the controller 20, the white image of the controller 20 is not acquired.

[0052] FIG. 8 (a) shows a binarized image of the current frame image. In the binarized image,

The high-brightness area in 7 (a), that is, the LED that emits light, and the illumination lamp are binarized and converted to a white image. In order to prevent other noises, it is preferable that the threshold value set in the binarization process is set to a luminance value obtained by subtracting a predetermined margin value from the LED emission luminance.

Figure 8 (b) shows the logical product image. The logical product image performs a logical product operation on the difference image shown in Fig. 7 (b) and the binarized image shown in Fig. 8 (a). In the difference image and the binarized image, the pixel value of the white pixel is “1”, and the pixel value of the black pixel is “0”. Therefore, only pixels that exist as white pixels in both images will be generated as white pixels in the AND image.

[0053] By taking the logical product of the difference image and the binarized image in this way, it is possible to generate a logical product image that is easy to extract the LED image. The logical product image is sent to the 1_ 曰 0 image extraction unit 1 4 2. When the AND image is generated, the previous frame data held in the previous frame buffer 1 34 is overwritten by the current frame data held in the current frame buffer 1 32. This overwriting process may be performed immediately after the previous frame buffer 1 3 4 is read by the difference image generation unit 136. Thereafter, the current frame data held in the current frame buffer 1 32 is overwritten by the next frame data, and the logical product image generation process in the next frame is executed.

[0055] The 1_ 曰 0 image extraction unit 142 extracts an LED image from the logical product image. In order to extract the LED image, first, candidate LED images are detected. Next, assuming that an arbitrary candidate is the first LED 40a, the surrounding area is searched to check whether there is a LED image candidate that can be the second LED 40b. If the second LED 40 b is found, the peripheral area of the first LED 40 a is searched again to investigate whether there is an LED image candidate that can become the third LED 40 c. If the third LED ED40c is found, the surrounding area is searched and an LED image candidate that can become the fourth LED40d is investigated. At this time, the LED image extraction unit 142 recognizes a two-dimensional pattern formed by the plurality of LEDs 40 in advance, and performs an LED image extraction process using the two-dimensional pattern. In the following, for convenience of explanation, when the first LED 40 a, the second LED 40 b, the third LED 40 c, and the fourth LED 40 d are arranged on the rear surface 29 of the controller 20 so as to form a square apex. Take as an example.

Note that the LED image extraction unit 1 42 does not use a logical product image, but uses a binary image.

It is also possible to extract LED images. The binarized image will also contain noise other than the LED image. For example, if the LED image cannot be acquired in the difference image, the binarized image will be used to obtain the position and orientation of the LED image. Can be detected. The processing procedure shown below can be executed using either a logical product image or a binarized image. However, the logical product image has fewer noise components other than the LED image, so it is better to use the logical product image. Processing speed is fast.

FIG. 9 is a flowchart showing a processing procedure for LED image candidate detection processing. The The LED image extraction unit 142 acquires, as a connected region, a region in which white pixels (pixels having a pixel value “1”) continuously exist in the logical product image (S 100). Next, the 1_ 巳 0 image extraction unit 142 extracts the edge portion of the white pixel connection region and determines the short side and the long side (S 102).

The LED image extraction unit 144 determines whether or not the short side is composed of pixels having a predetermined number of pixels Ma or more (S 104). When the short side is composed of pixels smaller than the number of pixels Ma (N in S 104), the LED image extraction unit 144 determines that the short image is not an LED image, and ends the processing of the white pixel connection region. When the short side is composed of pixels with the number of pixels Ma or more (Y in S 104), the LED image extraction unit 144 determines whether the long side is composed of pixels with the predetermined number of pixels Mb or less. (S 1 06). When the long side is composed of pixels larger than the number of pixels Mb (1 06 of 51006), the LED image extraction unit 142 determines that it is not an LED image, and ends the processing of the white pixel connection region. When the long side is composed of pixels with the number of pixels equal to or less than Mb (Y in S 106), the LED image extraction unit 144 acquires the number of pixels constituting the white pixel connection area as a candidate for the LED image. (S 1 08) This LED image candidate detection process is repeated for all white pixel connected regions. As a result, LED image candidates included in the logical product image can be detected. The coordinates of the LED image candidate are set at the center of gravity of the connected area.

FIG. 10 is a flowchart showing a processing procedure for LED extraction processing. The LED image extraction unit 142 assumes that an arbitrary LED image candidate among the detected LED image candidates is the first LED 40a (S120). Next, it is determined whether or not there exists an LED image candidate that satisfies the condition that can be the second LED 40 b for the first LED 40 a (S 122). If there is an LED image candidate that becomes the second LED 40 b (Y of S 1222), an LED image candidate that satisfies the conditions that can be the third LED 40 c is satisfied for the first LED 40 a and the second LED 40 b. It is determined whether or not it exists (S 1 24). If there is an LED image candidate that becomes the third LED 40 C (Y of S 1 24), it becomes the fourth LED 40 d for the first LED 40 a, the second LED 40 b, and the third LED 40 c. It is determined whether there is an LED image candidate that satisfies the following conditions (S 126). If there is an LED image candidate for the fourth LED 40d (S126, step S26), the first LED 40a to the fourth LED 40d of the controller 20 are extracted. If the second LED image candidate does not exist (N in S 1 22), if the third LED image candidate does not exist (S 1 24! \ 1), if the fourth LED image candidate does not exist (S 1 N) of 26 assumes that another LED image candidate is the first LE D40a, and re-executes the processing in Fig. 8.

In the LED extraction process, in order to reduce the processing load, it may be set as a processing condition that the attitude of the controller 20 is not inclined more than a predetermined angle. Since the LED 40 has a relatively strong directivity, if the controller 20 is tilted too much, it is difficult for the imaging device 2 to receive light. When searching for LED image candidates using the directivity of this LED 40, the LED image extraction process is performed on the condition that it is not inclined from the first LED 40a by a predetermined angle (for example, 45 °). As a result, the processing load can be greatly reduced.

In this embodiment, since the difference image between the previous frame image and the current frame image is used, the image of LED 40 is excluded from the difference image when the controller 20 does not move. Therefore, when using a difference image, the LED image is not always found by the LED extraction process shown in FIG. If the LED image is not found by the LED extraction process shown in FIG. 10, the position estimation unit 104 estimates that the controller 20 has not moved from the position of the previous frame. Get posture information. At this time, the position estimation unit 104 may extract the LED image using the binarized image.

FIG. 11 is a flowchart showing the procedure of the second LED detection process shown in S 122 of FIG. The number of pixels in the candidate LED image assumed to be the first LED 40a is N. First, the LED image extraction unit 142 determines whether or not the number of pixels of the LED image candidate is 12 XN or more and 2 N or less (S 140). If this is an LED image, the size and shape are as large as the 1st LE D40a. Based on the fact that it doesn't change. If the number of constituent pixels of the LED image candidate is within a predetermined range with respect to the number of constituent pixels N (丫 of S 140), whether or not the LED image candidate exists on the right side of the first LED 40a. (S 1 4 2). Judgment as to whether or not it is to the right is determined by whether or not it exists within the range of 45 ° from the upper right to 45 ° from the upper right of the first 1_ 曰 0403. When it is on the right side of the first LED 40a (丫 of S 1 42), it is determined whether or not the distance D ₁₂ between the first LED 40a and the LED image candidate is greater than D a and less than D b. Determine (S 1 4 4). Here, when the distance between each other is too close or too far away, it is determined that the second LED 40b is not suitable. If the distance D ₂ is less and D b be more than Da (丫of 1 44), it is determined that the LED image candidate may be the 2 LED 40 b, the presence of the 2 LED 40 b is confirmed (S 1 46). On the other hand, if the number of constituent pixels of the LED image candidate is not within the predetermined range (51 40 1 \ 1), if it does not exist to the right of the LED 40 a (N of S 1 42), the distance D ₂ is the predetermined distance If it is not within the range (1 \ 1 of S 144), it is determined that the LED image candidate is not the second LED 40b.

FIG. 12 is a flowchart showing the procedure of the third LED detection process shown in S 1 24 of FIG. The number of pixels in the candidate LED image assumed to be the first LED 40a is N. First, the LED image extraction unit 142 determines whether or not the number of pixels of the LED image candidate is 12 XN or more and 2 N or less (S 160). If the number of constituent pixels of the LED image candidate is within a predetermined range with respect to the number of constituent pixels N (S 1 60 丫), whether or not the LED image candidate exists below the first LED 40 a is determined. Judgment is made (S 1 62). Judgment as to whether or not it is below is determined by whether or not it exists within the range of 45 ° from the lower left to 45 ° of the lower left of the first 1_ 曰 0403. If it is below the first LED 40a (Y of S 1662), the distance between the first LED 40a and the LED image candidate is 17 "2 x D ₂ or more and V" 2 D ₂ or less. It is determined whether or not there is (S 1 64). Here, if the distance between each other is too close or too far away, it is determined that the third LED 40 C is not suitable. The distance is 1 V "2 XD ₂ or more and If V "2 D ₁₂ or less (S 1 64 Y), the angle of the line segment from the first LED 40a to the second LED 40b and the line segment from the first LED 40a to the LED image candidate is It is determined whether or not it is within the predetermined range (S 1 66) .At this time, if it is within the predetermined range (丫 of S 1 66), the LED image candidate can be the third LED 40 0 c. It is determined that the presence of the third LED 40 c is confirmed (S 1 68) On the other hand, if the number of constituent pixels of the LED image candidate is not within the predetermined range (N of S 1 60), the lower part of the LED 40 a Is not within the specified range (N of S 1 62), if the distance from the first LED 40a is not within the specified range (51 64 1 \ 1), if the angle is not within the specified range (S 1 66 1 \ 1), it is determined that the LED image candidate is not the third LED 40 c (S 1 70).

FIG. 13 is a flowchart showing the procedure of the fourth LED detection process shown in S 1 26 of FIG. The number of pixels in the candidate LED image assumed to be the first LED 40a is N. First, the LED image extraction unit 142 determines whether or not the number of pixels of the LED image candidate is 12 XN or more and 2 N or less (S 180). If the number of constituent pixels of the LED image candidate is within a predetermined range with respect to the number of constituent pixels N (Y of S 1 80), the vectors of the first LED 40a to the second LED 40 b, and the third LED 40 c It is determined whether or not the angle formed by the LED image candidate vectors is equal to or smaller than a predetermined value (S 1 82). If the angle between the vector of the first LED 40a to the second LED 40b and the vector of the third LED 40c and the LED image candidate is less than the predetermined value (Y in S 1 82), the first LED 40a It is determined whether or not the angle formed between the vector of the third LED 40c and the vector of the second LED 40b and the LED image candidate is equal to or smaller than a predetermined value (S184). If the angle between the vector of the first LED 40a to the third LED40c and the vector of the second LED40b and the LED image candidate is less than the predetermined value (丫 of S 1 84), then the ED image candidate Is determined to be the fourth LED 40 d and the presence of the fourth LED 4 O d is confirmed (S 186). On the other hand, when the number of constituent pixels of the LED image candidate is not within the predetermined range (N in S 1 80), the vectors of the first LED 40a to the second LED 40 b, and the third LED 40 C and the LED image candidate Vector If the angle formed exceeds the predetermined value (N in S 1 82), the first LED 40a to the first

3 If the angle between the LED 40 c vector and the second LED 40 b and the LED image candidate vector exceeds the specified value (S 1 84! \!), The LED image candidate is the 4th. It is determined that it is not LED40 d (S 1 88).

As described above, the 1_ 曰 0 image extraction unit 142 can extract the images of the first LED 40 a to the fourth LED 40 d from the logical product image. The extracted position information of the images of the first LED 40a to the fourth LED 40d is sent to the position estimation unit 104.

[0066] Based on the two-dimensional arrangement of the LEDs 40 in the controller 20, the position estimation unit 104 obtains position information regarding the frame images of the images of the first 1_ £ 0403 to the 41st_ £ 04001. The position estimation unit 104 calculates the affine transformation of the extracted first LED 40a, second LED 40b, third LED 40c, and fourth LED 40d, and calculates the position and orientation in real space. The position in real space is the coordinates (X, Y, Z) in the Cartesian coordinate system, and the attitude in real space is the rotation angle in the X, Y, and Z axes. By obtaining the coordinates and rotation angles in the three axes, the position estimation unit 104 can estimate the position in the space including the distance from the imaging device 2. The position estimation unit 104 may estimate the position of the controller 20 in the frame image by obtaining affine transformation. For example, the position in the frame image can be obtained by taking the XY coordinates in the XYZ space. The position estimation unit 104 sends the estimated position and orientation of the controller 20 to the application processing unit 110.

[0067] Although the position and orientation of the controller 20 may be calculated by affine transformation as described above, for example, the game apparatus 10 has three of the four LEDs 40 for a specific controller 20. It is also possible to obtain the position and orientation of the controller 20 from the three LED images that are lit. The game apparatus 10 uses three methods similar to the algorithm described above, and uses three LEDs 40, such as the first LED 40a and the second LED.

40 b, 3rd LED 40 c lighting images are extracted, and position estimation unit 104 power The position and orientation of the controller 20 may be estimated. At this time, the position estimation unit 104 knows the positional relationship among the first LE D40a, the second LE D40b, the third LE D40c, and the fourth LE D40d in advance, and the first LE D40a The position and orientation of the controller 20 can be estimated by extracting LED image candidates that satisfy the positional relationship between the second LED 40b and the third LED 40c.

[0068] In order to avoid erroneous recognition of LE D40, the position estimator 104 acquires this time when the position of the previously acquired LED image and the amount of change in the position of the LED image acquired this time exceed a predetermined value. Treat the position of the LED image as an error. In other words, if the position or posture changes by more than a certain value during one frame, it is possible that another object was misrecognized as LED40 due to a disturbing element, so the position of the extracted LED 40 Discard information.

[0069] However, even in such a case, the controller 20 may actually move with a large amount of change, so the position estimation unit 104 counts the number of errors that occur continuously. When the count value reaches a predetermined value, the position of the acquired LED image may be processed as normal

[0070] The position estimation unit 104, for example, based on the current frame image or logical product image,

It is also possible to estimate the moving direction and speed of LEDD40. For example, if the shutter speed of the imaging device 2 is lowered, the LED 40 image is captured as an ellipse or a rectangular parallelepiped. Therefore, it is possible to estimate not only the moving direction but also the moving speed from the shape of the LED image. For example, if the moving direction and speed can be estimated, the position and orientation of the controller 20 can be predicted even when the imaging device 2 suddenly disappears. If the shape of all the LEDs 40 is biased in the same direction, the moving direction and speed of the controller 20 may be estimated.

[0071] The application processing unit 110 reflects the estimated position and orientation of the controller 20 in the game application processing as game input. this There are various reflection methods, but most intuitively, the virtual object is displayed following the position in the game image corresponding to the estimated position of the controller 20. When the user moves the controller 20 with respect to the imaging device 2, the virtual object in the game image can be moved following the movement.

[0072] In an ideal state, the position and orientation of the LED 40 may be obtained for each frame image and reflected in the game image. On the other hand, depending on the orientation of the controller 20 and the ambient light environment, etc. A situation may occur in which the position and orientation of the LED 40 are not required in the frame image. If the position and posture of the LED 40 are suddenly requested after such a situation continues, the virtual object that has been stopped suddenly moves to a new position, giving the user a sense of incongruity. Therefore, even if the position and orientation of the LED 40 can be detected from the logical product image, the game image is not moved to the position where the virtual object was acquired, but if there is information loss. However, follow-up processing is performed so that smooth movement can be realized.

FIGS. 14 (a) and 14 (b) are diagrams for explaining an algorithm for the virtual object corresponding to the controller to realize a smooth follow-up operation in the game screen. For convenience of explanation, we will explain the change in the position of the virtual object, but the same algorithm can be applied to the change in posture.

[0074] In the position estimation unit 104, the position parameters derived by obtaining the affine transformation are P i-P i, P _{i +1} (the subscript of P indicates the game screen frame number). . First, consider the case where the frame image changes and the position parameter changes from P _i to P _{i +1} .

When a new position parameter (P _{i +1} ) is obtained, the position parameter to be reflected on the game screen is obtained by the following formula.

Q _{i +1} = (Q i + P _{i +} 2

That is, instead of moving directly to the new position parameter P _{i +1} , the position parameter of the previous frame in the game screen and the new position parameter of the current frame The midpoint of meter P _{i +1} is set as a new position parameter on the game screen. The position parameter Q _{i +1} is the location parameter Q i and need not be the midpoint position parameter one data P _{i +1,} the line segment between the position parameter Q i position parameter P _{i +} 1 A : It may be a point divided into B.

[0075] Fig. 14 (b) is an example of the case where P _{i +1} could not be acquired as the position parameter. In such a case, the effect of this algorithm can be demonstrated. In this case, the latest position parameter is used instead of P _{i +1} . Where the latest position parameter is P i,

Q _{i +1} = (Q i + P :) 2

As required.

When the position / parameter Q in the game image is obtained by this algorithm, the virtual object can always move little by little even if the controller 20 in the space is greatly displaced or cannot be extracted from the logical product image. Therefore, it is possible to avoid a situation in which the virtual object suddenly stops moving or suddenly moves in the game image, and smooth follow-up processing can be realized for the controller 20.

[0076] In this embodiment, the controller 20 is imaged by the imaging device 2, the position and orientation of the controller 20 are estimated from the frame image, and the position information and orientation information are reflected in the processing of the game application. did. The LED 40 of the controller 20 may change the lighting pattern according to the progress of the game application, for example.

[0077] Immediately after the user joins the game application, the game apparatus 10 may transmit a control signal for lighting the four LEDs 40 at a predetermined low frequency to the controller 20. The blinking instruction unit 106 generates a control signal that instructs the controller 20 to blink a predetermined low frequency. This low frequency blinking control signal is transmitted from the wireless communication module 108 to the controller 20. When there are a plurality of controllers 20, a blinking frequency is set for each controller 20. The controller 20 lights the LED 40 at the designated low frequency, and the imaging device 2 captures a frame image at an imaging speed of 30 frames. The game apparatus 10 extracts LED images that are turned on and off at predetermined intervals from a plurality of frame images. When there are multiple controllers 20 in the game system 1, the lighting cycle is set for each controller 20. As a result, the position of the controller 20 can be confirmed, and each controller 20 can be recognized separately. This process is ideally performed once after joining the game application, but the controller 20 may suddenly disappear from the imaging device 2 in the real space. Therefore, each controller 20 may blink the LED 40 at a low frequency set for each controller at a predetermined cycle such as every few seconds.

[0079] For example, when the four LEDs 40 are turned on and off at a low frequency, a frame in which the LEDs 40 can be imaged and a frame in which the LEDs 40 cannot be imaged are switched at a predetermined cycle. If the LED 40 is controlled to turn on at 45 Hz for the imaging cycle (30 frames / second), the situation is repeated where two consecutive frames are taken and the next is not taken. Since frame images may or may not be captured, position information can be acquired even when the LED 40 is stationary.

[0080] Here, low-frequency lighting means lighting at a frequency such that a frame in which LED light emission is captured and a frame in which imaging is not performed can exist with respect to the imaging cycle. On the other hand, high-frequency lighting means lighting at a frequency such that a frame image is always captured for the imaging cycle (30 frames / second) of the imaging device 2. For example, even if the LED 40 is turned on at 1 kHz, for example, if the imaging device 2 has a very high imaging speed, a frame in which LED emission is captured and a frame in which imaging is not performed appear. The position information can be obtained even when the LED 40 is stationary.

[0081] The position and posture of the LED 40 that is lit at a high frequency are used as input for the game application. Therefore, the controller estimated by the position estimation unit 104 The location information of LA 20 is sequentially sent to the application processing unit 110 and reflected in the processing of the abrasion. An input by the user operating the direction key 21 etc. of the controller 20 is also transmitted to the application processing unit 110 via the wireless communication module 1008. The application processing unit 110 progresses the game from the position / posture information of the controller 20 and the operation input from the controller 20, and generates an image signal and an audio signal indicating the processing result of the game application.

[0082] As a modification, it is also possible to recognize a specific LED 40 by lighting one of the four LEDs 40 with a blinking pattern different from that of the other LEDs 40. For example, three LEDs can be turned on at a high frequency, and one LED can be turned on at a low frequency, so that it is possible to check which of the first LEDs 40a in the frame image, for example. At this time, not only L E D 40 can be uniquely recognized, but also its posture can be acquired.

[0083] The features of the invention disclosed in the embodiments may be defined by the following items.

(Item 1)

An image processing apparatus that acquires the position of a light emitter image from an input frame image,

A difference image generation unit that generates a difference image between the previous frame image and the current frame image, a binarization image generation unit that generates a binarized image by binarizing the current frame image using a predetermined threshold;

A logical operation unit that generates a logical operation image obtained by performing a logical operation on the difference image and the binarized image;

An extractor for extracting a light emitter image from the logical operation image;

An image processing apparatus comprising: a position acquisition unit that acquires a position of a light emitter image in a frame image.

(Item 2) Item 2. The item 1, wherein the logical operation unit generates, for each pixel, a logical operation image by performing an AND operation on a pixel value of the difference image and a pixel value of the binarized image. Image processing apparatus.

(Item 3)

Item 1 or 2 wherein the extraction unit recognizes a two-dimensional pattern formed by a plurality of light emitters in advance and performs a light emitter image extraction process using the two-dimensional pattern. An image processing apparatus according to 1.

(Item 4)

The position acquisition unit processes the position of the illuminant image acquired this time as an error when the amount of change between the position of the previously acquired illuminant image and the position of the illuminant image acquired this time exceeds a predetermined value. The image processing device according to any one of items 1 to 3.

(Item 5)

The position acquisition unit counts the number of consecutive errors, and when the count value reaches a predetermined value, processes the position of the illuminator image acquired at that time as normal. Item 5. The image processing device according to Item 4,

(2) Method for Acquiring Position Information and / or Posture Information Using Two-Dimensional Code In the method for acquiring position information and / or posture information of controller 20 described below, in game system 1 shown in FIG. The controller 20 is configured to have a two-dimensional code instead of the LED 40. This two-dimensional code is picked up by the image pickup device 2 and used to acquire position information and posture information in the real space of the controller 20 in the game device 10.

[0085] FIG. 15 shows a controller 20 having a two-dimensional code. The controller 20 has a two-dimensional code 70 for uniquely identifying itself on the rear surface 29 of the housing. The two-dimensional code 70 and its identification method are the same as the two-dimensional code and its identification method disclosed in, for example, Japanese Patent Application Laid-Open No. 2 0 6-7 7 6 6 7 filed by the present applicant. It may be. FIG. 16 shows an example of a two-dimensional pattern constituting the two-dimensional code 70. The two-dimensional code 70 may be stamped on the housing rear surface 29 of the controller 20 or may be affixed as a printed matter. In the two-dimensional code 70, a direction indicating section 1 1 and an identification section 12 are formed. The direction instructing unit 11 is provided to indicate the upper part of the two-dimensional code 70, and the identification unit 12 is provided to express a characteristic part for uniquely identifying the two-dimensional code 70. The identification unit 12 is code information created with a plurality of blocks in a predetermined section. Among a plurality of blocks, a square block is commonly given to a plurality of two-dimensional codes 70, and a characteristic part is formed by blocks other than the four corners. The square block is used to measure the distance from the imaging device 2.

Returning to FIG. 5, when the image processing unit 100 recognizes the two-dimensional code 70 in the frame image, the image processing unit 102 based on the length between the four blocks in the two-dimensional code 70 image. The distance between the imaging device 2 and the two-dimensional code 70 is calculated and obtained. Further, the image processing unit 102 detects the direction indicating unit 11 of the two-dimensional code 70 and obtains the direction in which the two-dimensional code 70 is facing in the frame image. In this case, the direction in which the direction instructing unit 11 exists is higher than the identifying unit 12. Furthermore, the image processing unit 102 acquires the identification information of the two-dimensional code 70 from the block other than the four corners of the identification unit 12.

[0088] The image processing unit 102 has a function of recognizing through image analysis even when the two-dimensional code 70 is inclined in real space. The result of the image analysis performed by the image processing unit 10 2 is sent to the position estimation unit 10 4. The position estimation unit 10 4 estimates the position and orientation of the two-dimensional code 70 based on the image analysis result in the image processing unit 10 2.

FIG. 17 shows the configuration of the image processing unit 1 0 2. The image processing unit 10 2 includes a frame image acquisition unit 1 3 0, an actual object extraction unit 1 5 0, a state determination unit 1 5 2, an identification information acquisition unit 1 5 4, an identification information storage unit 1 5 6 and a transmission Part 1 6 0 is provided. The identification information storage unit 1 56 stores information on the characteristic part for identifying the real object and identification information for identifying the real object in association with each other. Ingredients Specifically, the pattern information of the identification unit 12 of the two-dimensional code 70 in FIG. 16 and the identification information are stored in a one-to-one correspondence. For example, the identification information may be used for character assignment in the gaming apparatus 10. In particular, in a game application that permits the existence of a plurality of two-dimensional codes 70, the two-dimensional codes 70 can be recognized by associating each of the two-dimensional codes 70 with identification information. The state determination unit 1 5 2 determines the state of the actual object in the set coordinate system. Specifically, the state determination unit 1 6 2 determines the posture, and the position determination unit 1 6 4 determines the position. Have

The frame image acquisition unit 1 3 0 acquires a real space frame image captured by the imaging device 2. The imaging device 2 periodically acquires frame images, and preferably acquires frame images at intervals of 130 seconds.

[0091] The real object extraction unit 15 0 extracts the real object image, that is, the image of the two-dimensional code 70 from the frame image. This process is performed by converting the image information into a binary bit representation and extracting an image of the two-dimensional code 70 from the bit representation. This is so-called dot processing, and image extraction processing may be performed by detecting bit on and off. This process may be performed by a known image matching technique. In that case, the real object extraction unit 150 registers the image information of the real object to be used in advance in a memory (not shown) for the matching process. By matching the pre-registered image information with the captured image information, the image of the two-dimensional code 70 can be cut out from the frame image.

The position determination unit 1 64 determines the two-dimensional position in the frame image of the actual object image. Specifically, the position in the frame image is determined based on the coordinates of the center point of the real object image in the frame image. Further, the position determination unit 16 4 determines the distance between the imaging device 2 and the two-dimensional code 70 based on the length between the four corners of the identification unit 12 in the image of the two-dimensional code 70. As a result, the position of the controller 20 in the three-dimensional real space can be acquired together with the two-dimensional position in the frame image. [0093] The posture determination unit 1 62 determines the posture of the actual object image. For example, the posture of the actual object image is determined by the position of the direction indicating unit 11 with respect to the identifying unit 12. In addition, when the controller 20 is tilted, the distances between the four corners of the identification unit 12 are not equal, so that the tilted posture can be determined based on the amount of change.

The identification information acquisition unit 1 5 4 extracts a feature portion from the actual object image and acquires corresponding identification information from the identification information storage unit 1 56. The game system 1 according to the present embodiment can cope with a plurality of different two-dimensional codes 70. For example, when two different two-dimensional codes 70 are provided for a plurality of controllers 20 respectively, Different identification information is associated with each dimension code 70.

[0095] The position information, posture information, and identification information acquired by the identification information acquisition unit 1 5 4 determined by the state determination unit 1 52 are associated with each other and sent from the transmission unit 1 60 to the position estimation unit. 1 0 Sent to 4. When a plurality of two-dimensional codes 70 are included in the frame image, position information, posture information, and identification information are associated with each two-dimensional code 70 to estimate the position from the sending unit 160. Sent to part 1 0 4. As described above, the position estimation unit 10 4 can acquire the position information, posture information, and identification information of the controller 20, and can estimate the position and posture of the controller 20.

[0096] As described above, the process of associating the identification information with the position information and the posture information can be executed even when the position information and the posture information of the controller 20 are acquired using the LED 40. it can.

[0097] (3) Acquisition method of position information or posture information using acceleration sensor etc.

In addition to the internal configuration of the controller shown in FIG. 4, the controller 200 may further include a triaxial acceleration sensor and a gyro sensor.

FIG. 18 illustrates an internal configuration of the controller 200 according to the embodiment. In FIG. 18, the configuration of the input reception unit 5 2, the PWM control unit 5 4, the drive unit 5 6, etc. shown in FIG. The illustration of the composition is omitted. The controller 200 includes a three-axis acceleration sensor 3 0 0 that detects acceleration components in the three axes directions of XYZ, and a gyro sensor 3 0 2 that detects an angular velocity in the housing horizontal plane (XZ plane). Here, the longitudinal direction of the case is set as the X axis, the height direction is set as the Y axis, and the short direction (depth direction) is set as the Z axis. The gyro sensor 30 2 is disposed in the central region in the housing, for example, in the vicinity of the lower portion of the LED button 31 in the top view of the housing shown in FIG. Similarly, the three-axis acceleration sensor 300 is preferably arranged near the center of the housing. Detection values by the three-axis acceleration sensor 300 and the gyro sensor 30 2 are supplied from the main control unit 50 to the wireless communication module 58 at a predetermined cycle. The wireless communication module 58 uses the detection values of the 3-axis acceleration sensor 30 0 and the gyro sensor 3 0 2 as well as operation inputs such as the direction keys 2 1 and the operation buttons 2 6 and the wireless communication of the game apparatus 10 at a predetermined cycle. Sent to communication module 1 0 8

FIG. 19 shows a configuration of the game apparatus 10 according to the embodiment. The game apparatus 10 includes a wireless communication module 10 8, a three-axis acceleration component measured by the three-axis acceleration sensor 3 0 0 and the gai mouth sensor 3 0 2 in the controller 2 0 0, and a horizontal plane of the housing. Get the angular velocity component around the direction (Y-axis direction) perpendicular to (XZ plane). The triaxial acceleration component and the angular velocity component are supplied from the wireless communication module 108 to the position estimation unit 250. The position estimation unit 2500 can estimate the position and orientation of the controller 2100 based on the acquired 3-axis acceleration component and angular velocity component. Specifically, the position estimating unit 2500 can acquire the attitude of the controller 2100 based on the 3-axis acceleration component. Also, the moving speed can be obtained by integrating the 3-axis acceleration component once, and the position can be obtained by integrating twice.

[0099] In the controller 20 00 of the embodiment, the angular velocity component around the Y-axis direction is detected by the gyro sensor 30 02. Therefore, the position estimation unit 2500 may estimate the rotation state, that is, the rotation posture of the controller 2100 on the horizontal plane based on the detection value by the gyro sensor 30.2. The 3-axis acceleration sensor If the servo 300 can acquire the triaxial acceleration component with high accuracy, the rotation state of the controller 200 in the horizontal plane may be estimated from the triaxial acceleration component.

[0100] As described above, the controller 2 0 0 includes the triaxial acceleration sensor 3 0 0 and the gyro sensor 3 0 2, so that the controller 2 0 0 does not need to analyze the frame image by the imaging device 2. The position and posture can be acquired. Note that the accuracy of estimation of the position and orientation of the controller 20 may be further increased by further using the analysis result of the frame image obtained by the imaging device 2. Since the position and orientation of the controller 2 0 0 can be estimated by using the measured values of the 3-axis acceleration sensor 3 0 0 etc., the position and orientation information of the controller 2 0 0 can be reflected in the processing of the game application. Possible

[0101] In the following, application examples that can be executed in the game system according to the embodiment are shown on the premise that the positions and postures of the controller 20 to the controller 20 0 can be acquired. Hereinafter, the controller 200 will be described on behalf of the controller. The game device 10 can estimate the posture and position of the controller 20 from the captured image even for the force controller 20 that estimates the posture and position of the controller 20 based on the three-axis acceleration component and the angular velocity component. The following contents can also be realized for the controller 20.

FIG. 20 shows the configuration of the application processing unit 1 1 0. The application processing unit 110 includes an input receiving unit 4 0 0, an acquisition unit 4 0 2, a change detection unit 4 0 4, an object control unit 4 0 6, and an image signal generation unit 4 0 8. FIG. 20 shows only a configuration for generating an image related to an object. Actually, the background image of the object is generated by another configuration, and the audio signal is also generated by another configuration.

The input receiving unit 4 0 0 receives an operation input input from the wireless communication module 1 0 8 via the game controller 2 0 0. The acquisition unit 4 0 2 is a position estimation unit. From 2 50, position information or attitude information of game controller 2 200 in real space is acquired. The change detection unit 40 4 detects a change in the position or posture of the game controller 2 0 0 from the position information or posture information acquired continuously in time by the acquisition unit 40 2. The change detection unit 40 4 can acquire the moving speed and moving direction of the controller 20 0 by detecting a change in position, and can detect the rotational speed and speed of the controller 2 0 0 by detecting a change in posture. The direction of rotation can be acquired.

[0104] As a basic function, the object control unit 40 06 controls the operation of the object in the virtual space based on the operation input received by the input reception unit 400. The object control unit 40 6 of this embodiment further has a function of controlling the movement of the object using the position information or the posture information acquired by the acquisition unit 4 02. Further, the object control unit 40 6 has a function of controlling the movement of the object using the change in position or posture acquired by the change detection unit 40 4.

[0105] For example, the object control unit 40 6 may determine the posture of the object in the virtual space using the position information or the posture information. In addition, the object control unit 40 6 controls the operation of the object 利用 using the position information or the posture information when the change detection unit 40 4 detects a predetermined change in the position or posture. It may start or end.

[0106] Acquisition unit 4 0 2 force When acquiring the three-dimensional position information or posture information of the game controller 2 0 0 in real space, the object control unit 4 0 6 obtains the three-dimensional position information or posture information. It may be used to control the movement of objects in a virtual 3D space. If the motion of the controller 200 is large and the object displayed on the image display device 3 is likely to be framed from the screen, the object control unit 40 6 may execute the notification process for the user. Good. The object whose operation is controlled by the object control unit 40 6 is converted into an image signal by the image signal generation unit 40 8 together with other background images. [0107] Application examples are shown below.

Tennis game>

In a tennis game, the height of the controller 20 0 in the frame image is the hitting height of the tennis ball, the direction of the controller 2 0 0 is the ball hitting direction, and the speed at which the controller 2 0 is moved is the strength of hitting the ball. The game can be advanced by moving the controller 200. Since these movements approximate the movements of swinging an actual racket, it is possible to give the user a feeling close to playing tennis. For example, controller controller 200 may be operated in the same manner as a racket, and the angle and direction of the swing at this time may be used.

[0108] The object control unit 4 0 6 acquires the height of the controller 2 0 0 from the position information of the controller 2 0 0 acquired by the acquisition unit 4 0 2 and acquires the orientation of the controller 2 0 0 from the posture information To do. In addition, the object control unit 40 6 obtains the moving speed of the controller 20 00 from the change detection unit 40 04. Based on these pieces of information, the object control unit 40 6 determines the posture of the object such as a racket or a character and operates it.

[0109] The object control unit 40 06 may control the operation of each virtual object based on the position information and the posture information of the controller 20 00 acquired by the acquisition unit 4 02. However, for example, a difference from the reference position and the reference posture may be taken, and the motion of each virtual object may be controlled based on the difference value. For example, the object control unit 4 06 uses the position and orientation of the controller 2 00 when the user operates the buttons of the controller 2 0 0 as the reference position and reference orientation used for the object motion control process. You may get it. The object control unit 40 6 detects a change amount (difference) between the reference position and the reference posture from the position information and the posture information acquired by the acquisition unit 40 2, and based on the change amount. The motion control of the virtual object is executed. The object control unit 40 6 notifies the reference position and reference posture to the change detection unit 40 4, and the change detection unit 40 4 detects the amount of change from the reference position and reference posture. May be. In the following, an example is shown in which the motion control of an object is performed based on the amount of change from the reference position and reference posture, but the object motion control based on the position and posture in real space coordinates can be executed in the same way.

[0110] By making it possible to use the buttons and key operations of the controller 200 as game inputs as they are, it is possible to make a game input that is sensibly innovative while maintaining the conventional game operation inputs.

For example, the strength of hitting the ball may be determined by a button operation of the controller 200. Combining conventional button operations with innovative game input, new gameplay can be realized. For example, tennis that can hit a strong serve by moving the controller 2 0 0 by moving the controller 2 0 0 while pressing the button while specifying the position to drop the ball on the opponent's court in the direction of the controller 2 0 0 It is also possible to make a game.

[0111] Using the position information and / or posture information of the controller 200, it is also possible to shoot shots such as a pole, a smash, a rob, and a drop shot. For example, the object control unit 40 6 causes the character to move in a positive direction when the amount of change in the height direction from the reference position of the controller 200 is small, and when the amount of change is large in the positive direction (upward). You can move the smash and move the lob when it is negative (downward).

[0112] Fig. 21 (a) shows a state where the character performs a pole, and Fig. 21 (b) shows a state where the character smashes. In the example of Fig. 21, two virtual objects, a game character and a racket, are represented. However, by reflecting the position information of the controller 200 not only on the character's posture but also on the inclination of the racket. It is possible to give the user a sense of actually playing tennis.

[01 1 3] <Motorcycle game>

In a motorcycle drive game, it may be used as a game input for controlling the buy-by by operating the posture of the controller 200. At this time, the speed may be determined by the speed of moving the controller 200, Alternatively, it may be determined by a button input. In this way, conventional button input, static input based on the position and orientation of the controller 200, and dynamic input based on state changes such as the moving speed of the controller 200 are performed. By using in combination, the operability of the game application can be improved.

[01 14] <Character input game>

In a game where characters are entered on the screen, characters can be entered on the screen by moving the position of the controller 2 0 0. The object control unit 40 6 is a diagram object that is output to the screen of the image display device 3 so as to follow the two-dimensional positional information in the X-axis and Y-axis directions supplied from the acquisition unit 40 2. Is generated. In the controller 200, the longitudinal direction of the casing is set as the X axis, the height direction is set as the Y axis, and the short side direction (depth direction) is set as the Z axis. It is called the case Y-axis and case Z-axis. In the game application, in the screen of the image display device 3, the horizontal direction is set as the X axis, the vertical direction is set as the Y axis, and the depth direction is set as the Z axis. , Screen Y-axis, Screen Z-axis.

[01 15] This allows the movement in the housing XY plane to be traced on the screen XY plane. When this process is performed only by the movement of the controller 200, the start and end of the trace process may be determined by the movement of the controller 200. At this time, the change detection unit 40 4 detects a predetermined change in the position information or the posture information and notifies the object control unit 40 06, and the object control unit 40 06 receives the notification and performs the trace processing. Determine start and end. The movement changes that trigger start and end may be the same, or may be set for each start process and end process. If they are the same, the movement change is processed as a trigger for the start process if the trace process is not being executed, and as a trigger for the end process if the trace process is being executed.

[01 16] Object control unit 4 0 6 Force When using two-dimensional movement in the XY plane as a game input, the movement in the other direction, that is, the movement in the Z-axis direction, It may be used as a start or end trigger. Enclosure Since the movement in the z-axis direction is not used for motion control of diagram objects, it can be used favorably as a start or end trigger. That is, by using information other than the components necessary for the movement of the object during the game as instruction information for the game, the game application can be executed without using the key operation. For example, in a character input game, the rotation operation of the controller 2 00 is not used as a game input. Therefore, an instruction to start or end the trace process is input by rotating the controller 2 0 0, that is, by changing the posture. It is good.

[01 17] <Electrode bar game>

The electrode bar game is a game in which an electrode bar is placed in a metal course frame and the electrode bar is carried to the goal within the time limit so that it does not hit the course frame or obstacles. It can be executed as a game application. Currently, it exists for 2D electrode bar games, but not for 3D electrode bar games.

[01 18] The object control unit 4 06 uses the 3D position information and orientation information of the controller 2 00 to control the movement of the electrode rod, which is a virtual object, in the virtual 3D space. Also good. In other words, by providing the course frame 厶 also in the screen depth direction (screen Z-axis direction) in the game screen, the user can not only place the controller 200 on the case XY plane but also in the case Z-axis direction. It will move and give you the same experience as a real electrode stick game.

[01 19] While various application examples have been described above, when the object controller 4 0 6 causes the movement of the controller 2 0 0 to be traced on the screen, if the operation amount of the controller 2 0 0 increases, It may also occur when the controller 200 is off the screen of the image display device 3. For example, such a situation can occur in the character input game described above.

[0120] When the virtual object corresponding to the controller 2 0 0 is likely to be out of frame from the screen of the image display device 3, the object control unit 4 0 6 uses the position information of the controller 2 0 0 as a basis. It seems to frame out to the user Notification processing for notifying may be performed. For example, when the controller 2 00 is likely to flare, a vibration control signal for vibrating the vibrator to the controller 2 0 0 is generated and transmitted from the wireless communication module 1 0 8. May be. Also, if a virtual object that follows the movement of the controller 200 is displayed as a game image, moving the virtual object so that it stands out to the user can tell the user that it is likely to frame. Good. For example, the user may be notified by performing a special movement that is repelled inward from the frame of the screen.

[0121] Whether or not the frame is about to be out may be determined based on whether or not the position in the vicinity of the frame is obtained when the position information of the controller 200 is converted into the screen coordinates. At this time, the change detection unit 40 4 may detect the moving direction and speed of the controller 20 0 and predict whether or not to frame out within a predetermined time from the information. This prediction process may be executed on the assumption that it is in the vicinity of the frame.

[0122] When the imaging device 2 is used, it may be determined at the position of the controller 20 0 in the frame image, for example, and the controller 20 may exist near the edge of the frame image. If detected, this notification process may be performed.

Further, as described in the present embodiment, various applications can be executed based on the position information and the posture information of the controller 20 0 or the controller 20.

For example, by rotating the controller together with the input button, you can make the game object make a big soap bubble. In addition, the game object can be kneaded with the cooking game by rotating the controller or reciprocating the posture. In addition, using the position information of the controller, the game object can be operated and commanded by the tambourine. In addition, by using the controller's 3D position information, it is possible to apply a sphere by hitting a 3D block. You can also run a game that breaks down. In addition, the player can be switched by passing the controller 20 0 so as to catch a pole between players. If there is a pseudo camera in the game, the viewpoint of the pseudo force mela can be moved based on the position information of the controller. In addition, based on the rotation information of the controller, it is also possible to perform an action of turning a virtual door knob. In addition, by actually moving the controller, the virtual object can be moved following the movement. In addition, by capturing the three-dimensional movement of the controller, the movement is captured as a virtual flea movement, creating a sculpture, and by capturing the two-dimensional movement, the movement is captured by the virtual brush. You can create a picture by capturing it as a movement. Furthermore, by applying an instantaneous impact to the controller, it is possible to control the progress of the game, such as vibrating the controller when the impact level exceeds a predetermined value or reflecting the impact on the game object. Good. If you have an acceleration sensor, etc., you can detect the impact with that acceleration, and if you use an LED image, if the LED image becomes elliptical or is not captured instantaneously, the controller It may be determined that has been impacted.

Industrial applicability

The present invention can be used for a game device that executes a new application using new input information.

Claims

The scope of the claims

[1] An input receiving unit that receives an operation input input from a game controller, and an object control unit that controls the movement of an object in a virtual space based on the operation input received in the input receiving unit. A game device,

An acquisition unit for acquiring position information or posture information of the game controller in real space;

The object control unit uses the position information or the posture information.

A game device that controls the movement of the object.

[2] The object control unit uses the position information or the posture information.

The object posture in the virtual space is determined.

The game device according to 1.

[3] The apparatus may further comprise a change detection unit that detects a change in the position or posture of the game controller from the position information or the posture information acquired continuously in time by the acquisition unit. The game device according to 1 or 2.

[4] When the change detection unit detects a predetermined change in position or orientation, the object control unit starts or ends the motion control of the object using the position information or the orientation information. The game device according to claim 3.

[5] The acquisition unit acquires the three-dimensional position information or posture information of the game controller in real space,

The object control unit uses three-dimensional position information or posture information.

2. The game apparatus according to claim 1, wherein the game apparatus controls an operation of the object in a virtual three-dimensional space.

6. The game device according to claim 1, wherein the object control unit performs a notification process for a user when an object displayed on the image display device is likely to frame from the screen. .