JP5089078B2 - Game device - Google Patents

Description

  The present invention relates to a game device that executes a game application.

  In recent years, a technique has been proposed in which a two-dimensional code is imaged by a camera, code data is recognized, and a predetermined process associated with the code data is executed. Compared with a one-dimensional barcode, a two-dimensional code has a larger amount of information that can be coded, and various types of two-dimensional codes are now in practical use. Under such circumstances, there has conventionally been proposed a technique related to image recognition of a two-dimensional code (see, for example, Patent Document 1).

There are various measurement systems that use a magnetic method, an ultrasonic method, or an optical method in order to acquire the three-dimensional position and orientation of an object in real space. Magnetic measurement systems are easily affected by electronic devices and geomagnetism existing in the environment, and ultrasonic measurement systems are easily affected by atmospheric pressure and temperature. On the other hand, an optical measurement system uses a captured image captured by a camera, and thus is not affected by a surrounding magnetic field, atmospheric pressure, temperature, or the like. As a measurement system using an optical method, a plurality of LEDs provided on an object blink with a uniquely defined blink pattern, and a three-dimensional position and orientation are recognized using a photographed image by a two-dimensional image sensor. Has been proposed (see, for example, Patent Document 2). In Patent Document 2, an LED that functions as a marker has identification information that can be uniquely identified for each LED.
JP 2006-72667 A JP 2003-254716 A

  With recent technological advances, the hardware specifications of game systems have improved dramatically. Recently, a game system has been realized in which a game device optically recognizes a user's movement and uses it as an input to a game when the user moves the body in front of the camera by connecting the camera and the game device. Under such a game environment, user demands for game applications tend to diversify.

  In order to satisfy the diversified demands of users, the present inventor has conventionally used a camera connected to a game device to extract information of a captured image obtained by photographing a game controller as input information of the game device. Found the possibility of realizing no game application. Also, it has been found that a game application that is not available in the past can be realized by mounting an acceleration sensor or the like on the game controller and using the measurement information as input information of the game device. Moreover, it is considered that an effective application can be realized not only in a game system but also in other environments such as entertainment systems such as educational sites and business sites.

  Accordingly, an object of the present invention is to provide a game device that executes a new application using new input information.

  In order to solve the above problems, a game device according to an aspect of the present invention includes an input receiving unit that receives an operation input input from a game controller, and an object in a virtual space based on the operation input received by the input receiving unit. An object control unit for controlling the operation of The game apparatus further includes an acquisition unit that acquires position information or posture information in the real space of the game controller, and the object control unit controls the movement of the object using the position information or the posture information.

  According to the present invention, it is possible to provide a game device that executes a new application using new input information.

  An embodiment of the present invention provides a game device that can acquire position information and / or posture information in a real space of a game controller and execute a game application that can be executed based on the information. Therefore, in this specification, first, a technique for acquiring the position information and / or posture information of the game controller is shown. In the following, (1) a method using an LED, (2) a method using a two-dimensional code, (3) a method using a measurement value of an acceleration sensor, etc., followed by position information of the game controller and / or A game application that can be executed based on the posture information is shown.

(1) Method for Acquiring Position Information and / or Posture Information Using LED FIG. 1 shows a use environment of a game system according to an embodiment of the present invention. The game system 1 includes an imaging device 2, an image display device 3, an audio output device 4, a game device 10, and a controller 20. The imaging device 2, the image display device 3, the audio output device 4, and the controller 20 are connected to the game device 10.

  The controller 20 is an operation input device for a user to perform an operation input, and the game apparatus 10 processes a game application based on the operation input in the controller 20 and outputs an image signal indicating a processing result of the game application. A processing device to be generated. Note that the technology shown in the present embodiment can be realized not only in a game application but also in an entertainment system including a processing device that executes another type of application. Below, the game system 1 which runs a game application on behalf of an entertainment system is demonstrated.

  The imaging device 2 is a video camera that includes a CCD imaging device, a CMOS imaging device, or the like. The imaging device 2 captures a real space at a predetermined cycle and generates a frame image for each cycle. For example, the imaging speed of the imaging device 2 may be set to 30 frames / second so as to match the frame rate of the image display device 3. The imaging device 2 is connected to the game device 10 via a USB (Universal Serial Bus) or other interface.

  The image display device 3 is a display that outputs an image signal. The image display device 3 receives the image signal generated by the game device 10 and displays a game screen. The sound output device 4 is a speaker that outputs sound, and receives a sound signal generated in the game device 10 and outputs a game sound. The image display device 3 and the audio output device 4 constitute an output device in the game system 1.

  The game apparatus 10 and the image display apparatus 3 may be connected by wire or may be connected wirelessly. The game apparatus 10 and the image display apparatus 3 may be connected by an AV cable. A home network constructed by a network (LAN) cable, a wireless LAN, or the like may be constructed between the game apparatus 10 and the image display apparatus 3.

  The controller 20 has a function of transmitting an operation input by the user to the game apparatus 10, and is configured as a wireless controller capable of wireless communication with the game apparatus 10 in this embodiment. The controller 20 and the game apparatus 10 may establish a wireless connection using a Bluetooth (registered trademark) protocol. The game apparatus 10 enables wireless connection with a plurality of controllers 20, that is, in the game system 1, a 1-to-N connection between the game apparatus 10 and the controller 20 can be realized. The game device 10 functions as a parent device, that is, a master, and the controller 20 functions as a child device, that is, a slave. The controller 20 is not limited to a wireless controller, and may be a wired controller connected to the game apparatus 10 via a cable.

  The controller 20 is driven by a battery (not shown) and is configured to have a plurality of buttons and keys for performing an operation input for advancing the game. When the user operates a button or key of the controller 20, the operation input is transmitted to the game apparatus 10 by radio. The game apparatus 10 receives an operation input related to the game application from the controller 20, controls the game progress according to the operation input, and generates a game image signal and a game sound signal. The generated game image signal and game sound signal are output by the image display device 3 and the sound output device 4, respectively. The game apparatus 10 also has a function of transmitting a vibration control signal for vibrating the controller 20 to the controller 20 in accordance with the progress status of the game application. The controller 20 has a vibrator and vibrates the vibrator when receiving a vibration control signal.

  In the game system 1 of the present embodiment, the controller 20 includes a plurality of light emitting elements. The plurality of light emitting elements are LEDs of the same color, and serve as indicators that represent controller numbers set in the game application. Since the controller number in the game application is used by the user when selecting a game character at the start of the game, for example, it is necessary to notify the user of the controller number by some means. Therefore, in the controller 20, for example, if the first LED is lit, the controller number is 1, and if the second LED is lit, the controller number is 2 to the user. Inform your number. Note that the controller number can be expressed by a combination of a plurality of LEDs.

  When the controller 20 is a wired controller, the controller number can be determined at the position where the cable connector extending from the controller 20 is inserted into the port of the game apparatus 10. However, when a multi-port device having a plurality of ports is used externally attached to the game apparatus 10, the user cannot know the controller number immediately, so the controller is provided to the user using a plurality of LEDs. It is preferable to notify the number.

  In the game system 1, the LED of the controller 20 is used not only as an indicator of the controller number but also as a game input that affects the progress of the game application. In this case, the LED control is switched to lighting as an input to the game application instead of lighting as an indicator. The imaging device 2 images the LED of the controller 20, generates a frame image, and supplies it to the game device 10. The game apparatus 10 acquires the frame image, estimates the position and posture in the frame image of the controller 20 from the position of the LED image in the frame image, and acquires and acquires the position information and posture information in the real space. The position information and / or posture information is reflected in the processing of the game application. That is, the game apparatus 10 according to the present embodiment processes the game application using not only the operation input input by the controller 20 but also the acquired position information and posture information of the controller 20, and an image signal indicating the processing result. It has a function to generate.

  FIG. 2 shows an external configuration of the controller. The controller 20 is provided with a direction key 21, an analog stick 27, and four types of operation buttons 26. The direction key 21, the analog stick 27, and the operation buttons 26 are input units provided on the upper surface 30 of the housing. The four types of buttons 22 to 25 are marked with different figures in different colors in order to distinguish them from each other. In other words, the ○ button 22 is marked with a red circle, the X button 23 is marked with a blue cross, the □ button 24 is marked with a purple square, and the Δ button 25 is marked with a green triangle. A plurality of LEDs are provided on the housing rear surface 29 of the controller 20.

  The user operates the controller 20 by holding the left holding part 28a with the left hand and holding the right holding part 28b with the right hand. The direction key 21, the analog stick 27, and the operation button 26 are provided on the upper surface 30 of the housing so that the user can operate the gripper while holding the left grip portion 28a and the right grip portion 28b.

  A button 31 with LED is also provided on the upper surface 30 of the housing. The button 31 with LED is used as a button for causing the image display device 3 to display a menu screen, for example. Further, it also has a function of notifying the user of incoming mail or the state of charge of the battery of the controller 20 depending on the light emission state of the LED. For example, the LED is turned on so that the color is red during charging, green when charging is completed, and red when the remaining charge is low.

  Since the user plays a game while looking at the game screen displayed on the image display device 3, the image display device 3 exists in front of the controller 20 indicated by the arrow A. For this reason, the housing rear surface 29 on which the LEDs are arranged normally faces the image display device 3. In the present embodiment, since the imaging device 2 needs to capture an LED during execution of the game application, the imaging device 2 is arranged so that the imaging range faces the same direction as the image display device 3, and the housing rear surface 29 of the controller 20. It is preferable to face each other. In general, since a user often plays a game in front of the image display device 3, the imaging device 2 is arranged so that the direction of the optical axis thereof coincides with the front direction of the image display device 3. Specifically, the imaging device 2 is preferably arranged in the vicinity of the image display device 3 so that the imaging range includes a position where the user can visually recognize the display screen of the image display device 3. Thereby, the imaging device 2 can image the user's body and the controller 20.

  FIG. 3 shows an external configuration of the back side of the controller. In FIG. 3, illustrations of the direction key 21 and the operation button 26 on the upper surface 30 of the housing of the controller 20 are omitted. An LED arrangement area 42 is provided on the housing rear surface 29, and the first LED 40 a, the second LED 40 b, the third LED 40 c, and the fourth LED 40 d are arranged in the LED arrangement area 42. Hereinafter, the first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d are collectively referred to as “LED 40”. The LED arrangement region 42 is formed in the central region of the housing back surface 29. A USB connector 46 is provided at the center of the LED arrangement area 42. A USB cable extending from the game apparatus 10 is connected to the USB connector 46 so that the controller 20 can be charged. If a USB cable is connected, the controller 20 can be used as a wired controller.

  Operation buttons 48 a, 48 b, 48 c, 48 d are provided on the left and right sides of the housing rear surface 29 with the LED arrangement area 42 interposed therebetween. The operation buttons 48a, 48b, 48c, and 48d are provided at positions that can be operated with the tip of the index finger even when the user holds the left grip 28a and the right grip 28b. Thus, the LED 40 is not hidden by the index finger when operating the operation buttons 48a, 48b, 48c, and 48d.

  The imaging device 2 acquires RGB luminance values in each pixel. In order to accurately detect the emitted LED 40, it is preferable that the contrast between the LED 40 and the area around the LED 40 is large. Therefore, the LED arrangement region 42 has a color with a lightness lower than that of the adjacent housing color, for example, a black color scheme. The LED arrangement region 42 is configured as a black translucent plate, and the LED 40 is provided behind it, that is, inside the housing. Thereby, if LED40 is not lighted, it will not be visible from the imaging device 2, but it will image only when it lights. The black translucent plate also has a role of diffusing light and changing the narrow directivity of the LED 40 to a wide directivity. By providing the LED 40 in the LED arrangement region 42, the contrast with the emitted LED 40 can be increased, and the LED image can be effectively extracted from the frame image in the subsequent image processing.

  The first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d are arranged in a predetermined two-dimensional pattern. For example, 1st LED40a, 2nd LED40b, 3rd LED40c, and 4th LED40d are arrange | positioned in the position which comprises the vertex of a rectangle. The game apparatus 10 grasps this positional relationship in advance and uses it for LED image extraction processing. A number is imprinted or printed in the vicinity of each LED 40, and the user can know the controller number by looking at the number in the vicinity of the lit LED 40.

  FIG. 4 shows the internal configuration of the controller. The controller 20 includes a wireless communication module 58, a processing unit 60, an LED 40, and a vibrator 44. The wireless communication module 58 has a function of transmitting / receiving data to / from the wireless communication module of the game apparatus 10. The processing unit 60 executes intended processing in the controller 20.

  The processing unit 60 includes a main control unit 50, an input receiving unit 52, a PWM control unit 54, and a driving unit 56. The main control unit 50 transmits and receives necessary data to and from the wireless communication module 58.

  The input receiving unit 52 receives input information from the input unit such as the direction key 21, the operation button 26, the analog stick 27, and sends it to the main control unit 50. The main control unit 50 supplies the received input information to the wireless communication module 58, and the wireless communication module 58 transmits the input information to the game apparatus 10 at a predetermined timing. When the wireless communication module 58 receives the vibration control signal from the game apparatus 10, the wireless communication module 58 supplies the vibration control signal to the main control unit 50, and the main control unit 50 operates the drive unit 56 that vibrates the vibrator 44 based on the vibration control signal. Let The drive unit 56 may be configured as a switch for driving the vibrator 44, or may be configured as a PWM control unit that varies the duty ratio of the supply voltage.

  In this embodiment, the PWM control unit 54 includes a first PWM control unit 54a, a second PWM control unit 54b, a third PWM control unit 54c, and a fourth PWM control unit 54d. The first PWM control unit 54a, the second PWM control unit 54b, the third PWM control unit 54c, and the fourth PWM control unit 54d are provided to control the lighting of the first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d, respectively. The PWM control unit 54 controls the voltage applied to the LED 40 by PWM (pulse width modulation). For example, the PWM control unit 54 can adjust the luminance of the LED 40 by performing PWM control of the applied voltage at a high frequency of several kHz. The PWM control unit 54 can recognize whether the LED 40 is turned on or off in the imaging device 2 by performing PWM control of the applied voltage at a low frequency of about several Hz to 100 Hz.

  In the game system 1 of the present embodiment, first, a controller 20 of a user who wishes to participate in the game system 1 establishes communication with the game apparatus 10. At this time, the identification information of the controller 20, for example, a Bluetooth address, is transferred to the game apparatus 10, and subsequent communication is performed based on the Bluetooth address. When Bluetooth is not used as a communication protocol, a device address such as a MAC address may be used. After the connection is established, the user can participate in the game application.

  At this time, information indicating a controller number is transmitted from the game apparatus 10 to the controller 20. The main control unit 50 turns on only the LED 40 corresponding to the set controller number based on the number instruction information. Thus, the user can know the controller number of his / her controller 20. Instead of the number instruction information, a control signal for instructing lighting of the LED 40 corresponding to the controller number may be transmitted from the game apparatus 10. As described above, the main control unit 50 can light any LED 40.

  In the game system 1 of the present embodiment, the game apparatus 10 estimates the position and orientation of the controller 20 in the real space based on the frame image captured with all the LEDs 40a to 40d turned on. The estimated position and orientation are used as input of the game application to be executed. The game apparatus 10 also accepts operation inputs such as the direction key 21 and the operation button 26 of the controller 20. Based on the input information, the game apparatus 10 generates game parameters of the game application, controls the progress of the game, and generates AV data of game images and game sounds. AV data is output from the image display device 3 and the audio output device 4. The user moves the controller 20 in the real space while looking at the game screen, and further plays the game by operating the direction keys 21 and the operation buttons 26. Hereinafter, processing in the game apparatus 10 will be described.

  FIG. 5 shows the configuration of the game apparatus. The game apparatus 10 includes an input unit 100, an image processing unit 102, a position estimation unit 104, a blinking instruction unit 106, a wireless communication module 108, an application processing unit 110, and an output unit 112. The processing function of the game apparatus 10 in the present embodiment is realized by a CPU, a memory, a program loaded in the memory, and the like, and here, a configuration realized by cooperation thereof is illustrated. The program may be built in the game apparatus 10 or may be supplied from the outside in a form stored in a recording medium. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof. In the illustrated example, the CPU of the game apparatus 10 functions as an image processing unit 102, a position estimation unit 104, a blinking instruction unit 106, and an application processing unit 110. Note that the game apparatus 10 may have a plurality of CPUs due to the hardware configuration. In such a case, one CPU functions as the application processing unit 110 that executes the game application, and the other CPU processes the image processing unit 102 that processes the captured image by the imaging device 2, the position estimation unit 104, and the blinking instruction unit 106. May function as

  The wireless communication module 108 establishes wireless communication with the wireless communication module 58 of the controller 20. In the synchronization establishment phase, the wireless communication module 108 performs a connection inquiry, that is, an “inquiry” process for terminal devices including the peripheral controller 20. Specifically, the wireless communication module 108 broadcasts an IQ (inquiry) packet to nearby terminal devices. The wireless communication module 58 of the controller 20 that has received the IQ packet returns an FHS (Frequency Hop Synchronization) packet including the Bluetooth address and Bluetooth clock information to the game apparatus 10. In the transmission / reception at this time, since the consent regarding the frequency hopping pattern has not been established between the game apparatus 10 and the controller 20, a fixed hopping pattern defined exclusively for inquiry is used.

  The wireless communication module 108 receives the FHS packet from the controller 20, grasps what kind of controller 20 exists, and then transmits an ID packet to the specific controller 20. This is a “calling” process by the wireless communication module 108. When a response to the ID packet is returned from the specific controller 20, the wireless communication module 108 transmits an FHS packet to the controller 20 to notify the controller 20 of its own address and clock. Thereby, the game apparatus 10 and the controller 20 can share the same hopping pattern.

  When “calling” is performed, a piconet is formed between the controller 20 and the game apparatus 10 to enter the “connected” state. A piconet means a network that is temporarily formed between terminals when the Bluetooth terminals are brought close to each other, and a maximum of eight Bluetooth terminals can participate in one piconet. The connected controller 20 is assigned a slave identifier from the wireless communication module 108, that is, a 3-bit address (1 to 7) given to the connected controller 20. This slave identifier is called AM_ADDR (active member address). In the “connected” state, a control packet for setting a communication link is transmitted and received, thereby enabling “data transfer”.

  In addition, although the example which calls the controller 20 from the game device 10 was demonstrated above, you may call the game device 10 from the controller 20. FIG. In this case, the controller 20 executes a calling process while the game apparatus 10 is executing a page scan. As a result, the controller 20 can call the game apparatus 10 and enter the connected state. Thereafter, the roles of the master and slave are switched, and the game apparatus 10 becomes the master and the controller 20 becomes the slave. By realizing such a master-slave relationship, the controller 20 can participate in the game application.

  When the connection state is established and the controller 20 participates in the game application, the application processing unit 110 assigns a controller number of the controller 20 and generates number instruction information indicating the controller number. The wireless communication module 108 transmits number instruction information to the controller 20. When the controller 20 receives the number instruction information, the controller 20 turns on the corresponding LED 40. Note that the wireless communication module 108 may transmit, to the controller 20, a control signal instructing lighting of the LED 40 corresponding to the controller number instead of the number instruction information.

  When the game application is started, the blinking instruction unit 106 generates a control signal that instructs the controller 20 to blink a pattern so as to blink at a predetermined cycle. This blinking control signal is transmitted from the wireless communication module 108 to the controller 20.

  The input unit 100 is configured as a USB interface, and acquires a frame image from the imaging device 2 at a predetermined imaging speed (for example, 30 frames / second). The image processing unit 102 extracts an LED image from the frame image according to an LED extraction algorithm described later. The position estimation unit 104 acquires the position information of the LED image from the extracted LED image, and estimates the position and orientation of the controller 20. Since the position estimation unit 104 acquires the position information of the LED image, the position estimation unit 104 may function as a position acquisition unit. Note that the position of the controller 20 may be determined at the position of the center of gravity of the four LED images.

  The position information and / or posture information of the controller 20 estimated from the positions and postures of the four LEDs 40 that are lit are used as input of the game application. Therefore, the position information and posture information of the controller 20 estimated by the position estimation unit 104 are sequentially sent to the application processing unit 110 and reflected in the application processing. An input by the user operating the direction key 21 or the like of the controller 20 is also transmitted to the application processing unit 110 via the wireless communication module 108. The application processing unit 110 advances the game from the position / posture information of the controller 20 and the operation input in the controller 20, and generates an image signal and an audio signal indicating the processing result of the game application. The image signal and the audio signal are sent from the output unit 112 to the image display device 3 and the audio output device 4, respectively, and output.

  FIG. 6 shows the configuration of the image processing unit. The image processing unit 102 includes a frame image acquisition unit 130, a current frame buffer 132, a previous frame buffer 134, a difference image generation unit 136, a binary image generation unit 138, a logical operation unit 140, and an LED image extraction unit 142.

  The frame image acquisition unit 130 acquires frame image data from the input unit 100 and temporarily stores it in the current frame buffer 132. The previous frame buffer 134 stores the previous frame image data. The difference image generation unit 136 generates a difference image between the current frame data stored in the current frame buffer 132 and the previous frame data stored in the previous frame buffer 134. The R pixel value of the previous frame data at coordinates (x, y) in the frame image is Pr, the G pixel value is Pg, the B pixel value is Pb, the R pixel value of the current frame data is Cr, the G pixel value is Cg, When the B pixel value is Cb, the difference image function F (x, y) is expressed by the following mathematical formula.

  Here, Th is a predetermined threshold value. If F (x, y) is greater than 0, the pixel value at coordinates (x, y) is encoded to “1”, and the pixel is white on display. On the other hand, if F (x, y) is 0 or less, the pixel value is encoded to “0”, and the pixel is black on display. Thus, the difference image generated by the difference image generation unit 136 is an image subjected to binarization processing. By generating the difference image, it is possible to remove the influence of an object that does not move between the previous frame image and the current frame image.

  The binarized image generation unit 138 binarizes the current frame data using a predetermined threshold value, and generates a binarized image. The binarized image generation unit 138 encodes the pixel value of the pixel holding the luminance greater than the predetermined threshold to “1”, and encodes the pixel value of the pixel holding the luminance equal to or lower than the predetermined threshold to “0”. To do. By generating a binarized image of the current frame image, only bright objects can be extracted from the current frame image.

  The logical operation unit 140 generates a logical operation image obtained by performing a logical operation on the difference image generated by the difference image generation unit 136 and the binarized image generated by the binarized image generation unit 138. Specifically, the logical operation unit 140 generates a logical product image by performing a logical product operation on the pixel value of the difference image and the pixel value of the binarized image for each pixel. This is generated by mathematically ANDing the pixel values of corresponding pixels of the difference image and the binarized image.

FIG. 7A shows the current frame image. In the current frame image, the user is holding the controller 20 and playing a game, and the right side is illuminated. At this time, all the LEDs 40 of the controller 20 are lit. All the LEDs 40 emit light with the same luminance by the same lighting pattern.
FIG. 7B shows a difference image between the current frame image and the previous frame image. When the controller 20 is moved from the position of the previous frame image, a difference image is generated by Expression (1). In this example, the controller 20 or at least a part of the body moving with the controller 20 is acquired as a white image (pixel value is “1”). Note that such a white image is not always acquired, and FIG. 7B is merely an example of a difference image generated when Expression (1) is satisfied. At this time, since the illumination does not move, it is not included in the difference image. If the user does not move the controller 20, the white image of the controller 20 is not acquired.

FIG. 8A shows a binarized image of the current frame image. In the binarized image, the high luminance region in FIG. 7A, that is, the light emitting LED and the illumination lamp are binarized and converted into a white image. In order to prevent other noises, it is preferable that the threshold value set in the binarization process is set to a luminance value obtained by subtracting a predetermined margin value from the light emission luminance of the LED.
FIG. 8B shows a logical product image. The logical product image performs a logical product operation on the difference image shown in FIG. 7B and the binarized image shown in FIG. In the difference image and the binarized image, the pixel value of the white pixel is “1”, and the pixel value of the black pixel is “0”. Therefore, only pixels that exist as white pixels in both images are generated as white pixels in the logical product image.

  Thus, by taking the logical product of the difference image and the binarized image, a logical product image from which an LED image can be easily extracted can be generated. The logical product image is sent to the LED image extraction unit 142.

  When the logical product image is generated, the previous frame data held in the previous frame buffer 134 is overwritten by the current frame data held in the current frame buffer 132. This overwriting process may be performed immediately after the previous frame buffer 134 is read by the difference image generation unit 136. Thereafter, the current frame data held in the current frame buffer 132 is overwritten by the next frame data, and the logical product image generation process in the next frame is executed.

  The LED image extraction unit 142 extracts an LED image from the logical product image. In order to extract an LED image, first, an LED image candidate is detected. Subsequently, an arbitrary candidate is assumed to be the first LED 40a, and its peripheral region is searched to check whether there is an LED image candidate that can be the second LED 40b. If the second LED 40b is found, the peripheral area of the first LED 40a is searched to check whether there is an LED image candidate that can become the third LED 40c. If the third LED 40c is found, the surrounding area is searched to check whether there is an LED image candidate that can be the fourth LED 40d. At this time, the LED image extraction unit 142 recognizes a two-dimensional pattern formed by the plurality of LEDs 40 in advance, and performs an LED image extraction process using the two-dimensional pattern. In the following, for convenience of explanation, the case where the first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d are arranged to form a square apex on the housing rear surface 29 of the controller 20 is taken as an example.

  The LED image extraction unit 142 can also extract the LED image from the binarized image without using the logical product image. The binarized image includes noise other than the LED image. For example, when the LED image cannot be acquired in the difference image, the binarized image is used to change the position and orientation of the LED image. It can be detected. The processing procedure shown below can be executed using either a logical product image or a binarized image. However, since the logical product image has less noise components other than the LED image, it is faster to use the logical product image. Is fast.

  FIG. 9 is a flowchart illustrating a processing procedure of LED image candidate detection processing. The LED image extraction unit 142 acquires, as one connected region, a region in which white pixels (pixels having a pixel value “1”) continuously exist in the logical product image (S100). Next, the LED image extraction part 142 extracts the edge part of a white pixel connection area | region, and determines a short side and a long side (S102).

  The LED image extraction unit 142 determines whether or not the short side is composed of pixels having a predetermined number of pixels Ma or more (S104). When the short side is composed of pixels smaller than the number of pixels Ma (N in S104), the LED image extraction unit 142 determines that the short image is not an LED image, and ends the processing of the white pixel connection region. When the short side is composed of pixels having the number of pixels Ma or more (Y in S104), the LED image extraction unit 142 determines whether or not the long side is composed of pixels having the predetermined number of pixels Mb or less (S106). ). When the long side is composed of pixels larger than the number of pixels Mb (N in S106), the LED image extraction unit 142 determines that it is not an LED image, and ends the processing of the white pixel connection region. When the long side is composed of pixels having the number of pixels Mb or less (Y in S106), the LED image extraction unit 142 obtains the number of pixels constituting the white pixel connection region as a candidate for the LED image (S108). ). This LED image candidate detection process is repeated for all white pixel connected regions. Thereby, LED image candidates included in the logical product image can be detected. Note that the coordinates of the LED image candidate are set at the center of gravity of the connected region.

  FIG. 10 is a flowchart illustrating the processing procedure of the LED extraction process. The LED image extraction unit 142 assumes that an arbitrary LED image candidate among the detected LED image candidates is the first LED 40a (S120). Next, it is determined whether or not there is an LED image candidate that satisfies the condition that can be the second LED 40b for the first LED 40a (S122). If there is an LED image candidate to be the second LED 40b (Y in S122), it is determined whether there is an LED image candidate that satisfies the conditions that can be the third LED 40c for the first LED 40a and the second LED 40b (S124). If there is an LED image candidate to be the third LED 40c (Y in S124), it is determined whether or not there is an LED image candidate that satisfies the condition that can be the fourth LED 40d for the first LED 40a, the second LED 40b, and the third LED 40c ( S126). If there is an LED image candidate to be the fourth LED 40d (Y in S126), the first LED 40a to the fourth LED 40d of the controller 20 are extracted. If there is no second LED image candidate (N in S122), no third LED image candidate (N in S124), or no fourth LED image candidate (N in S126), another LED image candidate. Is the first LED 40a, and the process of FIG. 8 is re-executed.

  In the LED extraction processing, in order to reduce the processing load, it may be set as a processing condition that the posture of the controller 20 is not inclined more than a predetermined angle. Since the LED 40 has relatively strong directivity, if the controller 20 is tilted too much, it is difficult to receive light in the imaging device 2. When searching for LED image candidates using the directivity of the LED 40, the processing load is reduced by performing the LED image extraction process on the condition that the LED 40 is not inclined at a predetermined angle (for example, 45 °) from the first LED 40a. Can be greatly reduced.

  In this embodiment, since the difference image between the previous frame image and the current frame image is used, the image of the LED 40 is excluded from the difference image when the controller 20 does not move. Therefore, when a difference image is used, an LED image is not always found by the LED extraction process shown in FIG. If no LED image is found by the LED extraction process shown in FIG. 10, the position estimation unit 104 estimates that the controller 20 has not moved from the position of the previous frame, and the position information and orientation information of the controller 20 in the previous frame. To get. At this time, the position estimation unit 104 may extract an LED image using the binarized image.

FIG. 11 is a flowchart showing the processing procedure of the second LED detection processing shown in S122 of FIG. The number of constituent pixels of the LED image candidate assumed to be the first LED 40a is N. First, the LED image extraction unit 142 determines whether the number of pixels of the LED image candidate is ½ × N or more and 2N or less (S140). This is based on the fact that the size and shape of an LED image are not significantly different from those of the first LED 40a. If the number of constituent pixels of the LED image candidate is within a predetermined range with respect to the number of constituent pixels N (Y in S140), it is determined whether or not the LED image candidate exists on the right side of the first LED 40a (S142). ). The determination as to whether or not it is right is determined by whether or not the first LED 40a is within a range of 45 ° from the upper right to 45 ° from the lower right. If the right side of the 1LED40a (S142 of Y), determines the distance _{D 12} between the first 1LED40a and LED image candidate, whether or less and be more than Da Db (S144). Here, when the distance between each other is too close or too far away, it is determined that the second LED 40b is not suitable. Distance If _{D 12} is less than and Db be more than Da (S144 of Y), it is determined that the LED image candidate may be the 2LED40b, the presence of the 2LED40b is confirmed (S146). On the other hand, if the case where the number of pixels of the LED image candidate is not within a predetermined range (S140 of N), is not present on the right side of the LED 40a (S142 of N), the distance _{D 12} is not within the predetermined range (S144 N), it is determined that the LED image candidate is not the second LED 40b.

FIG. 12 is a flowchart showing a processing procedure of the third LED detection processing shown in S124 of FIG. The number of constituent pixels of the LED image candidate assumed to be the first LED 40a is N. First, the LED image extraction unit 142 determines whether the number of pixels of the LED image candidate is ½ × N or more and 2N or less (S160). If the number of constituent pixels of the LED image candidate is within a predetermined range with respect to the number of constituent pixels N (Y in S160), it is determined whether or not the LED image candidate exists below the first LED 40a (S162). . The determination of whether or not it is below is determined by whether or not the first LED 40a exists within a range of 45 ° from the lower left to the lower right 45 °. If the bottom of the 1LED40a (S162 of Y), the distance between the first 1LED40a and LED image candidate, determines whether 1 / √2 × either be a _{D 12} or more and and √2D ₁₂ or less (S164 ). Here, when the distance between each other is too close or too far away, it is determined that the third LED 40c is not suitable. If distance is a to and √2D ₁₂ or less 1 / √2 × _{D 12} or more (S164 in Y), and the line segment from the 1LED40a until the 2LED40b, angle of the line segment from the 1LED40a until the LED image candidate Is determined to be within a predetermined range (S166). At this time, if it is within the predetermined range (Y in S166), it is determined that the LED image candidate can be the third LED 40c, and the presence of the third LED 40c is confirmed (S168). On the other hand, when the number of constituent pixels of the LED image candidate is not within the predetermined range (N of S160), when it is not present below the LED 40a (N of S162), when the distance from the first LED 40a is not within the predetermined range ( If the angle is not within the predetermined range (N in S166), it is determined that the LED image candidate is not the third LED 40c (S170).

  FIG. 13 is a flowchart showing the procedure of the fourth LED detection process shown in S126 of FIG. The number of constituent pixels of the LED image candidate assumed to be the first LED 40a is N. First, the LED image extraction unit 142 determines whether or not the number of pixels of the LED image candidate is ½ × N or more and 2N or less (S180). If the number of constituent pixels of the LED image candidate is within a predetermined range with respect to the number of constituent pixels N (Y in S180), the angle formed by the vector of the first LED 40a to the second LED 40b and the vector of the third LED 40c and the LED image candidate is It is determined whether or not it is equal to or less than a predetermined value (S182). If the angle formed by the vectors of the first LED 40a to the second LED 40b and the vector of the third LED 40c and the LED image candidate is equal to or smaller than a predetermined value (Y in S182), the vector of the first LED 40a to the third LED 40c and the vector of the second LED 40b and the LED image candidate It is determined whether or not the angle formed by is less than or equal to a predetermined value (S184). If the angle formed by the vectors of the first LED 40a to the third LED 40c and the vector of the second LED 40b and the LED image candidate is equal to or smaller than a predetermined value (Y in S184), it is determined that the LED image candidate can be the fourth LED 40d. Existence is confirmed (S186). On the other hand, when the number of constituent pixels of the LED image candidate is not within the predetermined range (N in S180), the angle formed by the vector of the first LED 40a to the second LED 40b and the vector of the third LED 40c and the LED image candidate exceeds a predetermined value ( If the angle formed by the vectors of the first LED 40a to the third LED 40c and the vector of the second LED 40b and the LED image candidate exceeds a predetermined value (N of S184), it is determined that the LED image candidate is not the fourth LED 40d. (S188).

  As described above, the LED image extraction unit 142 can extract the images of the first LED 40a to the fourth LED 40d from the logical product image. The extracted position information of the images of the first LED 40 a to the fourth LED 40 d is sent to the position estimation unit 104.

  The position estimation unit 104 acquires position information in the frame images of the images of the first LED 40a to the fourth LED 40d based on the two-dimensional arrangement of the LEDs 40 in the controller 20. The position estimation unit 104 calculates affine transformation of the extracted first LED 40a, second LED 40b, third LED 40c, and fourth LED 40d, and calculates the position and orientation in real space. The position in the real space is a coordinate (X, Y, Z) in the orthogonal coordinate system, and the posture in the real space is a rotation angle in the X axis, the Y axis, and the Z axis. By obtaining the coordinates and rotation angles in the three axes, the position estimation unit 104 can estimate the position in the space including the distance from the imaging device 2. The position estimation unit 104 may estimate the position of the controller 20 in the frame image by obtaining affine transformation. For example, the position in the frame image can be acquired by taking the XY coordinates in the XYZ space. The position estimation unit 104 sends the estimated position and orientation of the controller 20 to the application processing unit 110.

  As described above, the position and orientation of the controller 20 may be calculated by affine transformation. For example, the game apparatus 10 turns on three of the four LEDs 40 with respect to a specific controller 20. It is also possible to acquire the position and orientation of the controller 20 from the three LED images that have been instructed. The game apparatus 10 extracts lighting images of three LEDs 40, for example, the first LED 40a, the second LED 40b, and the third LED 40c, using a method similar to the algorithm described above, and the position estimation unit 104 determines the position and orientation of the controller 20 May be estimated. At this time, the position estimation unit 104 grasps in advance the positional relationship between the first LED 40a, the second LED 40b, the third LED 40c, and the fourth LED 40d, and extracts LED image candidates that satisfy the positional relationship among the first LED 40a, the second LED 40b, and the third LED 40c. By doing so, the position and orientation of the controller 20 can be estimated.

  In order to avoid erroneous recognition of the LED 40, the position estimation unit 104 determines the position of the LED image acquired this time when the position of the previously acquired LED image and the amount of change in the position of the LED image acquired this time exceed a predetermined value. Treat as an error. That is, if the position or orientation changes by a predetermined value or more during one frame, another object may be erroneously recognized as the LED 40 due to a disturbing element, and thus the extracted position information of the LED 40 is discarded.

  However, even in such a case, the controller 20 may actually move with a large amount of change. Therefore, the position estimation unit 104 counts the number of consecutive errors, and the count value is predetermined. When the value is reached, the position of the LED image acquired at that time may be processed as normal.

  The position estimation unit 104 can also estimate the moving direction and speed of the LED 40 based on, for example, the current frame image or logical product image. For example, when the shutter speed of the imaging device 2 is decreased, the image of the LED 40 is captured as an ellipse or a rectangular parallelepiped. Therefore, not only the moving direction but also the moving speed can be estimated from the shape of the LED image. For example, if the moving direction and speed can be estimated, the position and orientation of the controller 20 can be predicted even when the imaging apparatus 2 suddenly disappears. When the shapes of all the LEDs 40 are biased in the same direction, the moving direction and speed of the controller 20 may be estimated.

  The application processing unit 110 reflects the estimated position and orientation of the controller 20 in the game application process as a game input. Although various reflection methods can be considered, most intuitively, the virtual object is displayed following the position in the game image corresponding to the estimated position of the controller 20. When the user moves the controller 20 with respect to the imaging device 2, the virtual object in the game image can be moved following the movement.

  In an ideal state, the position and orientation of the LED 40 may be obtained for each frame image and reflected in the game image. On the other hand, depending on the orientation of the controller 20, the ambient light environment, etc. There may also be situations where the position or orientation of the LED 40 is not required. If the position and orientation of the LED 40 are suddenly obtained after such a situation continues, the virtual object that has been stopped suddenly moves to a new position, giving the user a sense of incongruity. For this reason, even if the position and orientation of the LED 40 can be detected from the logical product image, the game image is not moved to the position where the virtual object is acquired. The follow-up process is performed so that the movement can be realized.

  FIG. 14A and FIG. 14B are diagrams for explaining an algorithm for the virtual object corresponding to the controller to realize a smooth follow-up operation in the game screen. For convenience of explanation, the position change of the virtual object will be described, but the same algorithm can be applied to the attitude change.

The position parameters derived by obtaining the affine transformation in the position estimation unit 104 are P _i−1 , P _i , and P _{i + 1} (the subscript of P indicates the number of the game screen frame). First, consider the case where the frame image is switched and the position parameter changes from P _i to P _{i + 1} .
When a new position parameter (P _{i + 1} ) is obtained, the position parameter reflected on the game screen is obtained by the following equation.
Q _{i + 1} = (Q _i + P _{i + 1} ) / 2
In other words, instead of moving directly to a new position parameter P _{i + 1} set, and the position parameter on the game screen of the previous frame, the middle point between the new position parameter P _{i + 1} of the current frame, as a new position parameter on the game screen To do. The position parameter _{Q i + 1} need not be a position parameter _{Q i} and the position parameter _{P i + 1} of the midpoint, the line segment between the position parameter _{Q i} position parameter _{P i + 1} A: a point which divides the B May be.

FIG. 14B shows an example when _{Pi + 1} cannot be acquired as the position parameter. In such a case, the effect of this algorithm can be exhibited. In this case, the latest position parameter is used instead of _{Pi + 1} . Here with the latest position parameter is a P _i,
Q _{i + 1} = (Q _i + P _i ) / 2
As required.
When the position parameter Q in the game image is acquired by this algorithm, the virtual object can always move little by little even if the controller 20 in the space is greatly displaced or cannot be extracted from the logical product image. Therefore, it is possible to avoid a situation in which the virtual object suddenly stops moving or suddenly moves in the game image, and a smooth follow-up process can be realized for the controller 20.

  In this embodiment, the controller 20 is imaged by the imaging device 2, the position and orientation of the controller 20 are estimated from the frame image, and the position information and orientation information are reflected in the processing of the game application. The LED 40 of the controller 20 may change the lighting pattern according to the progress of the game application, for example.

  Immediately after the user participates in the game application, the game apparatus 10 may transmit a control signal for lighting the four LEDs 40 to the controller 20 at a predetermined low frequency. The blinking instruction unit 106 generates a control signal that instructs the controller 20 to blink a predetermined low frequency. This low frequency blinking control signal is transmitted from the wireless communication module 108 to the controller 20. When there are a plurality of controllers 20, a blinking frequency is set for each controller 20.

  The controller 20 lights the LED 40 at a designated low frequency, and the imaging device 2 captures a frame image at an imaging speed of 30 frames / second. The game apparatus 10 extracts LED images that are turned on and off at predetermined intervals from a plurality of frame images. When there are a plurality of controllers 20 in the game system 1, a lighting cycle is set for each controller 20. Thus, the position of the controller 20 can be confirmed, and each controller 20 can be distinguished and recognized. This process should ideally be performed once after participating in the game application, but the controller 20 may suddenly disappear from the imaging device 2 in the real space. Therefore, each controller 20 may blink the LED 40 at a low frequency set for each controller at a predetermined cycle such as every few seconds.

  For example, when the four LEDs 40 are turned on and off at a low frequency, a frame in which the LED 40 can be imaged and a frame in which the LED 40 cannot be imaged are switched at a predetermined cycle. When the LED 40 is controlled to be turned on at 45 Hz with respect to the imaging cycle (30 frames / second), the situation is repeated in which images are taken in two consecutive frames and the next is not taken. Since the frame image may be captured or not captured, the position information can be acquired even when the LED 40 is stationary.

  Here, the low-frequency lighting means lighting at a frequency such that a frame in which LED light emission is captured and a frame in which imaging is not performed can exist with respect to the imaging cycle. On the other hand, high-frequency lighting means lighting at a frequency such that a frame image is always captured with respect to the imaging cycle (30 frames / second) of the imaging device 2. For example, even if the LED 40 is turned on at 1 kHz, for example, if the imaging speed of the imaging device 2 is very high, a frame in which LED light emission is captured and a frame in which the LED emission is not captured appear. Even if it exists, position information can be acquired.

  The position and orientation of the LED 40 that is lit at a high frequency are used as an input for the game application. Therefore, the position information of the controller 20 estimated by the position estimation unit 104 is sequentially sent to the application processing unit 110 and reflected in the application processing. An input by the user operating the direction key 21 or the like of the controller 20 is also transmitted to the application processing unit 110 via the wireless communication module 108. The application processing unit 110 advances the game from the position / posture information of the controller 20 and the operation input in the controller 20, and generates an image signal and an audio signal indicating the processing result of the game application.

  As a modification, it is also possible to recognize a specific LED 40 by lighting one of the four LEDs 40 with a blinking pattern different from that of the other LEDs 40. For example, three LEDs perform high-frequency lighting, and one LED performs low-frequency lighting, so that, for example, it is possible to check which of the first LEDs 40a in the frame image is. At this time, not only the LED 40 can be uniquely recognized, but also its posture can be acquired.

The features of the invention disclosed in the embodiments may be defined by the following items.
(Item 1)
An image processing apparatus for acquiring a position of a light emitter image from an input frame image,
A difference image generation unit for generating a difference image between the previous frame image and the current frame image;
A binarized image generating unit that binarizes the current frame image using a predetermined threshold and generates a binarized image;
A logical operation unit that generates a logical operation image obtained by performing a logical operation on the difference image and the binarized image;
An extractor for extracting a light emitter image from the logical operation image;
A position acquisition unit for acquiring the position of the illuminant image in the frame image;
An image processing apparatus comprising:
(Item 2)
Item 1. The item according to Item 1, wherein the logical operation unit generates a logical operation image by performing a logical product operation on a pixel value of the difference image and a pixel value of the binarized image for each pixel. Image processing apparatus.
(Item 3)
Item 1 or 2 is characterized in that the extraction unit recognizes a two-dimensional pattern formed by a plurality of light emitters in advance and performs a light emitter image extraction process using the two-dimensional pattern. The image processing apparatus described.
(Item 4)
The position acquisition unit processes the position of the light emitter image acquired this time as an error when the amount of change between the position of the light emitter image acquired last time and the position of the light emitter image acquired this time exceeds a predetermined value. The image processing apparatus according to any one of items 1 to 3.
(Item 5)
The position acquisition unit counts the number of errors that occur continuously, and when the count value reaches a predetermined value, processes the position of the illuminator image acquired at that time as a normal one. Item 5. The image processing apparatus according to Item 4.

(2) Method for Acquiring Position Information and / or Posture Information Using Two-dimensional Code In the method for acquiring position information and / or posture information of controller 20 described below, in game system 1 shown in FIG. Instead of the LED 40, a two-dimensional code is provided. This two-dimensional code is picked up by the image pickup device 2 and used to acquire position information and posture information in the real space of the controller 20 in the game device 10.

  FIG. 15 shows a controller 20 having a two-dimensional code. The controller 20 is configured to have a two-dimensional code 70 for uniquely identifying itself on the housing rear surface 29. The two-dimensional code 70 and its identification method may be the same as the two-dimensional code and its identification method disclosed in, for example, Japanese Patent Application Laid-Open No. 2006-72667 filed by the present applicant.

  FIG. 16 shows an example of a two-dimensional pattern constituting the two-dimensional code 70. The two-dimensional code 70 may be stamped on the housing rear surface 29 of the controller 20 or may be affixed as a printed matter. In the two-dimensional code 70, a direction instruction unit 11 and an identification unit 12 are formed. The direction instructing unit 11 is provided to indicate the upper side of the two-dimensional code 70, and the identification unit 12 is provided to express a characteristic part for uniquely identifying the two-dimensional code 70. The identification unit 12 is code information created with a plurality of blocks in a predetermined section. Among the plurality of blocks, the quadrangular blocks are assigned in common to the plurality of two-dimensional codes 70, and the characteristic portions are substantially formed by blocks other than the quadrangular blocks. The quadrangular block is used to measure the distance from the imaging device 2.

  Referring back to FIG. 5, when the image processing unit 102 recognizes the two-dimensional code 70 in the frame image, the image processing unit 102 and the imaging device 2 are two-dimensionally based on the length between the four square blocks in the image of the two-dimensional code 70. The distance from the code 70 is calculated and obtained. Further, the image processing unit 102 detects the direction indicating unit 11 of the two-dimensional code 70 and obtains the direction in which the two-dimensional code 70 is facing in the frame image. In this case, the direction in which the direction instruction unit 11 exists is higher than the identification unit 12. Further, the image processing unit 102 acquires identification information of the two-dimensional code 70 from a block other than the four corners of the identification unit 12.

  The image processing unit 102 has a function of recognizing a state where the two-dimensional code 70 is tilted in real space by image analysis. The result of the image analysis performed by the image processing unit 102 is sent to the position estimation unit 104. The position estimation unit 104 estimates the position and orientation of the two-dimensional code 70 based on the image analysis result in the image processing unit 102.

  FIG. 17 shows the configuration of the image processing unit 102. The image processing unit 102 includes a frame image acquisition unit 130, a real object extraction unit 150, a state determination unit 152, an identification information acquisition unit 154, an identification information storage unit 156, and a transmission unit 160. The identification information storage unit 156 stores characteristic part information for identifying a real object and identification information for identifying the real object in association with each other. Specifically, the pattern information of the identification unit 12 of the two-dimensional code 70 in FIG. 16 and the identification information are stored in a one-to-one correspondence. For example, the identification information may be used for character assignment in the game apparatus 10. In particular, in a game application that permits the presence of a plurality of two-dimensional codes 70, the two-dimensional code 70 can be recognized by associating each of the two-dimensional codes 70 with identification information. The state determination unit 152 determines the state of the real object in the set coordinate system, and specifically includes a posture determination unit 162 that determines the posture and a position determination unit 164 that determines the position.

  The frame image acquisition unit 130 acquires a real space frame image captured by the imaging device 2. The imaging device 2 periodically acquires frame images, and preferably acquires frame images at 1/30 second intervals.

  The real object extraction unit 150 extracts a real object image, that is, an image of the two-dimensional code 70 from the frame image. This processing is performed by converting image information into a binary bit representation and extracting an image of the two-dimensional code 70 from the bit representation. This is so-called dot processing, and image extraction processing may be performed by detecting on and off of bits. Further, this processing may be performed by a known image matching technique. In that case, the real object extraction unit 150 registers image information of a real object to be used in advance in a memory (not shown) for the matching process. By matching the pre-registered image information with the captured image information, the image of the two-dimensional code 70 can be cut out from the frame image.

  The position determination unit 164 determines a two-dimensional position in the frame image of the real object image. Specifically, the position in the frame image is determined based on the coordinates of the center point of the real object image in the frame image. Further, the position determination unit 164 determines the distance between the imaging device 2 and the two-dimensional code 70 based on the length between the four corners of the identification unit 12 in the image of the two-dimensional code 70. Accordingly, the position of the controller 20 in the three-dimensional real space can be acquired together with the two-dimensional position in the frame image.

  The posture determination unit 162 determines the posture of the real object image. For example, the posture of the real object image is determined by the position of the direction instruction unit 11 with respect to the identification unit 12. Further, when the controller 20 is tilted, the distances between the four corners of the identification unit 12 are not equal, so that the tilted posture can be determined based on the amount of change.

  The identification information acquisition unit 154 extracts a feature portion from the real object image and acquires corresponding identification information from the identification information storage unit 156. The game system 1 according to the present embodiment can also cope with a plurality of different two-dimensional codes 70. For example, when a plurality of different two-dimensional codes 70 are provided in the plurality of controllers 20, On the other hand, different identification information is associated with each other.

  The position information determined by the state determination unit 152, the posture information, and the identification information acquired by the identification information acquisition unit 154 are associated with each other and sent from the transmission unit 160 to the position estimation unit 104. When a plurality of two-dimensional codes 70 are included in the frame image, position information, posture information, and identification information are associated with each two-dimensional code 70 and sent from the sending unit 160 to the position estimating unit 104. . As described above, the position estimation unit 104 can acquire the position information, the posture information, and the identification information of the controller 20, and can estimate the position and posture of the controller 20.

  Note that the process of associating the identification information with the position information and the posture information as described above can be executed even when the position information and the posture information of the controller 20 are acquired using the LED 40.

(3) Method for Acquiring Position Information and / or Posture Information Using Acceleration Sensor etc. In addition to the internal configuration of the controller shown in FIG. 4, the controller 200 may further include a triaxial acceleration sensor and gyro sensor. Good.
FIG. 18 illustrates an internal configuration of the controller 200 according to the embodiment. In FIG. 18, illustration of the configuration of the input receiving unit 52, the PWM control unit 54, the driving unit 56, and the like illustrated in FIG. The controller 200 includes a three-axis acceleration sensor 300 that detects acceleration components in the three-axis directions of XYZ, and a gyro sensor 302 that detects an angular velocity in the housing horizontal plane (XZ plane). Here, the longitudinal direction of the casing is set as the X axis, the height direction is set as the Y axis, and the short direction (depth direction) is set as the Z axis. The gyro sensor 302 is disposed in the central region in the housing, for example, in the vicinity of the lower portion of the button 31 with LED in the housing top view shown in FIG. Similarly, the three-axis acceleration sensor 300 is preferably arranged near the center of the housing. Detection values from the three-axis acceleration sensor 300 and the gyro sensor 302 are supplied from the main control unit 50 to the wireless communication module 58 at a predetermined cycle. The wireless communication module 58 transmits the detection values obtained by the three-axis acceleration sensor 300 and the gyro sensor 302 to the wireless communication module 108 of the game apparatus 10 at predetermined intervals together with operation inputs such as the direction key 21 and the operation button 26.

  FIG. 19 shows a configuration of the game apparatus 10 according to the embodiment. The game apparatus 10 uses the wireless communication module 108 to determine a direction (Y-axis direction) perpendicular to the three-axis acceleration component measured by the controller 200 by the three-axis acceleration sensor 300 and the gyro sensor 302 and the housing horizontal plane (XZ plane). ) Get the angular velocity component around. The triaxial acceleration component and the angular velocity component are supplied from the wireless communication module 108 to the position estimation unit 250. The position estimation unit 250 can estimate the position and orientation of the controller 200 based on the acquired triaxial acceleration component and angular velocity component. Specifically, the position estimation unit 250 can acquire the attitude of the controller 200 based on the triaxial acceleration component. Further, the moving speed can be acquired by integrating the three-axis acceleration component once, and the position can be acquired by integrating twice.

  In the controller 200 of the embodiment, the gyro sensor 302 detects the angular velocity component around the Y-axis direction. Therefore, the position estimation unit 250 may estimate the rotation state, that is, the rotation posture, of the controller 200 on the horizontal plane based on the detection value by the gyro sensor 302. When the triaxial acceleration sensor 300 can acquire the triaxial acceleration component with high accuracy, the rotation state of the controller 200 in the horizontal plane may be estimated from the triaxial acceleration component.

  As described above, the controller 200 includes the triaxial acceleration sensor 300 and the gyro sensor 302, so that the position and orientation of the controller 200 can be acquired without analyzing the frame image by the imaging device 2. . Note that the estimation accuracy of the position and orientation of the controller 200 may be increased by further using the analysis result of the frame image by the imaging device 2. Since the position and orientation of the controller 200 can be estimated by using measurement values of the triaxial acceleration sensor 300 and the like, the position and orientation information of the controller 200 can be reflected in the processing of the game application.

  In the following, application examples that can be implemented in the game system according to the embodiment are shown on the premise that the positions and orientations of the controller 20 to the controller 200 can be acquired. Hereinafter, the controller 200 will be described on behalf of the controller. The game apparatus 10 estimates the posture and position of the controller 200 based on the three-axis acceleration component and the angular velocity component. However, since the posture and position of the controller 20 can also be estimated from the captured image, the following contents are as follows: The controller 20 can also be realized.

  FIG. 20 shows a configuration of the application processing unit 110. The application processing unit 110 includes an input reception unit 400, an acquisition unit 402, a change detection unit 404, an object control unit 406, and an image signal generation unit 408. FIG. 20 shows only a configuration for generating an image related to an object. Actually, the background image of the object or the like is generated by another configuration, and the audio signal is also generated by another configuration.

  The input receiving unit 400 receives an operation input input from the game controller 200 from the wireless communication module 108. The acquisition unit 402 acquires position information or posture information in the real space of the game controller 200 from the position estimation unit 250. The change detection unit 404 detects a change in the position or posture of the game controller 200 from the position information or posture information acquired continuously in time by the acquisition unit 402. The change detection unit 404 can acquire the moving speed and moving direction of the controller 200 by detecting a change in position, and can acquire the rotating speed and rotating direction of the controller 200 by detecting a change in posture.

  The object control unit 406 controls the movement of the object in the virtual space based on the operation input received by the input receiving unit 400 as a basic function. The object control unit 406 of the present embodiment further has a function of controlling the movement of the object using the position information or the posture information acquired by the acquisition unit 402. Furthermore, the object control unit 406 also has a function of controlling the movement of the object using the change in position or orientation acquired by the change detection unit 404.

  For example, the object control unit 406 may determine the posture of the object in the virtual space using position information or posture information. Further, the object control unit 406 may start or end the motion control of the object using the position information or the posture information when the change detection unit 404 detects a predetermined change in the position or the posture.

  When the acquisition unit 402 acquires the three-dimensional position information or posture information of the game controller 200 in the real space, the object control unit 406 uses the three-dimensional position information or posture information to obtain an object in the virtual three-dimensional space. The operation may be controlled. In addition, when the movement of the controller 200 is large and the object to be displayed on the image display device 3 is likely to be out of the screen, the object control unit 406 may execute a notification process for the user. The object whose operation is controlled by the object control unit 406 is converted into an image signal by the image signal generation unit 408 together with other background images.

Examples of applications are shown below.
<Tennis game>
In the tennis game, the height of the controller 200 in the frame image is the hitting height of the tennis ball, the direction of the controller 200 is the ball hitting direction, and the speed at which the controller 200 is moved is the strength to hit the ball. The game can be advanced. Since these movements approximate the movement of swinging an actual racket, it is possible to give the user a feeling close to actually playing tennis. For example, the controller 200 may perform the same operation as the racket, and the angle and direction of the swing at this time may be used.

  The object control unit 406 acquires the height of the controller 200 from the position information of the controller 200 acquired by the acquisition unit 402, and acquires the orientation of the controller 200 from the posture information. Further, the object control unit 406 acquires the moving speed of the controller 200 from the change detection unit 404. Based on these pieces of information, the object control unit 406 determines the posture of an object such as a racket or character and operates it.

  The object control unit 406 may control the operation of each virtual object based on the position information and the posture information of the controller 200 acquired by the acquisition unit 402. For example, the object control unit 406 calculates a difference from the reference position and the reference posture. Thus, the motion of each virtual object may be controlled based on the difference value. For example, the object control unit 406 may acquire the position and orientation of the controller 200 when the user operates the buttons of the controller 200 as the reference position and reference orientation used for the object motion control process. The object control unit 406 detects a change amount (difference) between the reference position and the reference posture from the position information and the posture information acquired by the acquisition unit 402, and the motion of the virtual object based on the change amount. Execute control. Note that the object control unit 406 may notify the change detection unit 404 of the reference position and reference orientation, and the change detection unit 404 may detect the amount of change from the reference position and reference orientation. Hereinafter, an example in which the motion control of the object is performed based on the amount of change from the reference position and the reference posture will be described, but the motion control of the object based on the position and posture in the real space coordinates can be similarly executed.

By making it possible to use the buttons and key operations of the controller 200 as game inputs as they are, it is possible to make the game input sensibly novel while maintaining the conventional game operation inputs.
For example, the strength of hitting the ball may be determined by a button operation of the controller 200. Combining conventional button operations with innovative game input, new gameplay can be realized. For example, it is possible to create a tennis game in which a player can hit a strong serve by raising the ball by inputting a button while specifying the position at which the ball is dropped on the opponent's court in the direction of the controller 200 and further moving the controller 200. Is possible.

  Using the position information and / or posture information of the controller 200, it is also possible to sort shots such as volley, smash, lob, and drop shot. For example, the object control unit 406 causes the character to perform a volley operation when the amount of change in the height direction from the reference position of the controller 200 is small, and causes the character to perform a smash operation when the amount of change is large in the positive direction (upward). When it is large in the negative direction (downward), the lob may be operated.

  FIG. 21A shows a state where the character performs volley, and FIG. 21B shows a state where the character performs smash. In the example of FIG. 21, two virtual objects, a game character and a racket, are represented. However, by actually reflecting the position information of the controller 200 not only on the posture of the character but also on the inclination of the racket, the player actually plays tennis. It is possible to give the user a feeling to do.

<Motorcycle game>
In a motorcycle drive game, the attitude of the controller 200 may be operated to be used as a game input for controlling the motorcycle. At this time, the speed may be determined by the speed at which the controller 200 is moved, or may be determined by button input. In this way, conventional button input, static input based on the position and orientation of the controller 200, and dynamic input based on state changes such as the moving speed of the controller 200 are used in combination. Thus, the operability of the game application can be improved.

<Character input game>
In a game where characters are input on the screen, characters can be input on the screen by moving the position of the controller 200. The object control unit 406 generates a diagram object to be output to the screen of the image display device 3 so as to follow the two-dimensional positional information in the housing X axis and Y axis directions supplied from the acquisition unit 402. In the controller 200, the longitudinal direction of the casing is set as the X axis, the height direction is set as the Y axis, and the lateral direction (depth direction) is set as the Z axis. Called the Y-axis and the housing Z-axis. In the game application, in the screen of the image display device 3, the horizontal direction is set as the X axis, the vertical direction is set as the Y axis, and the depth direction is set as the Z axis. These are called the Y axis and the screen Z axis.

  Thereby, the movement in the housing XY plane can be traced on the screen XY plane. When this process is performed only by the movement of the controller 200, the start and end of the trace process may be determined by the movement of the controller 200. At this time, the change detection unit 404 detects a predetermined change in the position information or the posture information and notifies the object control unit 406, and the object control unit 406 receives the notification and determines the start and end of the trace processing. . The motion change that triggers the start and end may be the same, or may be set for each start process and end process. In the same case, the movement change is processed as a trigger for the start process if the trace process is not being executed, and as a trigger for the end process if the trace process is being executed.

  When the object control unit 406 uses a two-dimensional motion on the housing XY plane as a game input, the other direction, that is, the motion in the housing Z-axis direction, is used as a trigger for starting or ending the tracing process. May be. Since the movement in the housing Z-axis direction is not used for motion control of the diagram object, it can be suitably used as a start or end trigger. That is, by using information other than the components necessary for the movement of the object in the game as instruction information for the game, the game application can be advanced without using the key operation. For example, in the character input game, since the rotation operation of the controller 200 is not used as the game input, the controller 200 may be rotated, that is, the posture may be changed to input an instruction to start or end the trace process.

<Electrode bar game>
The electrode bar game is a game in which an electrode bar is placed in a metal course frame and the electrode bar is carried to the goal within a time limit so that it does not hit the course frame or obstacles. It can be executed as an application. Currently, there is a two-dimensional electrode bar game, but there is no three-dimensional electrode bar game.

  The object control unit 406 may control the operation of the electrode rod that is a virtual object in the virtual three-dimensional space using the three-dimensional position information and posture information of the controller 200. That is, by providing the course frame in the screen depth direction (screen Z-axis direction) in the game screen, the user moves the controller 200 not only on the housing XY plane but also in the housing Z-axis direction. It can give the same experience as an actual electrode bar game.

  As described above, various application examples have been described. When the object control unit 406 causes the movement of the controller 200 to be traced on the screen, the controller 200 moves off the screen of the image display device 3 when the operation amount of the controller 200 increases. Such a case also occurs. For example, such a situation may occur in the character input game described above.

  When the virtual object corresponding to the controller 200 is likely to be out of frame from the screen of the image display device 3, the object control unit 406 notifies the user that the frame is likely to be out of frame based on the position information of the controller 200. Processing may be performed. For example, when the controller 200 is about to be out of frame, a vibration control signal for vibrating the vibrator to the controller 200 may be generated and transmitted from the wireless communication module 108. In addition, when a virtual object that follows the movement of the controller 200 is displayed as a game image, the user may be notified that the frame is likely to be out by moving the virtual object so as to stand out from the user. For example, the user may be notified by making a special movement that is repelled inward from the frame of the screen.

  Whether or not the frame is about to be out may be determined based on whether or not the position information in the vicinity of the frame is obtained when the position information of the controller 200 is converted into the screen coordinates. At this time, the change detection unit 404 may detect the moving direction and speed of the controller 200 and predict whether or not to frame out within a predetermined time from the information. This prediction process may be executed on the premise that the prediction process is in the vicinity of the frame.

  When using the imaging device 2, for example, the position may be determined based on the position of the controller 200 in the frame image. This notification is made when it is detected that the controller 20 is present near the edge of the frame image. Processing may be performed.

Further, as described in the present embodiment, various applications can be executed based on the position information and posture information of the controller 200 or the controller 20.
For example, the game object can be made to make a large soap bubble by rotating the controller using the input button together. In addition, by rotating the controller or reciprocating the posture, it is possible to knead the game object with bread in a cooking game. Also, using the position information of the controller, the game object can be operated or commanded by a tambourine or maraca. In addition, by using the three-dimensional position information of the controller, it is possible to execute a game in which a block is broken by hitting a sphere with a three-dimensionally arranged block. It is also possible to switch the player by passing the controller 200 so as to catch the ball between players. If there is a pseudo camera in the game, the viewpoint of the pseudo camera can be moved based on the position information of the controller. Moreover, based on the rotation information of the controller, an operation of turning a virtual door knob can be executed. Further, by actually moving the controller, the virtual object can be moved following the movement. In addition, by capturing the three-dimensional movement of the controller, the movement is captured as a virtual flea movement, creating a sculpture, and by capturing the two-dimensional movement, the movement is moved to the movement of the virtual brush. Or create a painting. Furthermore, by applying an instantaneous impact to the controller, when the impact level exceeds a predetermined value, the progress of the game may be controlled such as vibrating the controller or reflecting the impact on the game object. . If you have an acceleration sensor etc., you can detect the impact by its acceleration, and if you use an LED image, the controller will give an impact if the LED image becomes elliptical or not captured instantaneously. You may determine that you have received.

It is a figure which shows the use environment of the game system concerning the Example of this invention. It is a figure which shows the external appearance structure of a controller. It is a figure which shows the external appearance structure of the back side of a controller. It is a figure which shows the internal structure of a controller. It is a figure which shows the structure of a game device. It is a figure which shows the structure of an image process part. (A) is a figure which shows the present frame image, (b) is a figure which shows the difference image of a present frame image and a previous frame image. (A) is a figure which shows the binarized image of the present frame image, (b) is a figure which shows a logical product image. It is a flowchart which shows the process sequence of a detection process of a LED image candidate. It is a flowchart which shows the process sequence of LED extraction process. It is a flowchart which shows the process sequence of the 2nd LED detection process shown to S122 of FIG. It is a flowchart which shows the process sequence of the 3rd LED detection process shown to S124 of FIG. It is a flowchart which shows the process sequence of the 4th LED detection process shown to S126 of FIG. It is a figure for demonstrating the algorithm for the virtual object corresponding to a controller to implement | achieve smooth tracking operation | movement within a game screen. It is a figure which shows the controller which has a two-dimensional code. It is a figure which shows an example of the two-dimensional pattern which comprises a two-dimensional code. It is a figure which shows the structure of an image process part. It is a figure which shows the internal structure of the controller concerning an Example. It is a figure which shows the structure of the game device concerning an Example. It is a figure which shows the structure of an application process part. An example of character movement is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Game system, 2 ... Imaging device, 3 ... Image display device, 4 ... Sound output device, 10 ... Game device, 20 ... Controller, 300 ... 3-axis acceleration Sensor 302, gyro sensor 110 application processing unit 400 input receiving unit 402 acquisition unit 404 change detection unit 406 object control unit 408 ..Image signal generator

Claims (6)

An acquisition unit for acquiring position information or posture information in the real space of the game controller;
An object control unit that controls the movement of the object in the virtual space using the position information or the posture information;
A change detection unit that detects a change in the position or posture of the game controller from the position information or the posture information acquired continuously in time in the acquisition unit;
When the change detection unit detects a predetermined change in position or posture, the object control unit starts or ends motion control of the object using the position information or the posture information. Game device.   2. The object control unit according to claim 1, wherein when the change detection unit detects a predetermined change in a position or posture that is not used for object motion control, the object control unit starts or ends the object motion control. Game device.   When the change in the position or orientation of the game controller that triggers the start and end of object motion control is the same, the object control unit executes a predetermined change in the position or orientation of the game controller to execute object motion control. The game apparatus according to claim 1, wherein the game device is processed as a trigger for a start process and as a trigger for an end process if an object motion control is being executed.   The game apparatus according to claim 1, wherein the object control unit determines the posture of the object in the virtual space using the position information or the posture information. The acquisition unit acquires three-dimensional position information or posture information of the game controller in real space,
5. The game device according to claim 1, wherein the object control unit controls the movement of the object in a virtual three-dimensional space using three-dimensional position information or posture information. On the computer,
A function of acquiring position information or posture information in the real space of the game controller;
A function of controlling the movement of the object in the virtual space using the position information or the posture information;
A function for detecting a change in the position or posture of the game controller from the position information or the posture information acquired continuously in time, and a program for realizing the function,
The object motion control function includes a function of starting or ending motion control of an object using the position information or the posture information when a predetermined change in position or posture is detected by the change detection function; A featured program.

Images