KR20090072435A - Wireless game controller with infra-red position tracker and accelerometer for user's motion recognition - Google Patents

Abstract

A wireless game controller combining an infrared position tracker and an accelerometer, a game server and a user operation recognition method are provided to simultaneously perform motion tracking by accelerometer and position tracking by an infrared position tracker, thereby quickly recognizing various motions. An acceleration tracking unit(22) produces acceleration data. An infrared transmitting unit(24) transmits an infrared signal to an infrared camera mounted on a game server. A wireless communication unit(23) performs wireless telecommunication with the game server. A central processing unit(21) transmits the acceleration data produced by the acceleration tracking unit to a server through the wireless communication unit. The central processing unit processes the inputted signal.

Description

Wireless Game Controller with Infra-red Position Tracker and Accelerometer for User's Motion Recognition}

The present invention relates to a wireless game control device and a game server, and more particularly, by simultaneously performing a motion tracking using an acceleration tracker and a location tracking using an infrared video camera, the advantages and disadvantages of acceleration tracking and infrared location tracking are complemented with each other. The present invention relates to a wireless game control device capable of recognizing an object, a game server, and a user motion recognition method.

Allowing game users to immerse themselves in a virtual environment is one of the important elements that a game system must provide. In addition to providing realistic visual and auditory effects to increase immersion, many of the more recent experience-based game systems provide users with a special handheld game control device that can give them a tactile touch. The game controller is not only used as an input device for the user to play the game through natural movement but also serves as an output device for providing a tactile touch.

In order to use the game controller as an input device, it is necessary to track the position of the game controller. The conventionally used tracking method is to measure the acceleration generated when the game controller is moved with an acceleration tracker inside. This acceleration measurement method is simple to implement because it requires only a modular acceleration tracker. Using an acceleration tracker, you can get three-axis acceleration information and get new acceleration information, typically 200 to 2000 times per second, depending on the performance of the acceleration tracker and the communication speed of the peripherals.

Acceleration information can be efficiently used to extract the part where the operation is made from sequentially incoming data. When the movement is made, it is impossible to carry out constant motion for a long time, so the acceleration component always appears. In other words, it is possible to efficiently extract the motion region by using the change in acceleration occurring in the section where the motion occurs.

However, since the method using the acceleration tracker knows only the acceleration information of the game controller, the game server can easily recognize the motion when the motion can be determined only by the change of the motion regardless of the position of the game controller. However, if location is an important operation, there is a problem that cannot be easily recognized. For example, in the case of using the numbers 0 and 6, it is difficult to distinguish the two operations only by the acceleration information. That is, at the beginning of both movements, the acceleration component to the left is shown. Gradually, the movement changes in the order of down, right, up, and left, and appears in the acceleration information. If the user performs the motion at a constant speed, the motion can be distinguished to some extent by the size and duration of each direction component.However, in actual use, the motion is different depending on the performance and the speed can be changed even within one motion. There is a limit.

On the other hand, in order to properly provide the user with experience-based content using the game user's motion by using the game controller, in many cases it is necessary to track the position and direction of the game controller. In order to track the position and direction of the game control device, there is a limitation in the application that can be applied only by its relative tracking information, so it is necessary to track the absolute position and direction around the screen that the user is looking at. For example, it is necessary to track the absolute coordinates of the game controller in order to apply it to an application such as a shooting game using the game controller as a position indicator device such as a laser pointer.

There are many ways to track a large area of absolute location, but vision tracking with infrared cameras is widely used. Using an infrared camera has the advantage of being able to track a position with a simple marker and having a relatively large tracking area, without adding a complex sensor to a tracking object, compared to other methods. The use of infrared cameras can be disadvantageous because they are sensitive to the ambient light environment and the problem of markers being hidden, but these disadvantages can be minimized due to the special nature of the indoor environment and limited game radius. In addition, if there is a camera and server responsible for tracking the vision, the module that performs the tracking can be completely separated from the game controller, so that complex calculations can be burdened on the high performance server side. This allows absolute positioning by simply attaching markers for positioning to the outside of the game controller, while maintaining the characteristics of the game controller that is essentially wireless and the current hardware configuration. In addition, by increasing the number of markers to be attached to the game controller up to four, not only the three-dimensional position but also three-dimensional direction information of the game controller may be obtained.

However, this vision tracking method has the disadvantage that the tracking speed is not very fast due to the limitations of the infrared camera and the large internal calculations. As a result, the amount of data that can be used to recognize each motion is reduced, and the performance is lower than that of using the acceleration information in the recognition of the general motion.

Accordingly, the present invention has been proposed to solve the above problems of the prior art, and complements the advantages and disadvantages of acceleration tracking and infrared location tracking by simultaneously performing a motion tracking using an acceleration tracker and a location tracking using an infrared video camera. An object of the present invention is to provide a wireless game controller capable of recognizing a user's motion, a game server, and a method of recognizing a user's motion.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

A game control apparatus according to the present invention for achieving the above object, the game control apparatus for wireless communication with the game server, acceleration tracking means for generating acceleration data; Infrared transmitting means for transmitting an infrared signal to an infrared camera mounted to the game server; Wireless communication means for wireless communication with the game server; And central processing means for transmitting the acceleration data generated by the acceleration tracking means to the server through the wireless communication means and processing the input signal.

In addition, a game server according to the present invention for achieving the above object, the game server for wireless communication with a game controller, comprising: an infrared camera for receiving an infrared signal from the game controller; Wireless communication means for receiving acceleration data from the game controller; Position tracking means for generating an image obtained by the infrared camera as a binary image, calculating contour curves for the binary image, and extracting three-dimensional position data; Preprocessing means for removing acceleration components due to gravity acceleration and removing noise from the acceleration data received through the wireless communication means; Block extraction means for extracting acceleration data of each operation region from the acceleration data preprocessed by the preprocessing means; Model learning performing means for learning a hidden Markov model using the position data extracted by the position tracking means and the acceleration data of each operation region extracted by the block extracting means; And determining a motion by calculating a similarity between the position data extracted by the position tracking means and the acceleration data of each motion region extracted by the block extracting means with the model trained by the model learning performing means. Means;

In addition, the user motion determination method according to the present invention for achieving the above object, in the user motion recognition method in the game server, (a) generates an image obtained by the infrared camera as a binary image, for the binary image Calculating three-dimensional position data by calculating a contour curve; (b) removing the acceleration component due to gravity acceleration from the acceleration data received from the game controller, and then extracting acceleration data of each motion region; (c) training a hidden Markov model using the extracted position data and the acceleration data of each of the extracted motion regions; And (d) substituting the model learned in step (c) with respect to the extracted position data and the acceleration data of each of the extracted motion regions to determine an operation through similarity calculation.

The present invention as described above, by performing the motion tracking using the acceleration tracker and the location tracking using the infrared video camera at the same time, by complementing the advantages and disadvantages of the acceleration tracking and infrared location tracking can recognize a variety of operations faster and more accurately than the conventional method have.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows a game system to which the present invention is applied. The game system includes a game controller 11 equipped with a game controller 13 and an infrared camera 12.

The game controller 13 is a device that serves to hold game hands and give game commands. Since the game controller 13 is a wireless portable device, it has an internal power tracker and has a Bluetooth wireless communication function for communication with the game server 11. It also has a device that can power an infrared LED that acts as a marker.

The game server 11 is connected to the infrared camera 12 by wire and performs location tracking of the game controller using the image received from the infrared camera 12. The game server 11 performs motion recognition using the location tracking information and the acceleration information transmitted from the game controller through the radio. In addition, the game server 11 executes the game based on the user's input signal, and provides the result to the user through various displays.

2 is a block diagram of a game controller according to the present invention, wherein the game controller includes a central processing unit 21, an acceleration tracker 22, an infrared transmitter 24, and a wireless communication unit 23.

The central processing unit 21 can be configured using the AvrMall Mega8 Board V2.1 module using the ATMEGA48 chip. The acceleration tracker 22 can be configured using the LIS3LV02DQ chip from STMicroelectronics. The chip operates at up to 16-bit resolution and 2560 Hz, but operates at 12-bit resolution and 640 Hz for stable communication and stable operation with the central processing unit. The acceleration tracker 22 transmits acceleration information applied to the current X, Y, and Z axes to the central processing unit 21 through SPI communication.

The wireless communication unit 23 is for performing wireless communication with the game server, and the present invention uses a Bluetooth method. The Bluetooth module can be configured using the ACODE 300A module from Comfile Technology. It operates at 115200bps for smooth communication and communicates with server-side Bluetooth receiver. The wireless communication unit 23 serves to deliver the stored acceleration information to the server side and to receive a server side command.

The infrared transmitter 24 performs a function of transmitting an infrared signal for location tracking. The structure of the infrared transmitter 24 will be described in more detail with reference to FIG. 3.

3 shows the attachment state of the light emitting diode of the infrared transmitter.

Infrared light emitting diodes are implemented so that the infrared camera and the game controller can be recognized with a probability of more than 98% when the position tracking is performed with a camera having a resolution of 640 horizontal and 480 vertically from 1 meter to 10 meters apart. In the present invention, the light emitting diode may be composed of four or more. As shown in FIG. 3, the four light emitting diodes 32 are arranged not to be located on one plane. In other words, the light emitting diodes are installed on the upper portions of the supports having different heights, and are arranged not to be located on the same plane. 3 and 31, the power supply terminal is connected.

Meanwhile, since the infrared rays emitted from the light emitting diodes are emitted only in a funnel shape of about 10 degrees in the direction toward which the light emitting diodes are directed, if the infrared camera is located outside the infrared rays cannot be detected. In addition, since the amount of light emitted toward the center toward which the light emitting diode is directed increases, when the light emitting diode is precisely directed toward the infrared camera, the image received by the camera turns white. This makes it impossible to track the exact location of the light emitting diode.

In order to solve this problem, in the present invention, the light emitting diode is wrapped with a thin paper tape so that infrared light emitted from the light emitting diode is scattered while passing through the paper and spread in all directions. As a result, the infrared camera receives the same amount of infrared light in which direction the light emitting diode is directed, thereby enabling accurate location tracking of which direction the light emitting diode is directed.

4 is a block diagram of a game server according to the present invention, wherein the game server includes an infrared camera 41, a position tracking unit 42, a wireless communication unit 43, a preprocessor 44, and a block extraction unit. 45, an operation determining unit 47, and a model learning performing unit 46.

The infrared camera 41 can use a general video camera equipped with an infrared filter in front of the lens. By attaching an infrared filter in front of the camera lens, all light in the visible region is blocked and only the infrared light is transmitted to the camera sensor (CCD or CMOS). The infrared camera can be connected to a game server via USB or IEEE1394 or a camera-specific capture board.

The position tracking unit 42 tracks the position of the game controller using the image input from the infrared camera 41. A detailed operation of the position tracking unit 42 will be described with reference to FIG. 5.

First, the positional relationship between the internal parameters of the infrared camera and the four light emitting diodes is measured for position tracking. This is done at the program development stage. The camera's internal parameters are unique to the camera, including the camera's focal length, image center, resolution, and screen distortion. The reason for measuring these properties is that camera-specific properties influence the extraction of three-dimensional positions. These intrinsic properties are fixed as long as the camera is not changed once measured, so you only need to measure it once. This parameter measurement process is collectively called the camera calibration process.

In order to track the three-dimensional position of the light emitting diode, the camera first extracts four positions of the light emitting diode on the two-dimensional screen for each camera frame. To this end, the following process is performed every frame.

First, the position tracking unit 42 removes noise on the screen by applying a Gaussian filter to the original image input from the infrared camera 41 (501 and 502). Gaussian filters help to blur the image and cancel out point noise in small areas.

The position tracking unit 42 generates a binary image by applying a threshold function to the entire image (503). In this process, the threshold value and the camera's sensitivity change according to the number of light emitting diodes found. In addition, bleeding of the light emitting diode is sometimes caused by abnormal infrared light, and the camera sensitivity is adjusted to prevent this.

The position tracking unit 42 then calculates a contour curve for the obtained binary image (504). Calculating the contour curve is a process of obtaining a closed outline for each distant area of the binary image. To this end, a Scan Line Contour Finding algorithm, which is an algorithm that traces a change point of a binary image and obtains a contour while following a scan line of an image, is used. The center of gravity of the found contour is the position of the light emitting diode. Through this process, two-dimensional coordinates of four light emitting diodes may be obtained (505).

The three-dimensional position and the direction of the game controller are calculated using the four two-dimensional positions thus obtained (506). The method for calculating the three-dimensional position and orientation using the two-dimensional position uses a POSIT (see US Patent No. 5,388,059) algorithm. This algorithm tracks the two-dimensional positions of four or more markers whose position relationships are known in advance, and tracks the three-dimensional positions and directions of the markers in real time.

The wireless communication unit 43 receives the acceleration information from the game controller. The preprocessor 44 removes the acceleration component due to gravity acceleration from the acceleration information transmitted from the wireless communication unit 43, and removes the noise using the low frequency filter.

The block extractor 45 extracts data of each operation region based on the absolute magnitude of the acceleration information input from the preprocessor 44 and the amount of change over time.

The model learning performer 46 collects a plurality of samples of the operations to be recognized and learns a model for each operation by using the samples. That is, the model learning performer 46 hides the moving mark using the absolute position data tracked by the position tracking unit 42 and the acceleration data of the corresponding operation region after the motion region extraction of the acceleration data is performed by the block extractor. Learn the model (HMM).

In addition, the motion determiner 47 may compare the position information input from the position tracker 42 with each model trained by the model learning performer 46 with respect to the motion region data input from the block extractor 45. It is calculated to recognize the behavior of the model with the highest similarity.

In the present invention, motion recognition uses a Hidden Markov Model. This is to model each motion as a hidden Markov model based on the acceleration information and the location information and perform motion recognition based on the learned model.

Referring to Figure 6 described in detail the motion recognition process according to the present invention as follows.

The acceleration information transmitted from the gay controller has components other than the acceleration indicated by the user's motion by the gravity acceleration. This is a component that appears by the direction of the game controller and must be removed for accurate motion recognition. The preprocessor removes the acceleration component due to gravity acceleration by using an offset filter. In addition, the acceleration information from the sensor due to the shaking of the user or the noise of the sensor has a high frequency noise, and the filter removes the noise (601).

In the next block extraction process, the preprocessed acceleration information is a signal that is continuously input over time and needs to be cut into a time domain corresponding to each operation. In the block extraction process, data of each operation region is extracted based on the absolute size of the acceleration information and the amount of change over time. That is, if the interval between blocks is shorter than the reference, the blocks are merged. If the size of the blocks is smaller than the reference, the blocks are discarded.

The motion determiner calculates the similarity by substituting the extracted acceleration data and the absolute position data into the learned motion models in operation 603. The operation determiner corrects the weight (604). Weight correction is a part of adjusting the possibility of selecting a specific motion according to the intention of the manufacturer. Even without this weight correction process, it is not difficult to determine the motion, but the result of motion recognition on the input data of the boundary portion of each motion can be adjusted. Finally, the motion determiner determines the motion recognition as the motion having the greatest similarity with respect to each motion model of the input data (605).

Here, a problem arises because the periods for acquiring the position data and the acceleration data of the operation region are different from each other. In the present invention, the problem is solved by merging the acceleration data based on the absolute position data having a long update period.

On the other hand, the method of the present invention as described above can be written in a computer program. And the code and code segments constituting the program can be easily inferred by a computer programmer in the art. In addition, the written program is stored in a computer-readable recording medium (information storage medium), and read and executed by a computer to implement the method of the present invention. The recording medium may include any type of computer readable recording medium.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

1 is a block diagram of a game system to which the present invention is applied;

2 is a block diagram of a game control device according to the present invention;

3 is a view showing the installation structure of the infrared light emitting diode according to the present invention;

4 is a block diagram of a game server according to the present invention;

5 is a flowchart illustrating a location tracking method according to the present invention;

6 is a flowchart illustrating a method of determining a user operation according to the present invention.

Claims (19)

In the game control device for wireless communication with the game server, Acceleration tracking means for generating acceleration data; Infrared transmitting means for transmitting an infrared signal to an infrared camera mounted to the game server; Wireless communication means for wireless communication with the game server; And Central processing means for transmitting the acceleration data generated by the acceleration tracking means to the server through the wireless communication means, and processing the input signal Game control device comprising a. The method of claim 1, The infrared transmission means, The game control device comprising a plurality of light emitting diodes, each light emitting diode is installed on the support having a different height so as not to be located on the same plane. The method of claim 2, The light emitting diode is a game control device, characterized in that the infrared rays are transmitted so as to be scattered. A game server in wireless communication with a game controller, An infrared camera for receiving an infrared signal from the game controller; Wireless communication means for receiving acceleration data from the game controller; Position tracking means for generating an image obtained by the infrared camera as a binary image, calculating contour curves for the binary image, and extracting three-dimensional position data; Preprocessing means for removing acceleration components due to gravity acceleration and removing noise from the acceleration data received through the wireless communication means; Block extraction means for extracting acceleration data of each operation region from the acceleration data preprocessed by the preprocessing means; Model learning performing means for learning a hidden Markov model using the position data extracted by the position tracking means and the acceleration data of each operation region extracted by the block extracting means; And Motion determining means for determining an operation by calculating a similarity between the position data extracted by the position tracking means and the acceleration data of each motion region extracted by the block extracting means and the model trained by the model learning performing means; Game server comprising a. The method of claim 4, wherein The position tracking means, Game server, characterized in that to remove the noise on the screen using a Gaussian filter, to generate a binary image. The method of claim 5, wherein The position tracking means, And a binary image is generated by applying a limit function to the entire image. The method of claim 5, wherein The position tracking means, The game server, characterized by calculating the contour curve by tracking the change point of the binary image while following the scanning line of the image. The method of claim 4, wherein The pretreatment means, The game server, characterized in that to remove the acceleration component due to gravity acceleration using the offset filter. The method of claim 4, wherein The block extracting means, Game server, characterized in that for extracting the acceleration data of each motion area based on the absolute size of the acceleration data and the amount of change over time. The method of claim 4, wherein The operation determining means, And assigning position data and acceleration data of each motion region to each motion model learned by the model learning performing means to measure similarity and determine motion recognition as the motion having the largest similarity. The method of claim 10, The operation determining means, And correcting the weight so as to determine an operation by the corrected weight. In the user gesture recognition method in the game server, (a) generating an image obtained by an infrared camera as a binary image, calculating contour curves for the binary image, and extracting three-dimensional position data; (b) removing the acceleration component due to gravity acceleration from the acceleration data received from the game controller, and then extracting acceleration data of each motion region; (c) training a hidden Markov model using the extracted position data and the acceleration data of each of the extracted motion regions; And (d) substituting the model trained in step (c) for the extracted position data and the acceleration data of each of the extracted motion regions to determine an operation through similarity calculation Method of determining a user action in a game server comprising a. The method of claim 12, In step (a), Removing noise from an image obtained by the infrared camera; Generating a binary image by applying a limit function to the entire image; Calculating a contour curve for the generated binary image; And Calculating a three-dimensional position direction using the two-dimensional coordinates obtained by the calculated contour curve Method of determining a user action in a game server comprising a. The method of claim 13, Removing noise from the image is performed using a Gaussian filter. The method of claim 13, The contour curve is, And a change point of the generated binary image is calculated by following the scanning line of the image. The method of claim 12, And removing the acceleration component due to gravity acceleration from the acceleration data received from the game controller in the step (b). The method of claim 12, In the step (b), the acceleration data of each operation region is A method for determining a user's motion in a game server, characterized in that it is extracted based on the absolute amount of acceleration data from which acceleration components are removed by gravity acceleration and the amount of change over time. The method of claim 12, In step (d), The user's motion in the game server, characterized in that the degree of similarity is measured by substituting the extracted position data and the acceleration data of each extracted motion region into the learned model, and determining the motion recognition as the motion having the largest similarity. How to decide. The method of claim 18, In step (d), And determining the motion by weight correction after measuring the similarity.

Images