JP5726811B2 - Method and apparatus for use in determining lack of user activity, determining user activity level, and / or adding a new player to the system - Google Patents

Description

  The present invention relates generally to an interface between a person and a computer, and more specifically to processing multi-channel inputs to track user operations of one or more controllers.

  Computer entertainment systems typically include a handheld controller, game controller or other controller. The user or player uses the controller to send commands or other instructions to the entertainment system to control the video game or other simulation being played. For example, the controller may include a manipulator (for example, a joystick) operated by the user. The manipulated variable of the joystick is converted from an analog value to a digital value and sent to the main frame of the game machine. The controller may include a button that can be operated by the user.

  With respect to these information and other background information, the present invention has been devised.

  One embodiment provides a method for determining a lack of user action associated with a gaming device that is used during operation of the gaming device. The method includes obtaining information that can be used to determine user actions relating to a predetermined object, and processing the information to determine whether a user is currently active with respect to the predetermined object. And providing a first signal to the game device for use in controlling operation of the game device when it is determined in the processing step that the user is not currently active.

  Another embodiment provides a method for determining a lack of user action associated with a gaming device that is used during operation of the gaming device. The method includes obtaining information that can be used to determine an activity level of a user of the gaming device, processing the information to determine whether the user is not currently active, Providing a first signal to the game device for use in controlling the operation of the game device when it is determined that it is not currently active.

  Another embodiment provides a method for use in operating a gaming device. The method includes obtaining information that can be used to determine the activity of a plurality of predetermined objects used to control the gaming device, and processing the information to determine a plurality of predetermined objects during the game. In response to determining whether one of the selected objects is inactive and determining that one of the plurality of predetermined objects is currently active after determining that one of the plurality of predetermined objects is not active. Adding a new player to the ongoing game.

  Another embodiment provides a system for use in operating a gaming device. The system includes means for obtaining information that can be used to determine the activity of a plurality of predetermined objects used to control the game device, and processing the information to determine a plurality of predetermined objects while the game is in progress. In response to a means for determining whether one of the selected objects is inactive and a determination that one of the plurality of predetermined objects is currently active after determining that one of the predetermined objects is not active. And means for adding a new player to the ongoing game.

  The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

1 is a schematic diagram illustrating a video game system operating in accordance with an embodiment of the present invention. FIG. 2 is a perspective view of a controller made in accordance with one embodiment of the present invention. FIG. 3 is a three-dimensional schematic diagram illustrating an accelerometer that can be used in a controller according to an embodiment of the present invention. 1 is a block diagram of a system for mixing various control inputs according to one embodiment of the present invention. FIG. 1 is a block diagram of a system according to an embodiment of the present invention. FIG. 3 is a flow diagram of a method according to an embodiment of the invention. FIG. 3 is a flow diagram of a method according to an embodiment of the invention. 1 is a block diagram illustrating a system that can be used to implement, implement, and / or perform the methods and techniques described herein in accordance with one embodiment of the present invention. FIG. 6 is a block diagram illustrating a processor that can be used to implement, implement, and / or perform the methods and techniques described herein in accordance with an embodiment of the invention. FIG. 5 is a flow diagram illustrating a method according to an embodiment of the present invention. FIG. 5 is a flow diagram illustrating a method according to an embodiment of the present invention.

  While the following detailed description includes many specific details for purposes of explanation, those skilled in the art will recognize that many variations and modifications to the following details are within the scope of the present invention. Let's go. Accordingly, the exemplary embodiments of the invention described below are described without impairing the generality of the claimed invention and without limiting it.

  Various embodiments of the methods, apparatus, schemes and systems described herein provide for detection, capture, and tracking of movement, movement and / or manipulation of the entire controller body by a user. The movement, movement and / or manipulation of the entire controller body by the detected user can be used as additional commands to control various aspects of the game or other simulation being played.

  Detection and tracking of user operations on the game controller main body can be performed in various ways. For example, an inertial sensor such as an accelerometer or gyroscope, or an image acquisition device such as a digital camera can be used with a computer entertainment system to detect the movement of the handheld controller body and convert it into an in-game action be able to. Examples of controller tracking using inertial sensors are described, for example, in US Patent Application No. 11 / 382,033, “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUT CONTROL” (Attorney Docket Number SCEA06INRT1), see Is incorporated herein by reference. An example of tracking a controller using image capture is described, for example, in US patent application Ser. No. 11 / 382,034, “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY” (Attorney Docket Number SCEA05082US00). Is incorporated herein by reference. In addition, the controller and / or user can be tracked acoustically using a microphone array and appropriate signal processing. An example of such acoustic tracking is described in US patent application Ser. No. 11 / 381,721, which is incorporated herein by reference.

  Sound detection, inertia detection and image capture can be used individually or in any combination to detect many different types of movement of the controller. For example, up / down movement, twisting movement, lateral movement, sudden movement, movement like a cane, depression movement, and the like. Such movements may correspond to various commands that are converted into in-game actions. Many different types of games, simulations, etc. can be implemented using detection and tracking of operations by the user of the game controller body. This allows users to fight with a sword or lightsaber, trace the shape of an item with a cane, participate in many different types of sports competitions, engage in battles on the screen or other encounters, etc. The game program can be configured to track movement and recognize a pre-recorded specific gesture from the tracked movement. Recognition of one or more of these gestures can cause a change in game state.

  In an embodiment of the present invention, controller path information obtained from these different sources can be mixed before analyzing gesture recognition. Tracking data from different sources (eg, sound, inertia, and image capture) can be mixed to improve the likelihood of gesture recognition.

  Referring to FIG. 1A, a system 100 that operates according to one embodiment of the present invention is shown. As shown, the computer entertainment console 102 can be connected to a television or other video display 104 to display images of a video game or other simulation. The game or other simulation may be stored on a DVD, CD, flash memory, USB memory, or other memory medium 106 inserted into the console 102. A user or player 108 operates the game controller 110 to control a video game or other simulation. As shown in FIG. 2, the game controller 110 includes an inertial sensor 112 that generates a signal corresponding to the position, movement, direction, or change in direction of the controller 110. In addition to the inertial sensor, the game controller 110 may include a conventional control input device such as a joystick 111, buttons 113, Rl, Ll, and the like.

  In operation, the user 108 physically moves the controller 110. For example, the controller 110 can be moved by the user 108 in any direction, that is, up and down, left and right, twisting, rotation, shaking, sudden movement, and depression. These movements of the controller 110 itself can be detected and acquired by the camera 112 by tracking by signal analysis from the inertial sensor 112 in the manner described below.

  Referring again to FIG. 1, the system 100 may optionally include a camera or other image acquisition device 114. These can be positioned such that the controller 110 is within the field of view 116 of the camera. Analysis of the image from the image acquisition device 114 can be used along with analysis of data from the inertial sensor 112. As shown in FIG. 2, the controller 110 may optionally include a light source such as light emitting diodes (LEDs) 202, 204, 206, 208 to facilitate tracking through video analysis. These may be attached to the main body of the controller 110. In this specification, the term “main body” means a part of the game controller 110 that a person holds (or wears in the case of a wearable game controller) by hand.

  Video image tracking for tracking the controller 110 is described in US patent application Ser. No. 11 / 382,034, “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY” filed May 6, 2006. SCEA05082US00), which is incorporated herein by reference. The console 110 may include an acoustic transducer such as a microphone array 118. The controller 110 may include an acoustic signal generator 210 (eg, a speaker) that provides a sound source to facilitate acoustic tracking of the controller 110 using the microphone array 118 and appropriate acoustic signal processing. This is described in US patent application Ser. No. 11 / 381,724, which is incorporated herein by reference.

  Generally, signals from the inertial sensor 112 are used to generate position data and direction data for the controller 110. Such data can calculate many physical situations of the movement of the controller 110. For example, the acceleration and velocity of the controller along any axis, its tilt, pitch, yaw, roll, and any telemetry point of the controller 110. As used herein, telemetry generally refers to remotely measuring and reporting information of interest to a system or system designer or operator.

  The ability to detect and track the movement of the controller 110 makes it possible to determine whether any predefined movement of the controller 110 is performed. That is, a specific movement pattern or gesture of the controller 110 can be predefined and used as an input command for a game or other simulation. For example, a gesture for dropping the controller 110 can be defined as one command, a gesture for twisting the controller 110 can be defined as another command, and a gesture for shaking the controller 110 can be defined as another command. Etc. Thus, the manner in which the user 108 physically moves the controller 110 is used as another input for controlling the game. This provides the user with a more exciting and interesting experience.

  As an example, the inertial sensor 112 may be an accelerometer, but is not limited thereto. FIG. 3 shows an example of an accelerometer 300 in the form of a single mass 302 that is elastically coupled to the frame 304 by springs 306, 308, 310, 312, for example. The pitch axis and roll axis (represented by X and Y, respectively) lie in a plane across the frame. The yaw axis Z faces in a direction perpendicular to a plane including the pitch axis X and the roll axis Y. The frame 304 can be attached to the controller 110 in any suitable form. As frame 304 (and joystick controller 110) accelerates and / or rotates, mass 302 displaces relative to frame 304, and the amount and direction of translational acceleration and / or rotational acceleration and / or pitch, roll and / or yaw. The springs 306, 308, 310, 312 expand and contract according to the angle. Displacement of the mass 302 and / or expansion and contraction of the springs 306, 308, 310, 312 is sensed using, for example, suitable sensors 314, 316, 318, 320, and known methods depending on the pitch and / or roll amount, acceleration. Or converted into a signal in a determinable manner.

  There are many different ways to track mass position and / or force on mass, including resistive strain gauge materials, photonic sensors, magnetic sensors, Hall effect devices, piezoelectric devices, capacitive sensors, etc. . Embodiments of the present invention can include any number and type of sensors, or combinations of multiple types of sensors. As an example, the sensors 314, 316, 318, 320 may be, but are not limited to, gap-closed electrodes positioned above the mass 302. As the position of the mass with respect to each electrode changes, the capacitance between the mass and each electrode changes. Each electrode is connected to a circuit that generates a signal related to the capacitance (ie, proximity) of the mass 302 to the electrode. Further, the springs 306, 308, 310, 312 may include a resistance strain gauge sensor that generates a signal relating to the expansion and contraction of the spring.

  In some embodiments, the frame 304 may be a gimbal attached to the controller 110 such that the accelerometer 300 maintains a fixed orientation relative to the pitch, roll and / or yaw axis. According to this method, the X, Y, and Z axes of the controller can be directly mapped to the corresponding axes in the real space without having to consider the inclination of the controller axis with respect to the coordinate axes in the real space.

  As described above, data from inertial sources, image capture sources, and acoustic sources can be analyzed to generate a path that tracks the position and / or orientation of the controller 110. As shown in the block diagram of FIG. 4, a system 400 according to one embodiment of the invention may include an inertial analyzer 402, an image analyzer 404, and an acoustic analyzer 406. Each of these analyzers receives a signal from the environment 401 to be detected. The analyzers 402, 406, 408 can be implemented in hardware, software (or firmware), or a combination of two or more of these. Each analyzer generates tracking information regarding the position and / or orientation of the target object. As an example, the target object may be the controller 110 described above. The image analyzer 404 operates in conjunction with the following fields to form a field and relates to the method described in US patent application Ser. No. 11 / 382,034 (Attorney Docket No. 86321 SCEA05082US00). The inertial analyzer 402 operates with the following fields to form a field, which is described in “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUT CONTROL” (Attorney Docket No. SCEA06INRT1) of US Patent Application No. 11 / 382,033. Related to the method. The acoustic analyzer 406 operates in conjunction with the following fields to form a field and is related to the method described in US patent application Ser. No. 11 / 381,724.

  The analyzers 402, 404, 406 may be considered to be associated with input channels with different position information and / or direction information. The mixer 408 can accept multiple input channels. This channel typically includes sample data that characterizes the sensed environment 401 from the channel's field of view. Position and / or direction information generated by inertial analyzer 402, image analyzer 404 and acoustic analyzer 406 can be coupled to the input of mixer 408. The mixer 408 and the analyzers 402, 404, 406 may be configured to be queried by the game software 410 and interrupt the game software in response to an event. Events include gesture recognition events, gearing changes, setting changes, setting noise levels, setting sampling rates, mapping chain changes, and the like. These examples will be described later. The mixer 408 operates in conjunction with the following fields to form fields and is associated with the methods described herein.

  As described above, signals from different input channels such as inertial sensors, video images and / or acoustic sensors, for example, are analyzed by inertial analyzer 402, image analyzer 404, and acoustic analyzer 406, respectively, according to the method of the present invention. The motion and / or direction of the controller 110 can be determined during video game play. Such a method may be implemented as a series of processor-executable program code instructions stored on a processor-readable medium and executed by a digital processor. For example, as shown in FIG. 5A, the video game system 100 may include a console 102 including an inertial analyzer 402, an image analyzer 404, and an acoustic analyzer 406, each implemented in hardware or software. As an example, analyzers 402, 404, 406 can be implemented as software instructions running on a suitable processor unit 502. As an example, the processor unit 502 may be a digital processor such as, for example, a type of microprocessor commonly used in video game machines. A portion of the instructions may be stored in the memory 506. Alternatively, inertial analyzer 402, image analyzer 404, and acoustic analyzer 406 may be implemented in hardware as, for example, an application specific integrated circuit (ASIC). Such analyzer hardware may be located on the controller 110 or console 102 or may be located remotely elsewhere. In a hardware implementation, the analyzers 402, 404, 406 may be programmable in response to external signals from, for example, the processor 502 or from other remotely located sources such as USB cables, wireless connections, networks, etc. Good.

  Inertial analyzer 402 may include or implement instructions that analyze signals generated by inertial sensor 112 and utilize information regarding the position and / or orientation of controller 110. Similarly, the image analyzer 404 may implement instructions for analyzing an image captured by the image acquisition device 114. Further, the acoustic analyzer may implement instructions for analyzing the image captured by the microphone array 118. These signals and / or images may be received by analyzers 402, 404, 406 as shown in block 512, as shown in flow diagram 510 of FIG. 5B. As indicated by block 514, the signals and / or images are analyzed by analyzers 402, 404, 406 to determine inertial tracking information 403, image tracking information 405, and acoustic tracking information 407 regarding the position and / or orientation of controller 110. May be. The tracking information 403, 405, 407 may be associated with one or more degrees of freedom. Preferably 6 degrees of freedom are tracked to characterize the operation of the controller 110 or other tracked object. This degree of freedom may relate to controller tilt, yaw, roll and position, velocity or acceleration along the x, y, and z axes.

  As indicated by block 516, the mixer 408 mixes the inertial information 403, the image information 405, and the acoustic information 407 to generate improved position and / or direction information 409. As an example, the mixer 408 may take a weighted average by applying different weights to the inertia information 403, the image information 405, and the acoustic information 407 based on the game situation or the environmental situation. Further, the mixer 408 includes its own mixer analyzer 412 that analyzes the combined position and / or orientation information, and obtains its own “mixer” information that includes a combination of information generated by other analyzers. It may be generated.

  In an embodiment of the invention, the mixer 408 may assign distribution values to the tracking information 403, 405, 407 from the analyzers 402, 404, 406. As described above, a particular set of input control data may be averaged. However, in this embodiment, values are assigned to the input control data before averaging, so that the importance of analysis of the input control data from some analyzers may be higher than that from other analyzers. Good.

  The mixer 408 may assume several functions in the context of other processing and analysis observations, corrections, stabilization, differentiation, combination, routing, mixing, reporting, buffering, and interruptions. This may be performed for the tracking information 403, 405, 407 received from one or more of the analyzers 402, 404, 406. The mixer 408 is implemented to optimize use of the received tracking information 403, 405, 407 and generate improved tracking information 409, while the analyzers 402, 404, 406 each receive or retrieve specific tracking information. It may be.

  The analyzers 402, 404, 406 and the mixer 408 are preferably configured to provide tracking information in a similar output format. Tracking information parameters from any analyzer element 402, 404, 406 can be mapped to a single parameter in the analyzer. Instead, the mixer 408 generates tracking information for any of the analyzers 402, 404, 406 by processing one or more tracking information parameters from the one or more analyzers 402, 404, 406. be able to. The mixer combines two or more elements of tracking information of the same parameter type obtained from the analyzers 402, 404, 406, performs functions across multiple tracking information parameters generated by the analyzer, and receives multiple inputs. A composite set of outputs can be created with the benefits generated from the channels.

  As indicated by block 518, improved tracking information 409 may be utilized during video game play on the system 100. In certain embodiments, location information and / or direction information may be used in connection with gestures made by the user 108 during game play. In some embodiments, the mixer 408 operates in conjunction with the gesture recognizer 505 to at least one action in the gaming environment and one or more user actions from the user (eg, operating the controller in space). Can be linked.

  In the flow diagram 520 of FIG. 5C, the path of the controller 110 may be tracked using position information and / or direction information, as indicated by block 522. As an example, the path may include, but is not limited to, a set of points that represent the position of the center of mass of the controller with respect to several coordinate systems. Each position point can be represented by one or more coordinates such as X, Y, and Z coordinates in a Cartesian coordinate system. Time may be associated with each point on the path so that both the shape of the path and the progress of the controller along the path can be monitored. Furthermore, each point in the set may be associated with data representing the direction of the controller (eg, one or more rotation angles around the center of mass of the controller). Further, each point on the path may be associated with the value of the velocity and acceleration of the center of mass of the controller, the angular rotation rate, and the value of the angular acceleration of the controller around the center of mass.

  The tracked path may also be compared to one or more stored paths corresponding to known and / or pre-recorded gestures 508 associated with the context of the video game being played, as indicated by block 524. Good. The recognition device 505 may be configured to recognize a user or process a voice authenticated gesture or the like. For example, the recognition device 505 can identify a user by a gesture. The gesture may be user specific. Such unique gestures are recorded and included in pre-recorded gestures 508 stored in memory 506. The recording process may selectively save a sound generated during gesture recognition. The detected environment is sampled and processed by a multi-channel analyzer. The processor can refer to the gesture model to determine, authenticate and / or identify the user or object with high accuracy and high performance based on the speech or acoustic pattern.

  As shown in FIG. 5A, data 508 representing a gesture may be stored in the memory 506. Examples of gestures include throwing objects such as balls, swinging objects such as bats or golf clubs, moving hand pumps, opening or closing doors or windows, turning handles or other vehicle control devices, and punching Martial arts movement, sanding movement, waxing, painting house, shaking, making sound, turning, throwing football, turning knob, 3D mouse movement, scrolling movement, movement by profile, optional Recordable movement, movement back and forth along any vector in any direction in the space to inflate the tire, movement along the path, movement with precise stop and start, recorded in the noise floor , Tracking, any repeatable time-based user operation, spline, etc., but not limited to. Each of these gestures is pre-recorded from the path data and can be recorded as a time-based model.

  The comparison between the path and the stored gesture can begin with a steady state assumption that if the path goes out of steady state, the deletion process can compare the path with the stored gesture. If there is no match at block 526, the analyzer 504 can continue to track the path of the controller 110 at block 522. If the path (or part of it) and the stored gesture sufficiently match, the game state can be changed as shown at 528. The game state change includes, but is not limited to, interruption, transmission of a control signal, change of a variable, and the like.

  An example of this is shown. If it is determined that the controller 110 has left a steady state, the analyzer 402, 404, 406 or 412 follows the movement of the controller 110. As long as the path of the controller 110 follows the path defined by the stored gesture model 508, these gestures are possible “hits”. If the controller 110 path deviates from any gesture model 508 (within the noise tolerance setting), that gesture model is removed from the hit list. Each gesture reference model includes a time base in which gestures are recorded. The analyzer 402, 404, 406, or 412 compares the controller path data with the stored gesture 508 at the appropriate time index. When the steady state condition occurs, the clock is reset. When out of steady state (ie, movement is tracked outside the noise threshold), the hit list is populated with all potential gesture models. The clock is started and the controller movement is compared against the hit list. The comparison is a walk through time. When any gesture in the hit list reaches the end of the gesture, it is a hit.

  In certain embodiments, the mixer 408 and / or individual analyzers 402, 404, 406, 412 notify the game program when certain events occur. Examples of such events include:

“Interruption of zero acceleration point reached (X, Y and / or Z axis)”
In certain game situations, if the controller acceleration changes at the inflection point, the analyzer 504 notifies the routine in the game program or interrupts the routine. For example, the user 108 may use the controller 110 to control a game avatar representing a quarterback in a football simulation game. The analyzer 504 may track the controller (representing football) through a path generated from the signal obtained from the inertial sensor 112. A specific change in the acceleration of the controller 110 may signal a football release. At this point, the analyzer may invoke another routine in the program (eg, a physics simulation package) to simulate a football trajectory based on the controller's position, velocity and / or direction at the release point.

"Interruption of a recognized new gesture"

  Further, the analyzer 502 may be configured with one or more inputs. Examples of such inputs include, but are not limited to:

“Setting noise level (X, Y or Z axis)”
The noise level may be a reference tolerance used when analyzing irritation (jitter) of the user's hand in the game.

“Setting sampling rate”
In this specification, the sampling rate refers to the frequency with which the analyzer 502 samples the signal from the inertial sensor. The sampling rate may be set to oversampling or averaging of the signal.

"Setting gearing"
As used herein, gearing generally refers to the ratio of controller movement to movement that occurs within a game. Examples of such "gearing" in the context of video game control can be found in U.S. Patent Applications Nos. 11 / 382,036, 11 / 382,041, 11 / 382,038 and 11 / 382,040, all Is incorporated herein by reference.

Configuration mapping chain
In this specification, the mapping chain refers to a map of a gesture model. A gesture model map can be created for a particular input channel (eg, path data originating only from inertial sensor signals) or for a hybrid channel formed with a mixer unit. Three input channels may be provided by two or more different analyzers similar to inertial analyzer 504. Specifically, these may include: an inertial analyzer 504 as described herein, eg, “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY” in US patent application Ser. No. 11 / 382,034. Personnel number SCEA05082US00) (incorporated herein by reference) and a video analyzer, for example as described in US patent application Ser. No. 11 / 381,721 (incorporated herein by reference). . The analyzer can be configured with a mapping chain. The mapping chain can be exchanged by the game during game play as a setting for the analyzer and mixer.

  Referring back to block 502 of FIG. 5A, those skilled in the art will recognize that there are numerous ways to generate a signal from inertial sensor 112. Some examples are described above with respect to FIGS. 3A-3E. Referring to block 504, there are a number of ways to analyze the sensor signal generated at block 502 to obtain information regarding the position and / or orientation of the controller 110. As an example, tracking information includes, but is not limited to, information about individual or any combination of the following parameters:

"Controller direction"
The direction of the controller 110 may be expressed by a pitch, a roll, or a yaw angle (for example, radians) with respect to some reference directions. The rate of change in the controller direction (eg, angular velocity or angular acceleration) may be included in the position information and / or direction information. If the inertial sensor 112 is a gyroscope sensor, the controller direction information may be obtained directly in the form of one or more output values proportional to the pitch, roll or yaw angle.

“Controller position” (eg, Cartesian coordinates X, Y, Z of controller 110 within some reference systems)
"Controller X-axis speed"
"Controller Y-axis speed"
"Controller Z-axis speed"
"Controller X-axis acceleration"
“Controller Y-axis acceleration”
"Controller Z-axis acceleration"

  Regarding position, velocity, and acceleration, position information and / or direction information may be expressed in a coordinate system other than the Cartesian coordinate system. For example, cylindrical coordinates or spherical coordinates may be used for position, velocity and acceleration. Acceleration information about the X axis, Y axis, and Z axis may be obtained directly from an accelerometer type sensor as described above with respect to FIGS. 3A-3E. X, Y, Z acceleration may be integrated with respect to time from several initial points to determine changes in X, Y, Z velocity. These velocities may be calculated by adding speed changes to the known values of the X, Y and Z velocities at the initial time. The X, Y, and Z velocities may be integrated with respect to time to determine the X, Y, and Z displacement of the controller. The X, Y, and Z positions may be obtained by adding displacement to the known X, Y, and Z positions at the initial time point.

"Steady state Y / N"
This specific information indicates whether or not the controller is in a steady state. The steady state can be defined as an arbitrary position and is subject to change. In a preferred embodiment, the steady state position may be that the controller is held approximately horizontally at approximately the same height as the user's waist.

“Time since last steady state” generally refers to data relating to the length of the period of time since the last steady state (described above) was detected. As described above, the time determination may be calculated in real time, in processor cycles, or in a sampling period. The “time from last steady state” data time is important in terms of resetting controller tracking to the initial point and ensuring the accuracy of character or object mapping in the gaming environment. This data is also important for determining the available actions / gestures (both exclusive and comprehensive) that can continue to be performed in the gaming environment.

The “recognized last gesture” generally refers to the last gesture recognized by a gesture recognition device 505 that can be implemented in hardware or software. The identification of the last recognized gesture is important with respect to the fact that the previous gesture is associated with a gesture that may subsequently be recognized, or some other action that occurs within the gaming environment.

"Last gesture time recognized"
The above output can be sampled at any time by a game program or software.

  In an embodiment of the invention, the mixer 408 may assign distribution values to the tracking information 403, 405, 407 from the analyzers 402, 404, 406. As described above, a particular set of input control data may be averaged. However, in this embodiment, values may be assigned to the input control values before averaging, thereby increasing the importance of analysis of input control data from some analyzers than from other analyzers. .

  For example, the mixer 408 may require tracking information related to acceleration and steady state. The mixer 408 receives the tracking information 403, 405, 407 as described above. For example, as described above, the tracking information may include parameters related to acceleration and steady state. Prior to averaging the data representing this information, the mixer 408 may assign distribution values to the tracking information data sets 403, 405, 407. For example, the x and y acceleration parameters from the inertia analyzer 402 may be weighted by 90%. However, the x and y acceleration data from the image analyzer 404 may be weighted only by 10%. Regarding the acceleration parameter, the tracking information 407 of the acoustic analyzer may be weighted by 0%, that is, the data may have no value.

  Similarly, the Z-axis tracking information parameter from the inertia analyzer 402 may be weighted by 10%, and the Z-axis tracking information of the image analyzer may be weighted by 90%. The acoustic analyzer tracking information 407 may be weighted 0%, the steady state tracking information from the acoustic analyzer 406 may be weighted 100%, and the remaining analyzer tracking information may be weighted 0%. .

  After the appropriate distribution weights are assigned, the input control data is averaged together with the weights to obtain a weighted average input control data set, which is then analyzed by the gesture recognition device 505 for a specific game environment. It may be associated with an action. The associated value may be predefined by the mixer 408 or a particular game title. This value may be the result of a mixer 408 that identifies the specific quality of the data coming from the various analyzers, thereby making dynamic adjustments as described below. This adjustment may be the result of building a historical knowledge base when specific data becomes a specific value in a specific environment and / or depending on the particularity of a given game title.

  The mixer 408 may be configured to operate dynamically during game play. For example, when the mixer 408 receives various input control data, it recognizes that the particular data is consistently outside the acceptable data range or data quality, or indicates error handling at the associated input device Incorrect data can be reflected.

  In addition, certain situations in the real world environment may change. For example, natural light in the game environment of the user's home may increase from morning to afternoon, causing problems in capturing image data. Furthermore, the neighborhood or home becomes louder during the daytime, causing problems in capturing audio data. Similarly, if the user continues to play for several hours, the user's quick reflection may be reduced, causing problems in the interpretation of inertial data.

  In these instances, or where the quality of a particular form of input control data is an issue, the mixer 408 may remove from a particular device to give higher or lower importance to the particular input control data as described above. Weights can be dynamically reassigned to a specific set of incoming data. Similarly, the game environment may change as the game progresses. The need for a particular game will change, thus requiring the reassignment of specific input control data values or needs.

  Similarly, the mixer 408 may process certain data being passed to the gesture recognizer 505 incorrectly or slowly, or may be processed based on errors being processed or feedback data generated by the gesture recognizer 505. Can recognize that not. In response to this feedback or upon recognition of these in-process issues (eg, an error occurs when correlated by the gesture recognizer 505 if the image analysis data is within an acceptable range), the mixer 408 It is possible to adjust what input control data is obtained from which analyzer at which point. The mixer 408 may require specific analysis and processing of the input control data by an appropriate analyzer before the input control data is passed to the mixer 408. The mixer reprocesses the data so that efficient and proper processing of the data passed to the gesture recognizer 505 is further ensured.

  In some embodiments, the mixer 408 recognizes that the particular data is unreliable, invalid, or outside the particular variable, and calls the particular input control data or a variable associated with that data. Can replace incorrect data or calculate specific data on the necessary variables appropriately.

  In general, with respect to the steady state conditions described above, when nothing happens, the game may optionally have the ability not to simply wait for input. Instead, the game may positively determine that the user is not currently active and generates a signal that controls the operation of the game device. For example, when it is determined that the user is not currently active, the operation of the game device can be paused. For example, the game device may react to a first signal for pausing execution of the game. If there is activity, it may be determined positively and a signal may be sent to the game. The activity may be an active action or a certain amount of action. For example, the activity may be a motion of swinging the arm, or the degree of pulling the arm back before swinging the arm. However, this is just one example.

  In some embodiments, the system may be equipped with the ability to determine steady state conditions and / or detect exit from steady state conditions. For example, one embodiment includes a method used to operate a gaming device to determine a lack of user activity on the gaming device. Referring to FIG. 8A, a method 800 according to one embodiment of the present invention is shown. The method 800 may be performed or performed, for example, on many different types of systems and devices such as entertainment systems and consoles, computers, consumer electronics devices, and the like. An example of a system used to perform the method 800 is described below.

  Method 800 includes obtaining a timeline of samples that includes information that can be used to determine user activity for a predetermined object. At step 804, the time series is processed to determine if the user is currently active with respect to the predetermined object. Then, if it is determined in step 806 that the user is not currently active in the time series processing step, a first signal that is used to control the operation of the game device is provided to the game device. In some embodiments, the predetermined object may be a controller, such as a game controller or similar device, a remote control, a cane-like device, a mouse or pointing device, or any other device. In this way, it can be determined that a specific user-operable object such as a controller is in a steady state.

  In some embodiments, it can be determined that a particular user-operable object is in a steady state among a plurality of objects. For example, one or more people who are not currently active among a plurality of users by executing time-series processing steps on the activities of individual users among a plurality of users customary for individual objects among a plurality of predetermined objects Individual users can be identified. Further, the step of providing the first signal may include identifying a particular user that is not currently active among the plurality of users.

  Some embodiments describe returning from steady state. For example, this can be explained by determining that there is activity for a particular user-operable object. For example, when a new player joins a game in progress in this scenario, a time-series processing step can determine whether the user is currently active in relation to a predetermined object. If the time series processing step determines that the user is currently active immediately after determining that the user is not active, a second signal is provided to the game device for use in controlling the operation of the game device. Also good. The step of processing the time series is performed on the activities of the individual users among the plurality of users related to the individual objects of the plurality of predetermined objects, and one or more of the plurality of users currently active Individual users can be identified. Further, providing the second signal may include identifying a particular user currently active among the plurality of users. Once the new player is identified as a particular user, the new player profile may be identified and loaded.

  In some embodiments, the step of processing the time series may determine that the user is not currently active unless the user's activity exceeds a threshold. The threshold may be modified during operation of the game device and during time series processing to determine whether the user activity exceeds the modified threshold. The game device may suspend execution of the game in response to the first signal.

  In some embodiments, the step of processing a time series for an activity of an individual user among a plurality of users related to an individual object among a plurality of predetermined objects is performed, and a current activity among the plurality of users is performed. One or more individual users may be identified. Providing the first signal may include identifying a particular user that is not currently active among the plurality of users.

  In some embodiments, the processing step may include providing a second signal when it is determined that all of the predetermined set of users among the plurality of users are not currently active. The time series may include information regarding a predetermined object motion, position, posture, and a predetermined sound level near the object. The step of acquiring the time series may include acquiring information from an image acquisition device in the vicinity of the screen capable of displaying the progress of the game by the game device.

  In some embodiments, the predetermined object may comprise a body and at least one optically detectable (PD) element that is assembled to the body. The PD element may be distinguishable from the main body by at least one of luminance, color, texture, pattern, reflectance, or illuminance. The time-series acquisition step may include acquiring information representing a position of at least one PD element in the image acquired by the image acquisition device. At least one PD element may be a light source. At least one PD element may be a light emitting diode (LED).

  In some embodiments, the predetermined object is a game controller with a main body, and the main body may have a front portion that faces the screen when the progress of the game is displayed on the screen. The game controller may include at least one input device assembled to the main body, and the input device may be operable by the user to register input from the user. Acquiring a timeline of samples may include capturing the position of the PD element in the acquired image by the image acquisition device when at least a portion of the front of the game controller is facing the screen. . The time series processing step may include determining that the user is not currently active when the movement of the position of the PD element in the image does not exceed a threshold over a predetermined period of time.

  In some embodiments, the game controller may comprise at least two PD elements, and obtaining the time series may include capturing the direction of the game controller relative to the x and y axes of the image plane. The game controller may include three PD elements that form a triangle. Obtaining the time series may further include capturing a direction of the game controller relative to a z-axis perpendicular to the image plane.

  In some embodiments, the game controller may include at least four PD elements that form a square. Obtaining the time series may further include capturing a direction of the game controller relative to a z-axis perpendicular to the image plane.

  In some embodiments, the processing step determines a direction of the game controller that includes direction data relating to at least one of pitch, yaw, and roll, and the user compares the direction data with predetermined direction data. It may include determining whether or not currently active. Direction data may relate to the pitch, yaw and roll of the game controller.

  In some embodiments, obtaining the time series includes sampling information acquired by an inertial sensor embedded in a predetermined object, and processing the time series comprises processing the time series. And determining that the movement of the predetermined object does not exceed the threshold value within a predetermined period. The inertial sensor may be at least one of an accelerometer, a gyroscope, or a laser gyroscope. The step of obtaining the time series includes obtaining information that can be used to obtain the acceleration of the predetermined object, and the step of processing the time series includes obtaining the acceleration of the predetermined object. But you can. The step of processing the time series may include obtaining a predetermined object velocity by integrating acceleration values obtained from the time series over a period of time. The step of processing the time series may include determining a predetermined object displacement by first integrating acceleration values obtained from the time series over a period of time and integrating the result over the period of time.

  In some embodiments, the predetermined object may be a game controller having a game controller having a main body, an inertial sensor attached to the main body, and at least one user input device assembled with the main body. The user input device may be operable by the user to register input from the user.

  In some embodiments, obtaining the time series includes sampling information obtained by the acoustic transducer in the vicinity of the object operated by the user, and processing the time series is performed by the acoustic transducer. It may include determining that the user is not currently active when the acquired information does not meet a predetermined criterion.

  In some embodiments, in the step of displaying the progress of the game on the screen, the acoustic information may be acquired by an acoustic transducer disposed near the screen.

  In some embodiments, the step of obtaining the time series includes sampling the acoustic information obtained by the acoustic transducer in the vicinity of the object operated by the user, and the step of processing the time series comprises: When the information acquired by the above does not satisfy a predetermined criterion specific to a specific user, it may include determining that the specific user among the plurality of users is not currently active.

  In some embodiments, the predetermined criteria include pitch, audio pulse frequency, audio pulse shape, audio pulse duty cycle, pulsation frequency, spectral signature including characteristic sound spectral content characteristics, and characterizing a particular user's audio. A spectral signature may include at least one criterion selected from the group consisting of a time-varying signature that characterizes a series of one or more phonemes that are linked together and can be recognized by speech recognition.

  In some embodiments, the time series processing step determines whether the user is currently active with respect to the predetermined object, and the time series processing step determines that the user is not active. If it is determined immediately after that that the user is currently active, the method may further include providing the game device with a second signal that is used to control operation of the game device.

  In some embodiments, the step of processing the time series may determine that the user is currently active when the user's activity exceeds a threshold. The threshold may be modified while the game device is operating, and the time series may be processed to determine whether the user activity exceeds the modified threshold.

  In some embodiments, the game device may resume game execution in response to the second signal.

  In some embodiments, the step of processing the time series is performed on an activity of an individual user of the plurality of users regarding the individual object of the plurality of predetermined objects, and is currently active among the plurality of users. One or more individual users may be identified, and providing the second signal includes identifying a particular user currently active among the plurality of users.

  In some embodiments, prior to processing the time series and in response to the second signal, a profile corresponding to each user is stored and stored corresponding to the particular user identified by the second signal. One or more profiles may be accessed and the execution of the gaming device may be controlled using profile information obtained from the accessed profiles.

  In some embodiments, a profile corresponding to each user may be saved before the step of processing the time series and in response to the second signal. The previously stored one of the profiles corresponding to the specific user identified by the second signal may be accessed, and the execution of the game device may be controlled using profile information obtained from the accessed profile.

  In some embodiments, time-series processing steps may be performed to determine activity for individual objects of a plurality of predetermined objects. Providing the second signal may include identifying a particular one of the plurality of objects in which a user of the particular predetermined object is currently active.

  In some embodiments, a profile corresponding to a particular user is saved prior to the step of processing the timeline, identifying a particular user of a particular predetermined object, and corresponding to a particular identified user A previously saved one of the profiles may be accessed and the execution of the game device may be controlled using profile information obtained from the accessed profile.

  In some embodiments, the step of identifying a particular user is processing a time series to obtain gesture information about a gesture that includes a series of movements performed by the user, and the gesture information is a plurality of corresponding to the particular user. Comparing with a saved gesture map may include identifying a particular user whose gesture map matches the gesture information.

  In some embodiments, at least some of the stored gesture maps may be specific to a particular user. At least a part of the stored gesture map may be stored from gesture information obtained by processing a time series of gestures performed by the user. At least some of the stored gesture maps may be selectable by a particular user without processing a time series of gestures performed by the particular user.

  In some embodiments, the time series may include information regarding at least one of a predetermined object movement, position, orientation, or a predetermined sound level near the object. The predetermined object is a game controller provided with a main body, the main body has a front portion that faces the screen when the progress of the game is displayed on the screen, and the game controller is attached to the main body. Two input devices may be provided, and the input device may be operable by the user to register input from the user. Acquiring a sample timeline may include capturing the position of the PD element in the acquired image by the image acquisition device when at least a portion of the front of the game controller is facing the screen. Processing the time series may include determining that the user is currently active when the movement of the position of the PD element within the image does not exceed a threshold over a predetermined period of time.

  In some embodiments, obtaining the time series includes sampling information acquired by an inertial sensor embedded in a predetermined object, and processing the time series comprises processing the time series. And determining that the predetermined movement of the object exceeds a threshold value within a predetermined period.

  In some embodiments, obtaining the time series includes sampling information obtained by the acoustic transducer in the vicinity of the object operated by the user, and processing the time series is performed by the acoustic transducer. It may include determining that the user is currently active when the acquired information satisfies a predetermined criterion.

  In some embodiments, the acoustic information may be stored prior to the time series acquisition step. The step of identifying a specific user processes the acoustic information acquired by the acoustic transducer with respect to the stored acoustic information when the processing step determines that it is currently active for a particular predetermined object You may include that. The stored acoustic information relates to at least one of the acoustic features of the utterance or the meaning of the utterance, and the acoustic information acquired by the acoustic converter is information acquired by conversion of the utterance Good.

  Referring to FIG. 8B, a method 820 used in one embodiment of the present invention is shown. This embodiment provides a method used during operation of the gaming device to determine a lack of user activity associated with the gaming device. The method includes a step 822 of obtaining a sample timeline that includes information that can be used to determine an activity level of a user of the gaming device. Step 824 includes processing the time series to determine if the user is currently active. Then, in step 826, when it is determined in the time-series processing step that the user is not currently active, a first signal is provided to the game device for use in controlling the operation of the game device. The determination of a user's activity level can be performed in many different ways. For example, it can be determined by detecting a user walking in the room, detecting the presence of the user, detecting applause, or detecting the activity level by other methods. Thus, by determining this, the sensor records that there is no sufficient user operational or change criterion. The sensor may be incorporated into an object worn by the user and the steady state determined.

  In some embodiments, a user profile or user customized gesture can be found when the system exits from steady state. Based on the profile, new players can be added to the game. The system can detect steady state in many different ways. The system can detect user activity based on auditory patterns, video patterns, or inertial patterns, or by detecting a user profile or gesture.

  In some embodiments, the step of processing the time series may determine that the user is not currently active unless the user activity exceeds a threshold. The threshold may be modified during the step of processing the time series. The game device may suspend the execution of the game in response to the first signal.

  In some embodiments, the step of processing the time series is performed on individual user activities of the plurality of users to identify one or more individual users of the plurality of users who are not currently active, Providing one signal may include identifying a particular user that is not currently active among the plurality of users.

  In some embodiments, the processing step may include providing a second signal to the game device when it is determined that all of the predetermined set of multiple users are not currently active.

  In some embodiments, the time series of samples can be displayed by the gaming device with movement of at least part of the user's body, position of the user's body, posture of at least part of the user's body, or game progress. Information regarding at least one of the sound levels in the vicinity of the screen may be included.

  In some embodiments, obtaining the time series may include obtaining information from the image acquisition device in the vicinity of a screen that can display the progress of the game by the game device. The step of acquiring a time series may include acquiring information representing a position of at least one optically detectable (PD) element in an image acquired by the image acquisition device. The PD element may be included in an object worn by the user. The PD element may be distinguishable by at least one of luminance, color, texture, pattern, reflectance, or illuminance. The processing step may determine that the user is not currently active when a change in the position of at least one PD element does not exceed a predetermined threshold over a period of time.

  In some embodiments, at least one PD element may be a light source. At least one PD element may be a light emitting diode (LED). The object may comprise at least two PD elements. Obtaining the time series may include capturing the direction of the object relative to the x-axis and y-axis of the image plane.

  In some embodiments, the object may comprise three PD elements that make up a triangle. Obtaining the time series may include capturing the direction of the object relative to the z-axis perpendicular to the image plane.

  In some embodiments, the object may comprise at least four PD elements that constitute a square. Obtaining the time series may include capturing the direction of the object relative to the z-axis perpendicular to the image plane.

  In some embodiments, the processing step determines a direction of an object that includes direction data relating to at least one of pitch, yaw, and roll, and compares the direction data to predetermined direction data so that the user can It may further include determining whether it is active. Direction data may relate to the pitch, yaw and roll of the object.

In some embodiments, obtaining the time series includes sampling information obtained by inertial sensors embedded in an object worn by the user, and processing the time series comprises: Processing may include determining that the predetermined object motion does not exceed a threshold within a predetermined period of time. The inertial sensor may be at least one of an accelerometer, a gyroscope, or a laser gyroscope.
In some embodiments, obtaining the time series may include obtaining information that can be used to determine the acceleration of the object, and processing the time series may include determining the acceleration of the object. . The step of processing the time series determines the speed of the object by integrating acceleration values obtained from the time series over a period of time, and the user is not currently active when the speed determined by this process does not exceed the speed threshold Determination may be included.

  In some embodiments, the step of processing the time series determines the displacement of the object by first integrating the acceleration values obtained from the time series over a period of time and integrating the result over the period of time. It may include determining that the user is not currently active when the displacement displaced does not exceed a displacement threshold.

  In some embodiments, obtaining the time series includes sampling information obtained by the acoustic transducer in the vicinity of the user, and processing the time series includes information obtained by the acoustic transducer. Determining that the user is not currently active when the user does not meet the predetermined criteria. The progress of the game may be displayed on the screen, and the acoustic information is acquired by an acoustic transducer disposed in the vicinity of the screen.

  In some embodiments, the progress of the game is displayed on a screen, the acoustic information is acquired by an acoustic transducer disposed in the vicinity of the screen, and the step of processing the time series includes the information acquired by the acoustic transducer. Determining that a particular user of the plurality of users is not currently active when not satisfying a predetermined criterion specific to the particular user may be included.

  In some embodiments, the predetermined criteria include pitch, audio pulse frequency, audio pulse shape, audio pulse duty cycle, pulsation frequency, spectral signature including characteristic sound spectral content characteristics, specific user audio. It may include at least one criterion selected from the group consisting of spectral signatures that characterize, a time-varying signature that characterizes a series of one or more phonemes that are linked together and can be recognized by speech recognition.

  According to one embodiment of the present invention, a video game system and method of the type described above can be implemented as shown in FIG. The video game system 600 may include a processor 601 and a memory 602 (eg, RAM, DRAM, ROM, etc.). Furthermore, when implementing parallel processing, the video game system 600 may include a plurality of processors 601. The memory 602 may include data including the portion configured as described above and game program code 604. Specifically, the memory 602 may include inertial signal data 606 including controller path information stored as described above. Memory 602 may include stored gesture data 608, for example, data representing one or more gestures associated with game program 604. The encoded instructions executed by processor 601 may implement a multi-input mixer 605 configured and functioning as described above.

  The system 600 may include well-known support functions 610, such as an input / output (I / O) element 611, a power supply (P / S) 612, a clock (CLK) 613, a cache 614, and the like. The device 600 may optionally include a mass storage device 615 such as a disk drive, CD-ROM drive, tape device or the like for storing programs and / or data. The controller may optionally include a display device 616 and a user interface unit 618 to facilitate interaction between the controller 600 and the user. Display device 616 may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numbers, graphic symbols or images. User interface 618 may be a keyboard, mouse, joystick, light pen, or other device. Further, user interface 618 may be a microphone, video camera, or other signal conversion device for directly acquiring the signal to be analyzed. Processor 601, memory 602 and other components of system 600 may exchange signals (eg, code instructions and data) with each other via system bus 620 as shown in FIG.

  The microphone array 622 may be connected to the system 600 via the I / O function 611. The microphone array may be about 2-8 microphones, preferably 4 microphones, located less than about 4 cm, preferably about 1-2 cm away from the adjacent microphone. Preferably, the microphones in array 622 are omnidirectional microphones. An optional image acquisition device 623 (eg, a digital camera) is coupled to the device 600 via an I / O function 611. One or more pointing actuators 625 mechanically coupled to the camera can exchange signals with the processor 601 via an I / O function 611.

  As used herein, the term I / O generally refers to any program, operation or device that exchanges data between system 600 and peripheral devices. Any data transfer can be viewed as an output from one device and an input to another device. Peripheral devices include devices such as a writable CD-ROM that functions as an input / output device, in addition to devices dedicated to input such as a keyboard and mouse, devices dedicated to output such as a printer. The term “peripheral device” includes external devices such as a mouse, keyboard, printer, monitor, microphone, game controller, camera, external Zip drive or scanner, CD-ROM drive, CD-R drive or built-in modem. Also included are internal devices or other peripherals such as flash memory reader / writers, hard disks.

  In particular embodiments of the present invention, the device 600 may be a video game unit and includes a joystick controller 630 connected to the processor via an I / O function 611 by wire (eg, USB cable) or wirelessly. May be. The joystick controller 630 may include an analog joystick control 631 that provides control signals commonly used during video game play and a conventional button 633. Such a video game may be implemented as processor readable data and / or instructions that can be stored in the memory 602, or as other processor readable media such as that associated with the mass storage device 615. Also good.

  The joystick control 631 is normally configured to send a motion signal along the X axis when the control stick is moved left and right, and send a motion signal along the Y axis when the stick is moved back and forth (up and down). In a joystick configured for three-dimensional movement, a signal of movement along the Z axis is sent when the stick is twisted leftward (counterclockwise) or rightward (clockwise). These three axes X, Y, and Z are often called roll, pitch, and yaw, respectively, particularly with respect to aircraft.

  In addition to conventional features, the joystick controller 630 may include one or more inertial sensors 632. The inertial sensor can provide position information and / or direction information to the processor 601 via an inertial signal. The direction information may include angle information such as tilt, roll or yaw of the joystick controller 630. As an example, inertial sensor 632 may be any number of accelerometers, gyroscopes, tilt sensors, and / or combinations thereof. In a preferred embodiment, the inertial sensor 632 includes a tilt sensor adapted to sense the direction of the joystick controller with respect to the tilt and roll axes, a first accelerometer that senses acceleration along the yaw axis, and a yaw axis. You may provide the 2nd accelerometer which senses angular acceleration. The accelerometer may be a MEMS device having a sensor that senses displacement of the mass in one or more directions and having a mass attached by one or more springs. The acceleration of the joystick controller 630 can be determined using a signal from the sensor in response to the mass displacement. Such a technique is implemented by program code instructions 604 stored in the memory 602 and executable by the processor 601.

  As an example, an accelerometer suitable as inertial sensor 632 may be a single mass that is elastically coupled to the frame at three or four points, eg, by a spring. The pitch axis and roll axis are in a plane across the frame, and the frame is attached to the joystick controller 630. As the frame (and joystick controller 630) rotates about the pitch and roll axes, the mass is displaced by the effect of gravity and the springs expand and contract depending on the pitch and / or roll angle. Mass displacement is sensed and converted into a signal depending on the amount of pitch and / or roll. Angular acceleration about the yaw axis, or linear acceleration along the yaw axis, produces a characteristic pattern of spring expansion or contraction or mass movement that can be sensed and converted to a signal depending on the amount of angular or linear acceleration. Such an accelerometer device can measure tilt, roll, angular acceleration about the yaw axis, and linear acceleration about the yaw axis by tracking mass movement or spring stretching force. There are many different ways to track mass position and / or force on mass, including resistive strain gauge materials, photonic sensors, magnetic sensors, Hall effect devices, piezoelectric devices, capacitive sensors, etc. .

  In addition, joystick controller 630 may include one or more light sources 634, such as light emitting diodes (LEDs). The light source 634 may be used to distinguish one controller from another. For example, this can be accomplished with one or more LEDs by flashing or holding the LED pattern code. As an example, five LEDs can be provided on the joystick controller 630 in a linear or two-dimensional pattern. Preferably, the LEDs are linearly arranged, but instead the LEDs are arranged in a rectangular pattern or arc shape to facilitate determination of the image plane of the LED array when analyzing the image of the LED pattern acquired by the image acquisition device 623. You may arrange by a pattern. Further, the LED pattern code may be used to determine the position of the joystick controller 630 during the game. For example, LEDs can help distinguish controller tilt, yaw, and roll. This detection pattern is useful for improving the user's feeling in games such as airplane flight games. The image acquisition device 623 can capture an image including the joystick controller 630 and the light source 634. Such an image can be analyzed to determine the position and / or orientation of the joystick controller. This type of analysis can be implemented by program code instructions 604 stored in memory 602 and executed by processor 601. In order to facilitate the capture of the image of the light source 634 by the image acquisition device 623, the light source 634 may be disposed on two or more different sides of the joystick controller 630, for example on the front and back (shown in dotted lines). . With this arrangement, the image acquisition device 623 can acquire images of the light source 634 in different directions of the joystick controller 630 depending on how the user holds the joystick controller 630.

  In addition, the light source 634 can provide a telemetry signal to the processor 601 in the form of, for example, pulse code, amplitude modulation, or frequency modulation. Such a telemetry signal can indicate which joystick button is pressed and / or how hard the button is pressed. The telemetry signal can be encoded into an optical signal by pulse coding, pulse width modulation, frequency modulation or light intensity (amplitude) modulation. The processor 601 decodes the telemetry signal from the optical signal and executes a game command in response to the decoded telemetry signal. The telemetry signal may be decoded from an image analysis of the joystick controller 630 acquired by the image acquisition device 623. Alternatively, the device 601 may comprise a separate optical sensor dedicated for receiving telemetry signals from the light source 634. For use of LEDs in determining intensity when interacting with a computer program, see, for example, Richard L. Marks et al., US patent application Ser. No. 11 / 429,414, filed May 4, 2006, “USE OF COMPUTER IMAGE AND”. AUDIO PROCESSING IN DETERMINING AN INTENSITY AMOUNT WHEN INTERFACING WITH A COMPUTER PROGRAM ”(agent serial number SONYP052), the entirety of which is incorporated herein by reference. In addition, analysis of the image including the light source 634 can be used for both telemetry and determination of the position and / or orientation of the joystick controller 630. Such techniques can be implemented by program code instructions 604 stored in memory 602 and executable by processor 601.

  The processor 601 receives the inertial signal from the inertial sensor 632 together with the optical signal from the light source 634 detected by the image acquisition device 623 and / or the sound source position and feature information obtained from the acoustic signal detected by the microphone array 622. Can be used to infer the position and / or orientation of the joystick controller 630 and / or its user. For example, using “acoustic radar” sound source locations and features with microphone array 622 to track moving voice while joystick controller movement is being tracked independently (by inertial sensor 632 or light source 634). Can do. In the acoustic radar, a pre-adjusted listening zone is selected at run time, and sounds originating from sound sources outside the pre-adjusted listening zone are removed. The pre-adjusted listening zone may include a listening zone corresponding to the focal size or field of view of the image acquisition device 623. Examples of acoustic radar are described in detail in U.S. Patent Application No. 11 / 381,724, filed May 4, 2006, Xiadong Mao's `` METHODS AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION '', which is incorporated herein by reference. Incorporated. Different combinations of any number of different modes that provide control signals to the processor 601 can be used with embodiments of the present invention. Such techniques can be implemented by program code instructions 604 that are stored in memory 602 and executable by processor 601. One or more instructions may be optionally included that select a pre-adjusted listening zone at run time and instruct one or more processors to remove sound originating from sound sources outside the pre-adjusted listening zone. The pre-adjusted listening zone may include a listening zone corresponding to the focus size and field of view of the image acquisition device 623.

Program code 604 is transmitted to microphones M ₀ . . . Instruct to generate a discrete time domain input signal x _m (t) from M _M , select a finite impulse response filter coefficient using a listening sector with semi-blind source separation, and from the input signal x _m (t) One or more instructions for separating different sound sources may be selectively included. Program 604 may include instructions for applying one or more non-integer delays to selected input signals x _m (t) other than input signal x ₀ (t) from reference microphone M ₀ . Each non-integer delay can be selected to optimize the signal-to-noise ratio of the discrete time domain output signal y (t) obtained from the microphone array. As the signal from the reference microphone M ₀ is first becomes temporally to the signal from the other microphone array may be selected fractional delay. Program 604 may include instructions for introducing a non-integer time delay Δ into the output signal y (t) of the microphone array. That is, y (t + Δ) = x (t + Δ) * b ₀ + x (t−1 + Δ) * b _i + x (t−2 + Δ) * b ₂ +. . . + X (t−N + Δ) b _N. Here, Δ is between 0 and ± 1. Examples of such techniques are described in detail in U.S. Patent Application No. 11 / 381,729, filed May 4, 2006, Xiadong Mao's "ULTRA SMALL MICROPHONE ARRAY", which is incorporated herein by reference. The

  Program code 604 may include one or more instructions that, when executed, cause system 600 to select a preconditioned listening sector that includes a sound source. Such an instruction allows the device to determine whether the sound source is within the inertial sector or on a specific side of the inertial sector. If the sound source is not in the default sector, at runtime, the instruction selects a different sector on a particular side of the default sector. Different sectors can be characterized by the attenuation of the input signal closest to the optimal value. These instructions can calculate the attenuation of the input signal from the microphone array 622 and the attenuation to the optimum value at runtime. At run time, an instruction can determine the attenuation value of the input signal to one or more sectors and cause the device to select the sector whose attenuation is closest to the optimum value.

  Examples of such techniques are described, for example, in U.S. Patent Application No. 11 / 381,725, filed May 4, 2006, Xiadong Mao's “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION”, the disclosure of which is incorporated by reference. Incorporated herein by reference.

  The signal from the inertial sensor 632 may provide part of the tracking information input, and the signal generated from the image acquisition device 623 by tracking one or more light sources 634 may provide another part of the tracking information input. As an example, such a “mixed mode” signal can be used in a football video game in which quarterback flicks his head to the left and then throws the ball to the right, but is not limited thereto. Specifically, a game player holding the controller 630 can make a pitch operation by rotating the head to the left and making a sound while swinging the controller in the right direction as if it were football. A microphone array 622 in conjunction with the “acoustic radar” program code can follow the user's voice. The image acquisition device 623 can track the user's head movement or other commands that do not require voice or the use of a controller. Sensor 632 can track the movement of a joystick controller (representing football). The image acquisition device 623 may track the light source 634 on the controller 630. The user can release the “ball” when the amount and / or direction of acceleration of the joystick controller 630 reaches a specific value, or by a key command activated by pressing a button on the joystick controller 630.

  In certain embodiments of the invention, the position of the joystick controller 630 can be determined using, for example, an inertial signal from an accelerometer or a gyroscope. Specifically, the acceleration signal from the accelerometer can be temporarily integrated to obtain a speed change, and the speed is time integrated to obtain a position change. If the values of the initial position and speed at a certain time are known, the absolute position can be obtained using these values, speed change, and position change. Although the inertial sensor can be used to locate faster than using the image acquisition device 623 and the light source 634, the inertial sensor 632 can be affected by a type of error known as “drift”. That is, errors accumulate over time, and a mismatch D may occur between the position of the joystick 630 calculated by the inertia signal (shown by a dotted line) and the actual position of the joystick controller 630. Embodiments of the present invention allow several ways to deal with such errors.

  For example, the drift can be manually canceled by resetting the initial position of the joystick controller 630 to be equal to the current calculated position. The user can use one or more buttons on the joystick controller 630 to activate a command to reset the initial position. Instead, image-based drift can be implemented by resetting the current position with reference to the position obtained from the image obtained from the image acquisition device 623. Such image-based drift correction can be performed manually, for example, when the user activates one or more buttons of the joystick controller 630. Alternatively, image-based drift correction can be performed automatically, for example at regular time intervals or in response to game play. Such a technique can be implemented by program code instructions 604 stored in memory 602 and executable by processor 601.

  In certain embodiments, it is desirable to compensate for spurious data in the inertial sensor signal. For example, the signal from the inertial sensor 632 may be oversampled, a moving average may be calculated from the oversampled signal, and spurious data may be removed from the inertial sensor signal. In some situations, the signal may be oversampled to remove high and / or low values from some subset of data points, and a moving average may be calculated from the remaining data points. In addition, other data sampling and data manipulation techniques can be used to adjust the inertial sensor signal to remove or reduce the significance of spurious data. The technique can be selected according to the nature of the signal, the calculations performed on the signal, the nature of the game play, or a combination of two or more of these. Such techniques can be implemented by program code instructions 604 stored in memory 602 and executable by processor 601.

  The processor 601 can analyze inertial signal data as described above according to the data 606 stored and read in the memory 602 and executed by the processor module 601 and the program code instructions of the program 604. The code portion of program 604 may conform to any one of a plurality of different programming languages such as assembly, C ++, JAVA, or some other language. The processor module 601 forms a general purpose computer that becomes a special purpose computer when executing a program such as the program code 604. Although it is described herein that program code 604 is implemented in software and executed on a general purpose computer, one of ordinary skill in the art will recognize hardware such as an application specific integrated circuit (ASIC) or other hardware circuit. You will understand that task management methods can also be implemented using hardware. Thus, it should be understood that all or part of embodiments of the present invention can be implemented in software, hardware or a combination of both.

  In one embodiment, program code 604 may include a set of processor readable instructions that implement methods having features similar to method 510 of FIG. 5B and method 520 of FIG. 5C, or combinations of two or more thereof. Good. The program code 604 generally instructs one or more processors to analyze the signal from the inertial sensor 632 to generate position information and / or direction information, and use one or more of this information during video game play. May also include instructions.

  Program code 604, when executed, causes image acquisition device 623 to monitor the field of view in front of device 623, identify one or more light sources 634 within the field of view, detect changes in light emitted from light sources 634, and detect changes. In response, processor executable instructions may optionally be included, including one or more instructions that activate input commands to the processor 601. Game controller action activation using LEDs with an image acquisition device is described in US Patent Application No. 10 / 759,782, filed January 16, 2004, "METHOD AND APPARATUS FOR LIGHT INPUT DEVICE" by Richard L. Marks. Which is incorporated herein by reference in its entirety.

  The program code 604, when executed, uses one or more signals that use signals from the inertial sensors and signals generated by the image acquisition device from tracking one or more light sources, for example, as input to a gaming system as described above. Processor-executable instructions, including instructions, may optionally be included. Program code 604 may optionally include processor-executable instructions, including one or more instructions that compensate for drift of inertial sensor 632 during execution.

  Although embodiments of the present invention have been described with respect to an example associated with a video game controller 630, embodiments of the present invention including the system 600 have inertial detection capabilities and the ability to transmit inertial sensor signals wirelessly or otherwise. Can be used with any user-operable body, molded object, knob, structure, etc.

  As an example, embodiments of the present invention can be implemented in a parallel processing system. Such parallel processing systems typically include two or more processor elements configured to execute program portions in parallel using separate processors. FIG. 7 illustrates a type of cell processor 700 according to one embodiment of the invention. Cell processor 700 may be used as processor 601 in FIG. 6 or as processor 502 in FIG. 5A. In the example shown in FIG. 7, the cell processor 700 includes a main memory 702, a power processor element (PPE) 704, and a plurality of synergistic processor elements (SPE) 706. In the example shown in FIG. 7, the cell processor 700 includes one PPE 704 and eight SPEs 706. In such a configuration, seven of the SPEs 706 may be used for parallel processing, and one may be reserved as a backup when one of the other seven units fails. Alternatively, the cell processor may include a plurality of groups of PPE (PPE groups) and a plurality of groups of SPEs (SPE groups). In this case, hardware resources can be shared between units in the group. However, SPE and PPE must appear to the software as independent elements. Thus, embodiments of the present invention are not limited to the use of the configuration shown in FIG.

  The main memory 702 is typically a general purpose non-volatile storage device and special purpose hardware used for functions such as system configuration, data transfer synchronization, memory mapped I / O, I / O subsystem, etc. Includes both wear registers and arrays. In the embodiment of the present invention, the video game program 703 may reside in the main memory 702. The video game program 703 may include an inertial analyzer, an image analyzer, and an acoustic analyzer and a mixer configured as described above or in combination with respect to FIGS. 4, 5A, 5B, or 5C above. The program 703 may be executed on the PPE. The program 703 may be divided into multiple signal processing tasks that can be executed on the SPE and / or PPE.

  As an example, PPE 704 may be a 64-bit power PC processor unit (PPU) with associated caches L1, L2. The PPE 704 is a general-purpose processing unit that can access system management resources (such as a memory protection table). As seen by the PPE, hardware resources may be explicitly mapped to real address space. Thus, the PPE can directly address any of these resources using an appropriate effective address value. The main function of the PPE 704 is task management and task assignment for the SPE 706 of the cell processor 700.

  Although only one PPE is shown in FIG. 7 as a cell processor implementation such as Cell Broadband Engine Architecture (CBEA), the cell processor 700 can be configured to have multiple PPEs organized into two or more PPE groups. You may have. These PPE groups may share access to the main memory 702. Further, the cell processor 700 may include two or more SPE groups. SPE groups may share access to main memory 702. Such a configuration is within the scope of the present invention.

  Each SPE 706 includes a synergistic processor unit (SPU) and its own local storage area LS. The local storage area LS may have one or more separate memory storage areas, each associated with a particular SPU. Each SPU may be configured to execute only instructions (including data loading and data storage operations) from within its associated local storage area. In such a configuration, data transfer between the local storage LS and any location in the system 700 is a direct memory access (DMA) command that transfers data in and out of the local storage area (in the individual SPE). Can be executed by issuing a memory flow controller (MFC). The SPU is a computational unit that is less complex than the PPE 704 in that it does not perform any system management functions. SPUs generally have single instruction, multiple data (SIMD) capabilities and process data to perform any necessary data transfer (access to characteristics set by the PPE) to perform assigned tasks. Start). The purpose of the SPU is to enable applications that require higher computational unit density and can effectively use the provided instruction set. A significant number of SPEs in the system managed by PPE 704 allow for cost-effective processing for a wide range of applications.

  Each SPE 706 can comprise a dedicated memory flow controller (MFC) having an associated memory management unit that can hold and process memory protection and access permission information. The MFC provides the primary method of data transfer, protection and synchronization between the cell processor main storage and the SPE local storage. The MFC command describes the transfer to be performed. A command for transferring data may be referred to as an MFC direct memory access (DMA) command (or MFC DMA command).

  Each MFC can support multiple DMA transfers simultaneously and can maintain and process multiple MFC commands. Each MFC DMA data transfer command may require both a local storage device address (LSA) and an effective address (EA). A local storage address may directly address only the local storage area of its associated SPE. Effective addresses may have more general applications. For example, if the effective address is aliased in the real address space, the main storage device including all SPE local storage areas can be referenced.

  To facilitate communication between SPE 706 and / or communication between SPE 706 and PPE 704, SPE 706 and PPE 704 may include a signal notification register that signals an event. PPE 704 and SPE 706 may be combined by a star topology that acts as a router from which PPE 704 sends messages to SPE 706. Alternatively, each SPE 706 and PPE 704 may have a one-way signal notification register called a mailbox. Mailboxes are used by SPE 706 and can host operating system (OS) synchronization.

  Cell processor 700 may include an input / output (I / O) function 708 that allows cell processor 700 to interact with peripheral devices such as microphone array 712, optional image acquisition device 713, and game controller 730. The game controller unit may include an inertial sensor 732 and a light source 734. In addition, an element interconnection bus 710 may connect the various components described above. Each SPE and PPE can access the bus 710 through the bus interface unit BIU. Cell processor 700 may include two controllers that are typically found in the processor. That is, the memory interface controller MIC that controls the flow of data between the bus 710 and the main memory 702, and the bus interface controller BIC that controls the flow of data between the I / O 708 and the bus 710. Although the requirements for MIC, BIC, BIU and bus 710 can vary widely for various implementations, those skilled in the art will be familiar with the functions and circuitry for implementing them.

  The cell processor 700 may include an internal interrupt controller IIC. The IIC component manages the priority of interrupts presented to the PPE. The IIC allows the cell processor 700 to handle interrupts from other components without using the main system interrupt controller. The IIC may be considered a second level controller. The main system interrupt controller may handle interrupts originating from outside the cell processor.

  In embodiments of the invention, certain calculations, such as non-integer delays, can be performed in parallel using PPE 704 and / or one or more SPEs 706. Each non-integer delay calculation can be performed as one or more separate tasks that can be undertaken as different SPEs 706 become available.

  While the above is a complete description of the preferred embodiments of the invention, various alternatives, modifications and equivalents may be used. Accordingly, the scope of the invention should not be determined with reference to the above description, but instead should be determined with reference to the full scope of the appended embodiments and their equivalents. Any feature described herein, whether suitable or not, can be combined with any other feature described herein. Except where expressly stated otherwise, in the claims, the indefinite article "A" or "An" refers to the quantity of one or more items that follow the article. Unless explicitly recited in a given claim using the phrase “means for”, the appended claims should not be construed as including means plus function limitations.

Claims (26)

A memory for storing game programs;
A method for use in operating a gaming device comprising: a processor configured to access a game program in the memory, execute the game program, and allow the user to play the game according to interaction with the user Because
Inertia tracking information obtained from an inertial sensor on the object, light source on the object as information that can be used by the processor to determine the activities of a plurality of predetermined objects used for controlling the game device At least two of image tracking information obtained by detecting the sound emitted from the object with a microphone connected to the processor Step to get one ,
The processor processes the inertia tracking information, image tracking information, and acoustic tracking information to determine whether one of the plurality of predetermined objects is active during the game. When,
In response to a determination that the processor is currently active after determining that one of the plurality of predetermined objects is not active, the processor adds a new player to the game in progress And steps to
Including methods. Before Symbol processor inertial tracking information further comprises the step of generating a mixed tracking information by mixing at least two of the image tracking information and the acoustic tracking information,
The processor of claim 1, wherein processing the information includes processing the mixed tracking information to determine whether one of the plurality of predetermined objects is currently active. The method described. Processing the information to determine whether one of the plurality of predetermined objects is active during the game,
The processor processes the mixed tracking information;
The processor determines that one of the plurality of predetermined objects is not currently active unless one of the plurality of predetermined objects exceeds a threshold;
The method of claim 2, wherein the processor includes modifying the threshold while a game is in progress. 4. A method according to any preceding claim, wherein the processor includes detecting individual movements of the plurality of predetermined objects in the step of obtaining the information.   5. The method of claim 4, wherein the processor detects motion based on acquiring images of one or more optically detectable (PD) elements each attached to the predetermined object. . 6. A method according to claim 4 or 5 , wherein the processor detects motion based on acoustic data. 7. A method according to any one of claims 3 to 6, wherein the processor detects movement based on data received from inertial sensors embedded in each of the predetermined objects. Determining whether one of the plurality of predetermined objects is active ;
The processor determines individual directions of the plurality of predetermined objects including direction data relating to at least one of pitch, yaw, and roll;
Wherein the processor, the direction data is compared with a predetermined direction data, to further comprise, claims 1 that determines whether one of the plurality of predetermined object is not active 7 The method in any one of. 9. A method as claimed in any preceding claim, wherein the plurality of predetermined objects include a plurality of game controllers. A system used for operating a game device,
Information that can be used to determine the activity of a plurality of predetermined objects used for control of the game device includes inertia tracking information obtained from inertia sensors on the object, and light sources on the object to the system. Acquire at least two of image tracking information obtained by detecting with a connected image acquisition device and acoustic tracking information obtained by detecting sound emitted from the object with a microphone connected to the system Means,
Means for processing the inertial tracking information, image tracking information, and acoustic tracking information to determine whether one of the plurality of predetermined objects is active during the game;
Means for adding a new player to an ongoing game in response to determining that one is currently active after determining that one of the plurality of predetermined objects is not active;
A system comprising: Inertial tracking information, further comprising means for generating a mixed tracking information by mixing at least two of the image tracking information and the acoustic tracking information,
The method of claim 10, wherein the means for processing the information processes the mixed tracking information to determine whether one of the plurality of predetermined objects is currently active. system. Means for obtaining said information comprises means for detecting individual movements of said plurality of predetermined objects;
12. The system of claim 11, wherein motion is detected based on acquiring images of one or more optically detectable (PD) elements each attached to the predetermined object. Means for processing the information to determine whether one of the plurality of predetermined objects is active during the game;
Process the mixed tracking information and one of the plurality of predetermined objects is not currently active unless one of the plurality of predetermined objects exceeds a threshold. The system according to claim 11 or 12 , characterized by comprising means for determining.   The system of claim 13, wherein the means for processing further comprises means for modifying the threshold during a game. 15. A system according to claim 13 or 14 , wherein motion is detected based on acoustic data. 16. A system according to any of claims 10 to 15 , wherein the means for obtaining information comprises means for obtaining individual directions of the plurality of predetermined objects. The processing means determines individual directions of the plurality of predetermined objects including direction data regarding at least one of pitch, yaw, and roll, and compares the direction data with predetermined direction data. to the one of the plurality of predetermined objects, further comprising means for determining whether or not active, according to any of claims 10 to 16 systems. A system according to any of claims 10 to 17 wherein the plurality of objects defined in advance, characterized in that it comprises a plurality of game controllers. A device used to control the progress of a game,
A memory for storing game programs;
A processor configured to access the game program in the memory, execute the game program, and allow the user to play the game in accordance with interaction with the user;
The processor is
Information that can be used to determine the activity of a plurality of predetermined objects used for controlling the game device includes inertia tracking information obtained from an inertia sensor on the object, and a light source on the object to the processor. Acquire at least two of image tracking information obtained by detecting with a connected image acquisition device and acoustic tracking information obtained by detecting sound emitted from the object with a microphone connected to the processor ,
Processing the inertial tracking information, the image tracking information, and the acoustic tracking information to determine whether one of the plurality of predetermined objects is active during the game;
Configured to add a new player to an ongoing game in response to determining that one of the plurality of predetermined objects is currently active after determining that one of the plurality of predetermined objects is not active The apparatus characterized by being made. Before Symbol processor, inertial tracking information, is further configured to generate a mixed tracking information by mixing at least two of the image tracking information and the acoustic tracking information,
The apparatus of claim 19, wherein processing the information includes processing the mixed tracking information to determine whether one of the plurality of predetermined objects is currently active. The processor is
Process the mixed tracking information and one of the plurality of predetermined objects is not currently active unless one of the plurality of predetermined objects exceeds a threshold. Judgment,
21. The apparatus of claim 20, further configured to modify the threshold while the game is in progress. The apparatus according to claim 20 or 21 , wherein the processor detects movement based on acoustic data. The plurality of predetermined objects include a game controller communicatively connected to the processor;
The game controller comprises a main body and at least one optically detectable (PD) element assembled to the main body;
The PD element is distinguishable from the main body by at least one of luminance, color, texture, pattern, reflectance, or illuminance.
The processor, and acquires the information representative of the position of at least one PD element within the image acquired by the image acquisition apparatus, apparatus according to any one of claims 20 to 22. An image acquisition device connected to the processor;
Receiving the information includes receiving an image recorded by the image acquisition device;
The processing of the information determines the position of the PD element in the image and determines the position of the PD element in the image at different times to determine whether a user associated with the game controller is currently active 24. The apparatus of claim 23, comprising: The main body of the game controller is
When the progress of the game controlled by the processor is displayed on the screen, the front facing the screen;
At least one input device that is assembled to the main body and operated by a user to input from the user;
25. The apparatus of claim 23 or 24 , further comprising: An image acquisition device connected to the processor;
The game controller further comprises at least one optically detectable (PD) element assembled to the body;
The position of the PD element in the image can be recorded by the image acquisition device when the front is substantially facing the screen;
26. An apparatus according to any of claims 23 to 25, wherein the position of the PD element at different time points quantifies the movement of the body in space.

Images