JP5465948B2 - How to get input to control the execution of a game program - Google Patents

Description

  The present invention relates generally to human-computer interfaces, and more particularly to multi-channel input processing for tracking user operation of one or more controllers.

  Computer entertainment systems typically include handheld controllers, game controllers, or other controllers. The user or player uses the controller to send commands and other commands to the entertainment system to control the video game being played and other simulations. For example, the controller may be provided with a manipulator operated by a user such as a joystick. Variables in the joystick operation are AD converted and sent to the main frame of the game machine. The controller is also provided with buttons that can be operated by the user.

  The present invention has evolved with respect to these conventional and other information elements.

  A method for obtaining input for controlling execution of a game program is disclosed. a) Information specifying the current position of the control stick movable by the user of the game controller in relation to the stop position of the control stick, or b) Specifying whether the switch included in the game controller is in operation. Controller input information including at least one of the information is received from a game controller operable by the user. Receive additional input information from the environment in which the controller is used. A combination input for controlling the execution of the game program is obtained by processing the controller input information and the additional input information to obtain a combination input.

  The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

1 illustrates a video game system that operates in accordance with one embodiment of the present invention. FIG. 2 is a perspective view of a controller made in accordance with one embodiment of the present invention. FIG. It is a three-dimensional schematic diagram showing an accelerometer used in a controller according to an embodiment of the present invention. 1 is a block diagram of a mixing system for various control inputs according to an embodiment of the present invention. It is a block diagram of the video game system part of FIG. FIG. 5 is a flowchart illustrating a controller tracking method of a video game system according to an embodiment of the present invention. It is a flowchart which shows the utilization method of a positional information and / or direction information during the game play in the video game system by one Embodiment of this invention. 1 is a block diagram illustrating a video game system according to an embodiment of the present invention. 1 is a block diagram of a cell processor implementation in a video game system according to an embodiment of the present invention. FIG.

  In the following detailed description, there are many specific details for explanation, and it is understood by those skilled in the art that many changes and modifications thereto are also included in the technical scope of the present invention. You will understand. Accordingly, the preferred embodiments of the invention described below are described without impairing or limiting the universality of the claimed invention.

  Various embodiments relating to the methods, apparatus, schemes and systems described herein provide for detection, capture and tracking of movement, movement and / or manipulation of the controller body itself by the user. The detected movement, movement, and / or operation of the entire controller body by the user is used as additional commands to control various aspects of the game being played and other simulations.

  Detection and tracking of the operation of the game controller main body by the user can be realized in various modes. For example, an inertial sensor such as an accelerometer or a gyroscope or an imaging device such as a digital camera can be used with a computer entertainment system to detect the movement of the controller body that is held and send the movement to an action in the game. An example of controller tracking by inertial sensors is disclosed, for example, in “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUT CONTROL” (Attorney Docket No. SCEA06INRTI) in US Patent Application No. 11 / 382,033. Is incorporated herein by reference. An example of controller tracking using an imaging device is disclosed, for example, in “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY” in US patent application Ser. No. 11 / 382,034 (agent number SCEA05082US00). The disclosure is incorporated herein by reference. In addition, the controller and / or user can be tracked acoustically using a microphone array and appropriate signal processing. An example of such acoustic tracking is disclosed in US patent application Ser. No. 11 / 381,721, the disclosure of which is hereby incorporated by reference.

  By using acoustic detection, inertia detection and imaging independently or in combination, many different ways of movement of the controller, such as vertical movement, torsion, horizontal movement, jerking movements, sticks, etc. Motion (wand-like motions), sudden descent motion, etc. can be detected. Such movement may correspond to various commands and be transferred to actions in the game. Many different types of games and simulations can be realized by detecting and tracking the operation of the game controller by the user. For example, fighting with a sword or lightsaber, using a stick to follow the shape of an object, participating in various sporting events, fighting on the screen, and other encounters can be performed. The game program may be configured to track the movement of the controller and recognize a pre-recorded gesture from the tracked movement. If one or more of these movements are recognized, the game state may be changed.

  In embodiments of the present invention, controller path information obtained from these different sources may be combined before analysis of gesture recognition. Tracking data from different sources (eg, sound, inertia and imaging) are combined in a manner that increases the likelihood of gesture recognition.

  Please refer to FIG. Shown is a system 100 that operates in accordance with one embodiment of the present invention. As shown, the computer entertainment console 102 is connected to a television or other video display 104 to display video games and other simulations on the screen. A game or other simulation can be stored on a DVD, CD, flash memory, USB memory, or other recording medium 106 inserted into the console 102. The user or player 108 operates the game controller 110 to control the video game or other simulation. As shown in FIG. 2, the game controller 110 includes an inertial sensor 112 that generates a signal in accordance with the position, movement, direction, or change in direction of the game controller 110. In addition to the inertial sensor, the game controller 110 may include a known control input device such as a joystick 111, buttons 113, R1, and L1.

  During operation, the user 108 physically moves the controller 110. For example, the controller 110 may be moved by the user 108 in any direction, such as up / down movement, horizontal movement, torsion, roll, vibration, sudden movement, or sudden drop. These movements of the controller 110 itself may be detected and imaged by the camera 114 to analyze and track the signal from the inertial sensor 112 in the manner described below.

  Returning again to FIG. 1, the system 100 may optionally include a camera or other video imaging device 114, which is positioned such that the controller 110 falls within the camera viewing angle 116. The analysis of the image from the image capturing device 114 is used in conjunction with the analysis of data from the inertial sensor 112. As shown in FIG. 2, the controller 110 may optionally be provided with a light source such as light emitting diodes (LEDs) 202, 204, 206, 208 to facilitate tracking by image analysis. These light sources may be attached to the main body of the controller 110. In this specification, the term “main body” means a part of the game controller 110 that is held by hand (or worn if it can be worn).

  Video image analysis for tracking purposes of the controller 110 is, for example, “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY” (Attorney Docket No. SCEA05082US00) of US patent application Ser. No. 11 / 382,034. The disclosure of which is incorporated herein by reference. The console 102 may include an acoustic transducer such as a microphone array 118. The controller 110 further includes an acoustic signal generator 210 (eg, a speaker) that provides a sound source that facilitates acoustic tracking of the controller 110 as disclosed in the microphone array 118 and US patent application Ser. No. 11 / 381,724. You may make it provide for an appropriate acoustic signal processing apparatus. The disclosures of the above-cited US patent applications are hereby incorporated by reference.

  In general, the signal from the inertial sensor 112 is used to generate position data and direction data for the controller 110. Such data includes many physical aspects related to the movement of the controller 110, such as telemetry of the location of the controller 110, as well as any axial acceleration and velocity, tilt, pitch, yaw, roll, etc. Can be used to calculate In the present specification, telemetry generally means that information of an object is measured remotely and reported to a system or a designer or operator of the system.

  By detecting and tracking the movement of the controller 110, it can be determined whether or not a predetermined movement is being performed for the controller 110. That is, a specific movement pattern or gesture of the controller 110 can be predetermined and used as an input command for a game or other simulation. For example, a gesture that causes the controller 110 to descend downward can be defined as one command, a twisting gesture can be defined as another command, and a gesture that vibrates can be defined as another command. In this way, the manner in which the user 108 physically moves the controller 110 can be used as another input to control the game, giving the user a more exciting and enjoyable experience.

  By way of example and without limitation, inertial sensor 112 may be an accelerometer. FIG. 3 shows an accelerometer 300 in the form of a single mass 302 that is telescopically attached to frame 304 at four points, for example, by springs 306, 308, 310, and 312. The pitch axis and roll axis (represented by the X axis and Y axis, respectively) lie in a plane that traverses the frame. The yaw axis Z is placed perpendicular to a plane including the pitch axis X and the roll axis Y. Frame 304 is attached to controller 110 in a suitable form. In response to acceleration and / or rotation of the frame 304 (and game controller 110), the mass 302 is displaced relative to the frame 304. The springs 306, 308, 310, 312 expand and contract according to the amount and direction of translational acceleration and / or rotational acceleration, and / or the pitch, roll, and yaw angles. The displacement of the mass 302 and / or the amount of expansion / contraction of the springs 306, 308, 310, 312 is measured, for example using suitable sensors 314, 316, 318, 320, and the pitch and / or in a known or determinable manner. It is converted into a signal corresponding to the amount of roll acceleration.

  There are many different ways to track the position of the mass and / or the forces acting on it, such as resistive strain gauges, acoustic sensors, magnetic sensors, Hall elements, piezoelectric elements, capacitive sensors, etc. Embodiments of the present invention may include any number of sensors, or any type, any combination of sensors. By way of example and without any limitation, the sensors 314, 316, 318, 320 can be gap-blocking electrodes disposed above the mass 320. The capacitance between the mass and each electrode changes according to the change in the position of the mass with respect to each electrode. Each electrode may be connected to a circuit that generates a signal related to the capacitance of the mass 302 to the electrode (and thus a signal related to the proximity of the mass). Further, the springs 306, 308, 310, 312 may include resistance strain gauges that generate signals related to expansion and contraction.

  In some embodiments, the frame 304 may be a gimbal that is attached to the controller 110 so that the accelerometer 300 maintains a constant orientation relative to the pitch, roll, and / or yaw axes. it can. In this manner, it is possible to directly map the X axis, Y axis, and Z axis of the controller to the corresponding axes in the real space without considering the inclination of the controller axis with respect to the real space coordinate axes.

  As described above, data from inertial sources, imaging sources, and acoustic sources can be analyzed to generate a path that tracks the position and / or orientation of the controller 110. As shown in the block diagram of FIG. 4, a system 400 according to an embodiment of the present invention may include an inertial analysis unit 402, an image analysis unit 404, and an acoustic analysis unit 406. Each of these analysis units receives a signal from the measured environment 401. The analysis units 402, 406, and 408 are implemented by hardware, software (or firmware), or a combination of these. Each analysis unit generates tracking information related to the position and / or orientation of the object. By way of example, the object may be the controller 110 described above. The image analyzer 404 may work together to form the following fields and fields for the method disclosed in US patent application Ser. No. 11 / 382,034 (Attorney Docket No. SCEA05082US00). The inertial analysis unit 402 works together to provide the following method and for the method disclosed in “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENTIONAL INPUT CONTROL” (Attorney Docket No. SCEA06INRT1) of US Patent Application No. 11 / 382,033. A field may be formed. The acoustic analyzer 406 may work together to form the following fields and fields for the method disclosed in US patent application Ser. No. 11 / 381,724.

  The analyzers 402, 404, and 406 may be considered to be associated with different channels of input regarding position information and / or direction information. The mixer 408 can accept a plurality of input channels, which may include sample data that characterizes the environment to be measured 401 from the channel viewpoint. The position and / or direction information generated by the inertial analysis unit 402, the image analysis unit 404 and the acoustic analysis unit 406 can be connected to the input of the mixer 408. The mixer 408 and the analysis units 402, 404, and 406 may receive an inquiry from the game software program 410, and may be configured to interrupt the game software in response to an event. Events may include gesture recognition events, gearing changes, configuration changes, noise level settings, sampling rate settings, mapping chain changes, and the like. These examples will be described later. Mixers 408 may work together to form the following fields and fields for the methods described herein.

  As described above, signals from different input channels such as inertial sensors, video images and / or acoustic sensors, for example, are analyzed by the inertial analysis unit 402, the image analysis unit 404 and the acoustic analysis unit 406, respectively, to obtain the present invention. The movement and direction of the controller 110 during video game play according to the method may be determined. Such a method may be implemented as a series of processors capable of executing program code instructions stored on a processor readable medium and executed on a digital processor. For example, as shown in FIG. 5A, the video game system 100 may include a console 102 having an inertial analysis unit 402, an image analysis unit 404, and an acoustic analysis unit 406 implemented in either hardware or software. Good. By way of example, the analyzers 402, 404 and 406 may be implemented as software instructions that run on a suitable processor 502. By way of example, the processor 502 may be a digital processor, such as, for example, a microprocessor commonly used in video game consoles. A part of the command may be stored in the memory 506. Alternatively, the inertial analysis unit 402, the image analysis unit 404, and the acoustic analysis unit 406 may be implemented as hardware such as an application specific integrated circuit (ASIC). Such analysis hardware may be provided on the controller 110 or console 102 or may be provided at a remote location. In a hardware implementation, the analyzers 402, 404, 406 can receive external signals from the processor 502 or external signals from, for example, a USB cable, a wireless connection or other remote source connected via a network. Accordingly, it may be programmable.

  Inertial analyzer 402 may include or implement instructions to analyze the signal generated by inertial sensor 112 and may utilize information regarding the position and / or orientation of controller 110. Similarly, the image analysis unit 404 may implement a command for analyzing an image captured by the imaging device 114. Furthermore, the acoustic analysis unit may implement a command for analyzing an image captured by the microphone array 118. As shown in the flow diagram 510 of FIG. 5B, these signals and / or images may be received by the analyzers 402, 404, 406, as indicated by block 512. The signals and / or images are analyzed by the analyzers 402, 404, 406 to obtain inertial tracking information 403, image tracking information 405, and acoustic tracking information 407 regarding the position and / or orientation of the controller 110 as indicated by block 514. You may decide. The tracking information 403, 405 and 407 is associated with one or more degrees of freedom. Preferably 6 degrees of freedom are tracked to characterize the operation of the controller 110 or other tracked object. Such degrees of freedom may be related to controller tilt, yaw, roll, and position, velocity or acceleration in the X-, Y- and Z-axis directions.

  As indicated by block 516, the mixer 408 combines the inertia information 403, the image information 405, and the acoustic information 407 to generate accurate position information and / or direction information 409. As an example, the mixer 408 may give different weights to the inertia information 403, the image information 405, and the acoustic information 407 based on the game and environmental conditions, and obtain the load average. Furthermore, the mixer 408 has its own mixer analysis unit, analyzes the combined position / direction information, and generates unique “mixer” information including the combination information generated by the other analysis units. Good.

  In the embodiment of the present invention, the mixer 408 may assign distribution values to the tracking information 403, 405, 407 from the analysis units 402, 404, 406. As described above, a part of the input control data set may be averaged. However, in the present embodiment, the input control data is assigned a value considering the importance of analysis before being averaged.

  The mixer 408 can have many functions in the context of the present system, including observation, correction, stabilization, derivation, combination, routing, mixing, reporting, buffering, interrupting other processes, and analysis. Such a function can be performed on tracking information 403, 405, 407 received from one or more analysis units 402, 404, 406. A mixer to optimize the use of the received tracking information 403, 405, 407 and generate accurate tracking information 409 while each of the analyzers 402, 404, 406 receives and / or derives specific tracking information 408 may be implemented.

  The analyzers 402, 404, 406 and the mixer 408 are preferably configured to provide tracking information having a similar output format. Tracking information parameters from any of the analysis units 402, 404, 406 can be mapped to a single parameter in the analysis unit. Alternatively, the mixer 408 processes one or more tracking information parameters from one or more analyzers 402, 404, 406 to create tracking information for any of the analyzers 402, 404, 406. May be. The mixer may combine multiple elements of tracking information having the same type of parameters obtained from the analyzers 402, 404, 406 and / or functions across multiple parameters of tracking information generated by the analyzer May be performed to produce a composite output set having the advantages of being generated from multiple input channels.

  Accurate tracking information 409 can be utilized during video game play with system 100, as indicated by block 518. In some embodiments, location information and / or direction information may be utilized in relation to user 108 gestures during game play. In some embodiments, the mixer 408 operates in conjunction with the gesture recognizer 505 to associate at least one action in the gaming environment with one or more actions from the user (eg, controller operation in space). May be.

  As shown in the flow diagram 520 of FIG. 5C, the path of the controller 110 may be tracked using position information and / or direction information, as indicated by block 522. By way of example and without giving any limitation, this path may include a set of points that indicate the center of mass position of the controller for several coordinate systems. Each position point may be represented by one or more coordinates, for example, X, Y, Z coordinates in a Cartesian coordinate system. Each point on the path may be associated with a time of day, so that both the shape of the path and the progress of the controller along the path can be monitored. Further, each point of the set may be associated with data representing the direction of the controller, such as one or more rotation angles about the center of mass of the controller. Further, each point on the path may be associated with a value of the speed and acceleration of the center of mass of the controller, and the angular rotation rate and angular acceleration rate of the controller around the center of mass.

  As shown in block 524, the tracking path is compared to one or more stored paths corresponding to known gestures and / or pre-recorded gestures 508 related to the status of the video game being played. May be. The recognition unit 505 may be configured to recognize a user or process a voice authentication gesture or the like. For example, the user may be specified by the recognition unit 505 through a gesture unique to the user. Such specific gestures are recorded and included in pre-recorded gestures 508 stored in memory 506. In the recording process, the sound generated during the recording of the gesture may be optionally stored. The measurement environment is sampled and processed by a multi-channel analysis unit. The processor can refer to the gesture model and determine, authenticate, and / or identify the user or object with high accuracy and efficiency based on the voice pattern or the acoustic pattern.

  As shown in FIG. 5A, data representing a gesture 508 is stored in memory 506. Examples of gestures include throwing something like a ball, shaking something like a bat or golf club, pushing a hand pump, opening or closing a door or window, turning a handle or other vehicle control, a fighter punching, etc. , Sanding motion, waxing and wiping, home paint, vibration, chatting, roll, football throwing, knob turning motion, 3D mouse motion, scrolling, motion with known contour, recordable Any movement, back-and-forth movement along any vector, i.e. tire inflation along any direction in space, movement along a path, movement with precise stop and start times, noise floor and Any time based on user operations that can be recorded, tracked and repeated within a spline, etc., but is not limited to these. Yes. Each of these gestures can be pre-recorded from route data and saved as a time-based model. The comparison between the path and the stored gesture starts from the assumption about the stable state that when the path deviates from the stable state, the path is compared with the stored gesture of removal processing. If there is no match at block 526, the analyzer can return to block 522 to continue the path tracking of the controller 110. If there is a sufficient match between the path (or a portion thereof) and the saved gesture, the game state can be changed, as shown at 528. Changes in the game state may include, but are not limited to, interruption, transmission of control signals, changes in variables, and the like.

  Here is one possible example. If the controller 110 determines that the path is out of the stable state, the analysis unit 402, 404, 406, or 412 tracks the movement of the controller 110. While the path of the controller 110 matches the path defined in the stored gesture model 508, these gestures may be “hits”. If the path of the controller 110 deviates from any gesture model 508 (within the noise immunity setting), that gesture model is removed from the hit list. Each gesture reference model has a time axis on which gestures are recorded. The analysis unit 402, 404, 406, or 412 compares the controller path data with the stored gesture 508 with an appropriate time index. Resets the clock when a steady state condition occurs. When out of steady state (ie, movement is tracked outside of the noise threshold), the hit list will contain all possible gesture models. The clock starts and the controller movement is compared to the hit list. Again, it is easy to compare in time series. If any gesture in the hit list reaches the end of the gesture, it is a hit.

  In certain embodiments, the mixer 408 and / or the individual analyzers 402, 404, 406, and 412 may notify the game program when a particular event occurs. Examples of such events include the following:

・ Suspend when reaching zero acceleration (X-axis, Y-axis, and / or Z-axis)
If the acceleration of the controller changes at the inflection point in a certain game situation, the analysis unit can notify the routine in the game program or interrupt the routine. For example, the user 108 can use the controller 110 to control a game avatar that represents a quarterback in a football simulation game. The analyzer may track the controller (representing football) via a path generated from the signal from the inertial sensor 112. A certain acceleration change in the controller 110 may signal that a football has been thrown. At this time, the analyzer can trigger another routine in the program (eg, a physics simulation package) to simulate a football trajectory based on the controller's position, velocity and / or direction at the throw point.

・ Suspend on recognition of new gesture

  The analysis unit may be configured with one or more inputs. Examples of such input are shown below, but are not limited thereto.

-Noise level setting (X-axis, Y-axis or Z-axis)
The noise level may be a reference tolerance when analyzing a user's hand tremor during a game.

-Sampling rate setting In this specification, a sampling rate means the sampling frequency of the signal from an inertial sensor by an analysis part. The sampling rate may be set to signal oversampling or averaging.

Gearing setting As used herein, gearing generally refers to the ratio of controller movement to movement occurring in the game. An example of such “gearing” in the context of video game control is disclosed in US patent application Ser. No. 11 / 382,040 (Attorney Docket SONYP058D) dated May 7, 2006, the disclosure of which is incorporated by reference. Incorporated herein by reference.

-Setting of mapping chain In this specification, a mapping chain means a map of a gesture model. A gesture model map is created for a particular input channel (eg, for path data generated solely from inertial sensor signals) or for a hybrid channel formed in a mixer device. Three input channels are provided by two or more different analyzers similar to inertial analyzer 402. Specifically, these include inertial analysis unit 402 described herein, eg, “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF” in US patent application Ser. No. 11 / 382,034, the disclosure of which is incorporated herein by reference. A GAME CONTROLLER BODY "(Attorney Docket No. SCE05082US00), Video Analysis Department disclosed by inventor Gary M Zalewski, and US Patent Application No. 11 / 381,721, the disclosure of which is incorporated herein by reference, for example. The acoustic analysis unit disclosed in the issue may be included. These analysis units can be configured using a mapping chain. The mapping chain can be exchanged by the game during game play so that settings for the analyzer and mixer can be made.

  Returning again to FIG. One skilled in the art will appreciate that there are many ways to generate a signal from inertial sensor 112. In particular, a few examples will be described here. Referring to block 514, there are many ways to analyze the sensor signal generated at block 512 to obtain tracking information related to the position and / or orientation of controller 110. By way of example and without limitation, the tracking information includes, but is not limited to, information about the following parameters individually or in any combination.

Controller Direction The direction of the controller 110 may be represented by a pitch, roll, or yaw angle (eg, radians) with respect to a certain reference direction. The rate of change in the controller direction (eg, angular velocity or angular acceleration) may be included in the position information and / or direction information. For example, if the inertial sensor 112 includes a gyroscope sensor, the controller direction information can be directly determined in the form of one or more output values proportional to the pitch angle, roll angle, or yaw angle.

Controller position (eg, Cartesian coordinates X, Y, Z of controller 110 in a certain reference frame)
・ Controller X-axis speed ・ Controller Y-axis speed ・ Controller Z-axis speed ・ Controller X-axis acceleration ・ Controller Y-axis acceleration ・ Controller Z-axis acceleration

  Note that for position, velocity, and acceleration, position information and / or direction information may be expressed in a coordinate system other than Cartesian. For example, a cylindrical coordinate system or a spherical coordinate system may be used for position, velocity, and acceleration. The acceleration information for the X axis, the Y axis, and the Z axis may be obtained directly from an accelerometer type sensor as described in the present specification, for example. In order to determine changes in X speed, Y speed, and Z speed, X acceleration, Y acceleration, and Z acceleration may be integrated with respect to time from a certain initial time. These speeds may be calculated by adding a speed change to the X speed, Y speed, and Z speed, which are known values at the initial time. To determine the X displacement, Y displacement and Z displacement of the controller, the X velocity, Y velocity and Z velocity may be integrated over time. The X position, Y position, and Z position may be determined by adding displacement to the known X position, Y position, and Z position at the initial time.

・ Stable state Y / N
This special information indicates whether the controller is in a stable state, can be defined as any position, and also varies. In a preferred embodiment, the stable position may be such that the controller is approximately level with the user's waist and is held approximately horizontally.

Time since last stable state “Time since last stable state” generally refers to data related to the length of the period that has elapsed since the last time a stable state (as described above) was detected. The time determination may be calculated in real time, processing cycle, or sampling period, as noted previously. The data time of “time from the previous stable state” is important for resetting the controller tracking with respect to the initial point to ensure the accuracy of the character or object mapping in the game environment. This data is also important for determining available actions and / or gestures that may be subsequently (exclusively or comprehensively) performed in the gaming environment.

Recognized last gesture “Recognized last gesture” generally refers to the last gesture recognized by the gesture recognizer 505 (which can be implemented in hardware or software). The identification of the last recognized gesture is important to the fact that the previous gesture may be related to gestures that may be recognized later or other actions that occur in the gaming environment.

・ Last gesture time recognized

  The above output can be sampled at any time by a game program or software.

  In one embodiment of the present invention, the mixer 408 may assign distribution values to the tracking information 403, 405, 407 from the analysis units 402, 404, 406. As described above, a particular set of input control data may be averaged. However, in this embodiment, the input control data is assigned a value before being averaged, so that the input control data from some analysis units is analytically more than the data from other analysis units. It is important.

  For example, the mixer 408 may request tracking information related to acceleration and steady state. Mixer 408 then receives tracking information 403, 405, 407 as described above. The tracking information may include parameters related to acceleration and a stable state as described above, for example. Prior to averaging the data representing this information, the mixer 408 may assign distribution values to the tracking information data sets 403, 405, 407. For example, the X acceleration parameter and the Y acceleration parameter from the inertia analysis unit 402 may be weighted with a value of 90%. However, the X acceleration data and Y acceleration data from the image analysis unit 406 need only be weighted by 10%. The tracking information 407 of the acoustic analysis unit related to the acceleration parameter is given a weight of 0%, that is, there may be no data value.

  Similarly, the Z-axis tracking information parameter from the inertia analysis unit 402 may be weighted by 10%, and the Z-axis tracking information of the image analysis unit may be weighted by 90%. The tracking information 407 of the acoustic analysis unit is again weighted by 0%, the stable state tracking information from the acoustic analysis unit 406 is weighted by 100%, and the tracking information of the remaining analysis units is weighted by 0%. May be made.

  After appropriate distribution weights are assigned, the input control data is averaged together with the weights to form a weighted average input control data set that is then analyzed by the gesture recognizer 505 and associated with a particular action in the game environment. It is done. The associated value may be predefined by the mixer 408 or a particular game title. These values may be the result of a mixer 408 that identifies specific quality data from various analyzers and thereby performs dynamic adjustment as described below. This adjustment may also be the result of building past knowledge about when specific data will have a specific value in a specific environment and / or depending on the uniqueness of a given game title.

  The mixer 408 may be configured to operate dynamically during game play. For example, when the mixer 408 receives various input control data, the corrupted data indicates that some data is continuously out of acceptable range or data quality, or that a processing error has occurred at the associated input device. You may recognize that it reflects.

  In addition, certain conditions may change in a real world environment. For example, in a home game environment, natural light increases from morning to afternoon, which causes a problem in capturing image data. Furthermore, as the day progresses, the neighborhood and the house become noisy, causing problems in capturing acoustic data. Similarly, if the user continues to play for several hours, the reflection becomes less sharp and this causes a problem in the interpretation of inertial data.

  In these examples, or in other examples where the quality of a particular form of input control data is an issue, the mixer 408 may be more or less important to the particular input control data as described above. Distribution weights may be dynamically reassigned to specific data sets from specific devices. Similarly, as the game environment changes in the course of the game, the needs for a particular game may change, and as a result, reassignment of numerical values and needs to particular input control data may be required.

  Similarly, the mixer 408 includes processing error data, data with a low processing speed, or data that is not based at all on processing errors or feedback of data generated by the gesture recognition unit 505 among the data passed to the gesture recognition unit 505. You may recognize that. In response to this feedback or recognizing these processing difficulties (eg, an error occurs when associated by the gesture recognizer 505 while the image analysis data is within acceptable limits), the mixer 408 determines which analysis You can adjust the input control data of any time from the department. The mixer 408 may further request analysis and processing of the input control data by an appropriate analyzer before passing the data, thereby re-processing the data (eg, averaging the data) and gesturing. It may be further ensured that data passed to the recognition unit 505 is effectively processed appropriately.

  In some embodiments, the mixer 408 recognizes that certain data is corrupted, invalid, or deviates from certain variables, etc., and requests certain input control data or variables associated with that data. Thus, the wrong data may be replaced, or the data may be appropriately analyzed or calculated for the necessary variables.

  In an embodiment according to the present invention, a video game system and method of the type described above can be implemented as shown in FIG. The video game system 600 includes a processor 601 and a memory 602 (for example, RAM, DRAM, ROM, etc.). Furthermore, the video game system 600 may include a plurality of processors 601 as long as parallel processing is implemented. The memory 602 includes data and game program code 604, and includes a portion configured as described above. Specifically, the memory 602 may include inertial signal data 606 including stored controller path information as described above. The memory 602 may include stored gesture data 608, such as data representing one or more gestures associated with the game program 604. A coded instruction executed by the processor 602 may include a multi-input mixer 605 configured and functioning as described above.

  The system 600 may further include known support functions 610 such as an input / output (I / O) device 611, a power supply (P / S) 612, a clock (CLK) 613, and a cache 614. The device 600 may optionally comprise a mass storage device 615 such as a disk drive, CD-ROM drive, tape drive, etc. for storing programs and / or data. The controller may optionally include a display device 616 and a user interface 618 to facilitate interaction between the controller 630 and the user. Display device 616 may be in the form of a cathode ray tube (CRT) that displays text, numbers, graph symbols and images, or a flat panel screen. User interface 618 may include a keyboard, mouse, joystick, light pen, and other devices. The user interface 618 may also include a microphone, video camera, or other signal converter to directly capture the signal to be analyzed. The processor 601, memory 602, and other components of the system 600 may exchange signals (eg, code instructions and data) with each other via a system bus 620 as shown in FIG.

  Microphone array 622 may be connected to system 600 via input / output device 611. The microphone array may comprise about 2 to about 8 microphones, preferably about 4 and separated from adjacent microphones by a distance of less than about 4 cm, preferably about 1-2 cm. Preferably, the microphones in array 622 are omnidirectional microphones. An optional imaging device 623 (for example, a digital camera) may be connected to the device 600 via the input / output device 611. One or more positioning actuators 625 mechanically connected to the camera may exchange signals with the processor 601 via the input / output device 611.

  As used herein, the term “input / output device” generally refers to any program, operation or device that sends and receives data to and from system 600 and peripheral devices. All data transmission and reception may be considered as output from one device and input to another device. Peripheral devices may include devices such as a writable CD-ROM that can operate as both an input device and an output device, in addition to an input-only device such as a keyboard and mouse, and an output-only device such as a printer. The term “peripheral device” includes an external device such as a mouse, keyboard, printer, monitor, microphone, game controller, camera, external Zip driver or scanner, CD-ROM drive, CD-R drive or internal modem, etc. Internal devices, or other peripheral devices such as flash memory read and write hard drives.

  In some embodiments of the present invention, the device 600 may be a video game device comprising a controller 630 connected to the processor via an input / output device 611 that is wired (eg, USB cable) or wireless. The controller 630 may include an analog joystick control 631 that provides control signals normally used during video game play and a known button 633. Such a video game may be implemented as processor readable data and / or instructions from a program 604 stored in memory 602 or other processor readable medium associated with mass storage device 615. In some embodiments, mixer 605 can receive input from analog joystick control 631 and button 633.

  In general, the joystick control 631 may be configured to transmit a signal along the X axis when the control stick is moved left and right, and to transmit a signal along the Y axis when the control stick is moved back and forth (up and down). In a joystick configured for three-dimensional movement, a signal of movement along the Z axis may be transmitted when the stick is twisted to the left (counterclockwise) or right (clockwise). These three axes of X, Y, and Z may be referred to as roll, pitch, and yaw, respectively, particularly when related to an aircraft.

  Game controller 630 may include a communication interface that operates to perform digital communication with at least one or both of processor 602 and game controller 630. The communication interface may comprise a universal asynchronous receiver transmitter (UART). The UART may be operable to receive a control signal for controlling the operation of the tracking device or for transmitting signals from the tracking device to communicate with other devices. Alternatively, the communication interface may comprise a USB controller. The USB controller may be operable to receive control signals for controlling the operation of the tracking device or for transmitting signals from the tracking device to communicate with other devices.

  Further, the controller 630 may include one or more inertial sensors 632 that can provide position and / or direction information to the processor 601 via inertial signals. The direction information may include angle information such as tilt, roll or yaw of the controller 630. By way of example, the inertial sensor 632 may include any number of accelerometers, gyroscopes or tilt sensors, or any combination thereof. In a preferred embodiment, the inertial sensor 632 includes a tilt sensor adapted to sense the direction of the game controller 630 relative to the tilt and roll axis, a first acceleration adapted to sense acceleration along the yaw axis. And a second accelerometer adapted to sense angular acceleration relative to the yaw axis. The accelerometer may be implemented, for example, as a MEMS device that includes a mass attached with one or more springs and a sensor that detects the displacement of the mass in one or more directions. A signal from the sensor that depends on the displacement of the mass may be used to determine the acceleration of the game controller 630. Such a technique is realized by instructions from the game program 604 stored in the memory 602 and executed by the processor 601.

  By way of example, an accelerometer suitable as the inertial sensor 632 may be a single mass that is telescopically connected to the frame at 3-4 points, for example, with a spring or the like. The pitch axis and the roll axis exist on a plane that intersects the frame provided in the game controller 630. As the frame (and game controller 630) rotates about the pitch and roll axes, the mass is displaced under the influence of gravity and the springs expand and contract in a manner that depends on the pitch and / or roll angle. The displacement of the mass is detected and converted into a signal depending on the pitch amount and / or the roll amount. Angular acceleration around the yaw axis or linear acceleration along the yaw axis generates characteristic patterns of spring expansion and contraction or mass movement, which are detected and signaled according to the amount of angular acceleration or linear acceleration. Converted. Such an accelerometer device can measure tilt, roll angular acceleration about the yaw axis, and linear acceleration along the yaw axis by tracking mass movement and spring stretching force. There are many different ways to track the position of the mass and / or the forces acting on it, including resistive strain gauges, acoustic sensors, magnetic sensors, Hall effect elements, piezoelectric elements, capacitive sensors, etc. included.

  In addition, the game controller 630 may include one or more light sources 634 such as light emitting diodes (LEDs). The light source 634 is used to distinguish the controller from other controllers. For example, if there are one or more LEDs, they can be identified by flashing or holding the LED pattern code. For example, five LEDs may be provided on the game controller 630 in a straight line or a two-dimensional pattern. Although it is preferable to arrange the LEDs in a straight line, as another method, the LEDs are arranged in a square shape or an arch shape to facilitate determination of the LED arrangement screen when analyzing the LED pattern image obtained by the imaging device 623. It may be made like this. Furthermore, the LED pattern code can also be used to determine the position of the game controller 630 during game play. For example, LEDs can help identify controller tilt, yaw and roll. With this detection pattern, it is possible to provide better emotions to the user in games such as airplane games. The imaging device 623 may capture an image including the game controller 630 and the light source 634. When such an image is analyzed, the position and / or orientation of the game controller can be determined. Such analysis may be implemented by program code instructions 604 stored in memory 602 and executed by processor 601. In order to facilitate imaging of the light source 634 by the imaging device 623, the light source 634 may be disposed at a plurality of different ends of the game controller 630, for example, forward and backward (shown as shadows in FIG. 6). With such an arrangement, the imaging device 623 can obtain an image of the light source 634 in various directions of the game controller 630 depending on how the user holds the game controller 630.

  Further, the light source 634 may provide a telemetry signal to the processor 601 in a pulse code, amplitude modulation, or frequency modulation format, for example. Such a telemetry signal may represent which joystick button is pressed and / or how hard the button is pressed. The telemetry signal may be encoded into an optical signal by, for example, pulse encoding, pulse width modulation, frequency modulation, or light intensity (amplitude) modulation. The processor 601 may decode the telemetry signal from the optical signal and execute the game command according to the decoded telemetry signal. The telemetry signal may be decoded from an analysis of the game controller 630 image obtained by the imaging device 623. Alternatively, the apparatus 600 may comprise an independent optical sensor dedicated to receiving telemetry signals from the light source 634. The use of LEDs in conjunction with intensity determination in an interface with a computer program is described, for example, in “Computer Image and Audio Processing of Intensity and Input Devices for Interfacing With” in US Patent Application No. 11 / 429,414, filed May 4, 2006. A Computer Program "(attorney docket number SONYP052), inventor Richard L. Marks et al., The disclosure of which is incorporated herein by reference in its entirety. The analysis of the image including the light source 634 can also be used to determine the position and / or orientation of telemetry and game controller 630. Such a technique may be realized by the instructions of the program 604 stored in the memory 602 and executed by the processor 601.

  The processor 601 uses the inertial signal from the inertial sensor 632 together with the optical signal from the light source 634 detected by the imaging device 623 and / or the sound source location and sound source characteristic information from the acoustic signal detected by the microphone array 622. , Information regarding the location and / or orientation of the controller 630 and / or the user may be inferred. For example, the “acoustic radar” sound source location and its characteristics may be used with the microphone array 622 to track the moving sound source while game controller movement is being tracked independently (through inertial sensor 632 and / or light source 634). To do. In acoustic radar selection, a calibrated listening area is selected during execution time, and sounds generated from sound sources other than the calibrated listening area are filtered out. The calibrated listening area may include a listening area corresponding to the volume at the focus or field of view of the imaging device 623. An example of an acoustic radar is disclosed in detail in US Patent Application No. 11 / 381,724 filed May 4, 2006, “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION”, inventor Xiadong Mao. Is incorporated herein by reference. Various combinations of various forms for supplying control signals to the processor 601 may also be used in conjunction with embodiments of the present invention. Such techniques may be implemented by program code instructions 604 stored in memory 602 and executed by processor 601 and optionally select a calibrated listening area during execution time, other than the calibrated listening area. The sound generated from the sound source may include one or more instructions that direct one or more processors to be filtered out. The calibrated listening area may include a listening area corresponding to the volume at the focus or field of view of the imaging device 623.

The program 604 optionally generates a discrete time domain input signal X _m (t) from the microphones M0-MM of the microphone array 622, determines the listening sector, and selects a finite impulse response filter coefficient semi-blind. One or more instructions may be included that direct one or more processors to separate different sound sources from the input signal X _m (t) using a listening sector in sound source separation. The program 604 may further include instructions for applying one or more fractional delays to a selected input signal X _m (t) other than the input signal X ₀ (t) from the reference microphone M _0. Good. Each non-integer delay may be selected to optimize the signal to noise ratio of the discrete time domain output signal y (t) from the microphone array. Fractional delays the signal from the reference microphone M ₀ may be chosen to be the first to signals from other microphones of the array. The program 604 sets the non-integer delay Δ to y (t + Δ) = x (t + Δ) * b ₀ + x (t−1 + Δ) * b ₁ + x (t−2 + Δ) * b ₂ An instruction to be introduced into the output signal y (t) of the microphone array so that +... + X (t−N + Δ) b _N may be included. Here, Δ is 0 to ± 1. Examples of such techniques are disclosed in detail in US Patent Application No. 11 / 381,729 “ULTRA SMALL MICROPHONE ARRAY” filed May 4, 2006, inventor Xiadong Mao, the disclosure of which is hereby incorporated by reference. Incorporated herein by reference.

  The program 604, when executed, may include one or more instructions that cause the system 600 to select a pre-calibrated listening area that includes a sound source. Such an instruction may cause the device to determine whether the sound source is in the initial region or on a particular side of the region. If the sound source is not in the default area, the instruction may select another area on a particular side of the default area when executed. Another region may be characterized by the attenuation of the input signal closest to the optimal value. When executed, these instructions may calculate the attenuation of the input signal from the microphone array 622 and the attenuation to an optimum value. The instructions may be executed such that when the device 600 is executed, the apparatus 600 determines an attenuation value of the input signal for one or more regions and selects the region where the attenuation is closest to the optimal value. An example of such technology is disclosed in US Patent Application No. 11 / 381,725 filed May 4, 2006, “METHOD AND APPARATUS FOR TARGETED SOUND DETECTION”, inventor Xiadong Mao, the disclosure of which is referred to Is incorporated herein by reference.

  The signal from inertial sensor 632 provides part of the tracking information input, and the signal generated from tracking one or more light sources 634 by imaging device 623 provides another part of the tracking information input. By way of example and without limitation, such a “mixed mode” signal is used in a football-type video game where the quarterback is applied to the right after the feint moving the head to the left. Throw into. Specifically, the game player holding the controller 630 makes a voice while turning his head to the right and throwing it like a football, with his head turned to the left. Along with the “acoustic radar” program code, the microphone array 622 can track the user's voice. The imaging device 623 can track the movement of the user's head or other commands that do not require sound or a controller. Sensor 632 may track the movement of the game controller (representing football). The imaging device 623 may also track the light source 634 of the controller 630. The user can throw a “ball” when the acceleration of the game controller 630 reaches a certain amount and / or in a certain direction, or based on a key command triggered by pressing a button on the controller 630.

  In some embodiments of the present invention, inertial signals such as from accelerometers or gyroscopes may be used to position controller 630. Specifically, the acceleration signal from the accelerometer may be temporally integrated once to determine the speed change, and the speed may be time integrated to determine the position change. If the initial position and speed at a certain time are known, the absolute position can be determined using these values, speed and position change. Although the position determination using the inertial sensor can be performed faster than the method using the imaging device 623 and the light source 634, the inertial sensor 632 is prone to a type of error known as “drift”. This is a phenomenon in which a mismatch D may occur between the position of the joystick (shown by a shadow in FIG. 6) calculated from the inertial signal and the actual position of the game controller due to errors accumulated over time. Embodiments of the present invention can handle such errors in various ways.

  For example, the drift can be manually canceled by setting the initial position of the controller 630 to be equal to the current calculated position. The user can trigger an instruction to reset the initial position using one or more buttons on the controller 630. Alternatively, if it is an image-based drift, it can be canceled by resetting the current position to a position determined from a reference image obtained from the imaging device 623. Compensation for such image-based drift can be performed manually, for example, when the user triggers one or more buttons of game controller 630. Alternatively, compensation for image-based drift can be performed automatically, for example, at regular intervals or in response to game play. Such a technique is implemented by program code instructions 604 stored in memory 602 and executed by processor 601.

  In some embodiments, it may be desirable to correct error data in the inertial sensor signal. For example, the signal from inertial sensor 632 can be oversampled and a sliding average can be calculated from the oversampled signal to remove error data from the inertial sensor signal. In certain situations, it may be desirable to oversample the signal, remove high and / or low values from some subset of data points, and calculate a moving average from the remaining data points. Also, other data sampling techniques and data processing techniques that adjust the signal from the inertial sensor to remove or reduce the superiority from the error data can be used. Which technique is selected depends on the nature of the signal, the computation on the signal, the nature of the game play, or a combination of these. Such a technique is realized by instructions of a program 604 stored in the memory 602 and executed by the processor 601.

  As described above, the processor 601 performs analysis of the inertia signal data 606 in accordance with the data 606 and the program code instructions of the program 604 stored and retrieved in the memory 602 and executed by the processor module 601. The code portion of program 604 may be compliant with any of many different programming languages such as assembly, C ++, JAVA, or many other languages. The processor module 601 forms a general-purpose computer that becomes a dedicated computer when executing a program such as the program code 604. Here, the program code 604 is described as being embedded in software and executed by a general-purpose computer. However, for those skilled in the art, a task management method is another method such as an application specific integrated circuit (ASIC) or other It will be understood that it is implemented using hardware such as a hardware circuit. Thus, it should be understood that embodiments of the present invention, whether in whole or in part, are implemented in software, hardware, and a combination of both.

  In some embodiments, among other things, the program code 604 comprises a processor readable instruction set that implements a method having features common to the method 510 of FIG. 5B and the method 520 of FIG. 5C, or combinations thereof. Also good. Program code 604 typically includes one or more instructions that direct one or more processors to analyze signals from inertial sensor 632 and generate position and / or direction information for video game play. Use the information inside.

  When executed, the program code 604 causes the imaging device 623 to monitor the front field of view, identify one or more light sources 634 within the field of view, detect changes in light from the light sources 634, and detect changes. In response, a processor executable instruction including one or more instructions that trigger an input instruction to the processor 601 may be optionally provided. Using an LED with an imaging device to trigger the action of a game controller is described in US Patent Application No. 10 / 759,782, filed January 16, 2004, "METHOD AND APPARATUS FOR LIGHT INPUT DEVICE", inventor Richard L. Marks, the disclosure of which is hereby incorporated by reference.

  When executed, program code 604 includes one or more instructions that, as described above, use a signal from an inertial sensor and a signal from an imaging device that tracks one or more light sources as inputs to the game system. Processor-executable instructions may optionally be included. Program code 604 may optionally comprise processor-executable instructions that, when executed, include one or more instructions that compensate for inertial sensor 632 drift.

  Further, the program code 604 may optionally comprise processor-executable instructions that, when executed, include one or more instructions that adjust the gearing and mapping of controller operations to the gaming environment. Such a feature allows the user to change the “gearing (ratio)” of the operation of the game controller 630 relative to the game state. For example, when the game controller 630 is rotated by 45 °, it matches the 45 ° rotation of the game object. However, the 1: 1 transmission ratio can be changed such that the controller's X ° rotation (or tilt, yaw, “operation”) changes to a Y ° rotation (or tilt, yaw, “operation”) of the game object. The transmission ratio can be 1: 1, 1: 2, 1: X or X: Y. Here, X and Y are arbitrary values. Furthermore, the mapping of input channels to the game controller can also change over time or instantly. The change includes a change of a gesture trajectory model, a change of a position, a scale, a gesture threshold, and the like. Such mapping may be programmed, random, tiered or staggered to provide operational dynamic range to the user. ). Changes in mapping, gearing or transmission ratio are adjusted by game program 604 depending on game play, game state via user change button (such as keypad) on game controller 630, or broadly depending on input channel Is done. Input channels include user sound, controller generated sound, controller generated sound tracking, controller button state, video camera output, and accelerometer data, tilt, yaw, roll, position, acceleration, and user or It can include, but is not limited to, controller remote control data including data from any other sensor that can track an object that the user operates.

  In some embodiments, the game program 604 can change the mapping or gearing according to elapsed time from one scheme, or the transmission ratio to another scheme, respectively, in a predetermined time-dependent manner. Gearing and mapping changes can be applied to the game environment in various ways. In one example, the video game character controls the character under a single gearing scheme when the character is healthy, and as the character's health is compromised, the video game character may adversely affect the controller's movement to communicate commands to the character. In addition, the system can adapt the controller command. Video game characters who have lost their sense of direction must change their input channel mapping as a user and may require, for example, input adjustments to regain control of the character under the new mapping. The mapping scheme that changes the conversion of input channels to game commands may change during game play. This conversion occurs in various ways depending on the game state or in response to a modification command issued under one or more elements of the input channel. Gearing and mapping may be configured to affect the configuration and / or processing of one or more elements of the input channel.

  In addition, a sound emitter 636 such as a speaker, buzzer, horn or pipe may be attached to the joystick controller 630. In some embodiments, the sound emitter may be removably attached to the “body” of the joystick controller 630. In an “acoustic radar” embodiment where the location of the sound detected by the microphone array 622 is identified and characterized by the program code 604, the sound emitter 636 can be detected by the microphone array 622 and used in the program code 604. An acoustic signal that tracks the position of the game controller 630 may be provided. Sound emitter 636 may also be used to provide an additional “input channel” from game controller 630 to processor 601. The acoustic signal from the sound emitter 636 may be periodically pulsed to provide a beacon for the acoustic radar to track the location. The acoustic signal (pulsed or otherwise) may be audible or ultrasonic. The acoustic radar tracks the user's operation of the game controller 630, where such operation tracking includes information about the position and direction of the game controller 630 (eg, pitch angle, roll angle, yaw angle, etc.). May be. The pulse may be triggered at an appropriate duty cycle applicable by those skilled in the art and may be initiated based on a control signal adjusted from the system. System 600 may coordinate the transmission of control signals between multiple joystick controllers 630 connected to processor 601 (via program code 604) to ensure that multiple controllers can be tracked.

  In some embodiments, the mixer 605 is configured to obtain inputs for controlling execution of the game program 604 using inputs received from known controls of the game controller 630, such as analog joystick controls 631 and buttons 633. May be. Specifically, the reception of the mixer 605 may be reception of controller input information from the controller 630. The controller input information includes: a) information specifying the current position of the control stick that can be moved by the user of the game controller in relation to the stop position of the control stick; or b) whether the switch included in the game controller is operating. It may include at least one of information specifying whether or not. The mixer 605 may further receive additional input information from the environment in which the controller 630 is used. By way of example and without limitation, additional input information may include i) information obtained from an imaging device in the environment (eg, imaging device 623), and / or ii) at least one of a game controller or user. And / or iii) acoustic information obtained from acoustic transducers in the environment (eg, acoustic information from microphone array 622, and possibly sound emitter 636). In addition to the acoustic signal generated in step 1).

The controller input information may further include information identifying whether or not the pressure sensitive button is activated.
The mixer 605 may obtain a combination input for controlling execution of the game program by processing the controller input information and the additional input information to obtain a combination input.

  The combination inputs may include individual fused inputs that control each individual function during execution of the game program 604. At least some of the individual fused inputs can be determined by fusing controller input information associated with a particular individual function with additional input information associated with a particular individual function. The combination input includes a fused input that controls the function during execution of the game program 604. At least a portion of the fused inputs can be determined by fusing controller input information associated with the function with additional input information associated with the function. In such a case, the fusion may be performed by averaging the value representing the controller input information and the value representing the additional input information. As an example, the value of the controller input information may be averaged at a ratio of 1: 1 with the value of the additional input information. Alternatively, different weights may be assigned to the controller input information and the additional input information, and an average may be obtained as a weighted average of the controller input information value and the additional input information value according to the assigned weight. .

  In some embodiments, the value of the first of the controller input information or additional input information is modified to control the still active function activated in response to at least one of the controller input information or the second of the additional input information It can be used as a correction input for the game program to be performed. The additional input information may include inertial sensor information obtained by actuation of the inertial sensor 632 and / or direction information representing the direction of the movable object of the user. Alternatively, the additional input information may include information indicating at least one of the position or direction of the movable object of the user. In the present specification, the movable object of the user may refer to the controller 630 or an object attached to the controller 630 main body. The additional input information may include information indicating the direction of the movable object of the user. For example, such direction information may include information indicating at least one of pitch, yaw, and roll.

  In some embodiments, by fusing the value of controller input information representing the position of a control stick (eg, one of the analog sticks 631) with the value of additional input information representing the direction of the user's movable object. The combination input may be obtained. As described above, the user's movable object may be an object attached to the game controller 630 and / or the game controller 630. When the control stick is moved backward while increasing the pitch to a positive value (rising), the combined input may reflect the increased pitch up input. Similarly, when the control stick is moved forward while decreasing the pitch to a negative value (down), the combined input may reflect the increased pitch down input.

  By assigning a value of controller input information that represents the position of the control stick as rough control information, and assigning a value of additional input information that represents the direction of an object that the user can move as fine control information, a combination input can be obtained. Good. Alternatively, controller input information that specifies whether or not the switch of the game controller is operating is assigned as rough control information, and additional input information that represents the direction of an object that the user can move as fine control information The combination input may be obtained by assigning a value of. Further, a combination input is obtained by assigning a value of additional input information indicating the direction of an object movable by the user as rough control information and assigning a value of controller input information indicating the position of the control stick as fine control information. May be. Furthermore, a value of controller input information that specifies whether or not the switch of the game controller is operating is assigned as fine control information, and a value of additional input information that represents the direction of the object that the user can move as rough control information A combination input may be obtained by assigning. In any or all of these cases, the combined input may represent a rough control information value adjusted by a relatively small amount in response to fine control information.

In some embodiments, the combination input is determined by additively combining the value represented by the controller input information and the value represented by the additional input information so that the combination input is either controller input information or A signal having a value smaller than any one of the values of the additional input information may be provided. Alternatively, the combination input may provide the game program with a signal having a smoothed value with a gentler time change than the single value of either the controller input information or the additional input information. . The combination input may provide a high resolution signal with increased signal content to the game program. The high resolution signal may be subjected to more rapid time changes than either the controller input information or the value of the additional input information alone.
Although embodiments of the present invention have been described as examples associated with a video game controller 630, embodiments of the present invention, including system 600, have inertial sensing capabilities, and inertial sensor signals, whether wireless or otherwise. It can be used for any of a main body, a molded product, a knob, or a structure operated by a user having transmission capability.

  By way of illustration, embodiments of the invention can be implemented in a parallel processing system. Such parallel processing systems typically include a plurality of processors configured to execute portions of a program in parallel using separate processors. By way of example and without limitation, FIG. 7 illustrates one type of cell processor 700 according to an embodiment of the present invention. The cell processor 700 can be used as the processor 601 in FIG. 6 or the processor 502 in FIG. 5A. In the example of FIG. 7, the cell processor 700 includes a main memory 702, a power processor element (PPE) 704, and a number of synergistic processor elements (SPE) 706. In the example of FIG. 7, the cell processor 700 includes one PPE 704 and eight SPEs 706. In such a configuration, seven SPEs 706 are used for parallel processing, and one is reserved as a backup when one of the other seven fails. Alternatively, the cell processor may include a plurality of groups consisting of a plurality of PPEs (PPE groups) and a plurality of groups consisting of a plurality of SPEs (SPE groups). Each unit shares hardware resources. However, SPE and PPE need to appear to be independent elements to the software. Thus, the embodiment of the present invention is not limited to use in the configuration shown in FIG.

  Main memory 702 is typically a dedicated hardware register or array used for functions such as system configuration, data synchronous transfer, memory mapped I / O, and I / O subsystems in addition to general purpose non-volatile storage. Also equipped. In an embodiment of the present invention, the video game program 703 may reside in the main memory 702. Video program 703 may comprise an inertial analysis unit, an image analysis unit, an acoustic analysis unit, and a mixer as described above for FIGS. 4, 5A, 5B and 5C, or a combination thereof. The program 703 may be divided into a plurality of signal processing tasks executed at the SPE and / or PPE.

  By way of example, PPE 704 may be a 64-bit power PC processor unit (PPU) with associated caches L1 and L2. The PPE 704 is a general-purpose processing device that can access system management resources (for example, a memory protection table). Hardware resources may be explicitly mapped to the real address space as seen by the PPE. Therefore, the PPE can address any of these resources directly using the appropriate effective address value. The main functions of the PPE 704 are task management and task assignment for the SPE 706 of the cell processor 700.

  In FIG. 7, only one PPE is shown, but in a cell processor implementation such as Cell Broadband Engine Architecture (CBEA), the cell processor 700 is divided into a plurality of PPE groups organized into a plurality of PPE groups. You may have PPE. These PPE groups may share access to the main memory 702. Further, the cell processor 700 may include two or more SPEs. SPE groups may also share access to main memory 702. Such a configuration is also within the scope of the present invention.

  Each SPE 706 includes an SPU and a unique local storage area LS. The local storage area LS includes one or more separation areas in the memory storage, and each area is associated with a dedicated SPU. Each SPU may be configured to execute only instructions (including data load and data save operations) from within its own associated storage area. In such a configuration, data transfer between the local storage area LS and the rest of the system 700 is performed by issuing a direct memory access (DMA) instruction from the memory flow controller (MFC) (for each individual SPE). ) Data is transferred to and from the local storage area. The SPU that does not perform any system management function is a computer device having a simpler structure than the PPE 704. SPUs typically have SIMD (single instruction, multiple data) functions, usually perform data processing and initiate the requested data transfer (according to the access characteristics set by the PPE) to perform the assigned task To do. The purpose of the SPU is to enable applications that require higher computer device densities so that the provided instruction set can be used effectively. A significant number of SPEs in systems managed by PPE 704 allow for cost-effective processing in a wide range of applications.

  Each SPE 706 includes a dedicated memory flow controller (MFC) with an associated memory management unit that can hold and process memory protection information and access permission information. The MFC provides the main method for data transfer, protection and synchronization between the main storage of the cell processor and the local storage of the SPE. The MFC instruction describes the transfer to be performed. Data transfer instructions may be referred to as MFC direct memory access (DMA) instructions (or MFCDMA instructions).

  Each MFC may support multiple DMA transfers simultaneously and maintain and process multiple MFC instructions. Each MFCDMA data transfer instruction request may be accompanied by both a local storage address (LSA) and an effective address (EA). A local storage address can be addressed directly only to the local storage address of the associated SPE. The effective address may comprise a more general application, for example, if it can be renamed to the real address space, it can refer to the main memory including all SPE local storage areas.

  To facilitate communication between SPEs 706 and / or between SPE 706 and PPE 704, SPE 706 and PPE 704 may include signal notification registers for signaling events. PPE 704 and SPE 706 may be connected by a star topology that acts like a router where PPE 704 sends messages to SPE 706. Alternatively, each SPE 706 and PPE 704 may have a one-way signal notification register called a mailbox. The mailbox is used by the SPE 706 to host OS synchronization operations.

  The cell processor 700 may include an input / output (I / O) function that can interface with peripheral devices such as a microphone array 712, an optional imaging device 713, and a game controller 730. The game controller unit may include an inertial sensor 732 and a light source 734. Furthermore, the element interconnection bus 710 can connect the various parts listed above. Each SPE and PPE can access the bus 710 via a bus interface unit (BIU). The cell processor 700 controls the data flow between the I / O 708 and the bus 710, the two controllers normally found in the processor, the memory interface controller (MIC) that controls the data flow between the bus 710 and the main memory 702. A bus interface controller (BIC). Although the requirements for MIC, BIC, BIU and bus 710 vary greatly from implementation to implementation, those skilled in the art will be familiar with their functionality and implementation circuitry.

  The cell processor 700 may include an internal interrupt controller (IIC). The IIC device manages the priority of interrupts presented to the PPE. The IIC can handle interrupts from other elements of the cell processor 700 without using the main system interrupt controller. The IIC can also be thought of as a second level controller. The interrupt controller of the main system may process an interrupt generated outside the cell processor.

  In embodiments of the invention, computations such as the non-integer delays described above may be performed in parallel using PPE 704 and / or one or more SPEs 706. Each non-integer delay calculation may be performed as one or more separate tasks undertaken by different available SPEs 706 separately.

  While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various modifications, changes and equivalents. Accordingly, the scope of the invention should not be determined with reference to the above content, but instead should be determined in conjunction with the appended claims and their full scope of equivalents. . All suitable or otherwise described features herein may be combined with any other suitable or otherwise described features herein. In the claims, the indefinite article "A" or "An" refers to one or more quantities of an item that follows the article, unless expressly stated otherwise. The appended claims should not be construed to include means plus function limitations, unless such limitations are expressly made using the phrase “means for”.

This application is filed as “ULTARA SMALL MICROPHONE ARRAY” filed May 4, 2006, US Patent Application No. 11 / 381,729, inventor Xiao Dong Mao; “ECHO AND NOISE” filed May 4, 2006 CANCELLATION ”, US patent application No. 11 / 381,728, inventor Xiao Dong Mao; filed on May 4, 2006“ METHOD AND APPARATUS FOR TARGETED SOUND DETECTION ”, inventor Xiao Dong Mao; filed on May 4, 2006 “NOISE REMOVAL FOR ELECTRONIC DEVICE WITH FAR FIELD MICROPHONE CONSOLE”, US patent application No. 11 / 381,727, inventor Xiao Dong Mao; “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION” filed May 4, 2006, US patent Application No. 11 / 381,724, inventor Xiao Dong Mao; and “SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACIVE PROCESSING”, filed May 4, 2006, US patent application No. 11 / 381,721, inventor Xiao Dong Mao Claiming priority over this, the entire disclosure of which is incorporated herein by reference. It is.

  This application is “METHODS AND APPARATUS FOR ADJUSTING A LISTENING AREA FOR CAPTURING SOUNDS” filed May 4, 2006, co-pending US patent application No. 11 / 418,988, inventor Xiao Dong Mao; filed May 4, 2006 "METHODS AND APPARATUSES FOR CAPTURING AN AUDIO SIGNAL BASED ON VISUAL IMAGE", co-pending US patent application No. 11 / 418,989, inventor Xiao Dong Mao; "METHODS AND APPARATUS FOR CAPTURING AN AUDIO SIGNAL" filed May 4, 2006 BASED ON A LOCATION OF THE SIGNAL ”, co-pending application No. 11 / 429,047, inventor Xiao Dong Mao;“ SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING ”, filed May 4, 2006, co-pending application No. 11 No. 429,133, inventor Richard Marks et al .; and “Computer Image and Audio Processing of Intensity and Input Devices for Interfacing With A Computer Program” filed May 4, 2006, co-pending application No. 11 / 429,414, invention Claiming priority to Richard Marks et al. The indications are incorporated herein by reference.

  This application further includes “MULTI-INPUT GAME CONTROL MIXER” filed on May 6, 2006, US Patent Application No. 11 / 382,031; “SYSTEM FOR TRACKING USER MANIPULATIONS WITHIN AN ENVIRONMENT” filed May 6, 2006, US Patent Application No. 11 / 382,032; “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUT CONTROL”, filed May 6, 2006, US Patent Application No. 11 / 382,033; filed May 6, 2006 INERTIALLY TRACKABLE HAND-HELD CONTROLLER ”, US Patent Application No. 11 / 382,035;“ METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TO VISUAL TRACKING ”, US Patent Application No. 11 / 382,036, filed May 6, 2006; “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TO INERTIAL TRACKING” filed on May 7, US patent application No. 11 / 382,041; and “SYSTEM FOR APPLYING GEARING EFFECTS TO ACOUSTIC TRACKING” filed May 6, 2006, US patent application 11 / 382,038; “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TO MULTI-CHANNEL MI” filed May 7, 2006 "XED INPUT", US Patent Application No. 11 / 382,040; "SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY", filed May 6, 2006, US Patent Application No. 11 / 382,034; May 6, 2006 “SCHEME FOR TRANSLATING MONEMENTS OF A HAND-HELD CONTROLLER INTO INPUTS FOR A SYSTEM”, US Patent Application No. 11/382, O37; “DETECTABLE AND TRACKABLE HAND-HELD CONTROLLER” filed May 7, 2006, US Patent Application No. 11 / 382,043; “METHOD FOR MAPPING MOVEMENTS OF A HAND-HELD CONTROLLER TO GAME COMMANDS” filed May 7, 2006; US Patent Application No. 11 / 382,039; filed May 6, 2006 “CONTROLLER WITH INFRARED PORT”, US Design Application No. 29 / 259,349; “CONTROLLER WITH TRACKING SENSORS” filed on May 6, 2006; US Design Application No. 29 / 259,350; “DYNAMIC” filed on May 6, 2006 TARGET INTERFACE ”, US Patent Application No. 60 / 798,031; and“ OBTAINING INPUT FOR CONTROLLING EXECUTION OF A GAME PROGRAM ”filed May 8, 2006. Claims priority to U.S. Patent Application No. 11 / 382,250, the disclosure of which is incorporated herein by reference.

  This specification is further directed to “SYSTEM AND METHOD FOR USING USER'S VISUAL / ADIOU ENVIRONMENT TO SELECT ADVERTISING” filed May 8, 2006, US Patent Application No. 11 / 430,594, inventors Gary Zalewski and Riley R. Russell. And claims the priority, the disclosure of which is incorporated herein by reference.

  This specification further takes precedence over “Using Audio / Visual Environment To Select Ads On Game Platform” filed May 8, 2006, US Patent Application No. 11 / 430,593, inventors Gary Zalewski and Riley R. Russell. And the disclosure of which is incorporated herein by reference.

  This specification further describes `` Method and apparatus for use in determining lack of user activity in relation to a system '' filed May 8, 2006, U.S. co-pending application No. 11 / 382,259, inventor Gary Zalewski et al. The disclosure of which is hereby incorporated by reference.

  This specification further describes “Method and apparatus for use in determining an activity level of a user in relation to a system” filed May 8, 2006, US Patent Application No. 11 / 382,258, inventor Gary Zalewski et al. And the disclosure of which is incorporated herein by reference.

  This specification further gives priority to “Hand-held controller having detectable elements for tracking purposes” filed May 8, 2006, US co-pending application No. 11 / 382,251, inventor Gary Zalewski et al. And the disclosure of which is incorporated herein by reference.

  This specification further claims the priority of “TRACKING DEVICE FOR USE IN OBTAINING INFORMATION FOR CONTROLLING GAME PROGRAM EXECUTION” filed on May 8, 2006, US co-pending application No. 11 / 382,252, and its disclosure. Is incorporated herein by reference.

  This specification further claims priority to "TRACKING DEVICE WITH SOUND EMITTER FOR USE IN OBTAINING INFORMATION FOR CONTROLLING GAME PROGRAM EXECUTION", filed May 8, 2006, US co-pending application 11 / 382,256, The disclosure of which is hereby incorporated by reference.

  This specification further claims priority to “VIDEO GAME CONTROLLER FRONT FACE” filed May 8, 2006, US co-pending application No. 29 / 246,744, the disclosure of which is incorporated herein by reference. Incorporated.

  This specification further claims priority to “VIDEO GAME CONTOLLER”, US Design Application No. 29 / 246,743, filed on May 8, 2006, the disclosure of which is incorporated herein by reference. .

  This specification further claims priority to “VIDEO GAME CONTROLLER” filed May 8, 2006, US copending design application No. 29 / 246,767, the disclosure of which is incorporated herein by reference. Is done.

  This specification further claims priority to “VIDEO GAME CONTROLLER” filed on May 8, 2006, US co-pending design application No. 29 / 246,768, the disclosure of which is incorporated herein by reference. Is done.

  This specification further claims priority to “ERGONOMIC GAME CONTROLLER DEVICE WITH LEDS AND OPTICAL PORTS” filed May 8, 2006, US co-pending design application No. 29 / 246,763, the disclosure content of which is referred to. Is incorporated herein by reference.

  This specification further claims priority to “GAME CONTROLLER DEVICE WITH LEDS AND OPTICAL PORTS” filed May 8, 2006, US co-pending design application No. 29 / 246,759, the disclosure of which is incorporated herein by reference. Incorporated herein by reference.

  This specification further claims priority to “DESIGN FOR OPTICAL GAME CONTROLLER INTERFACE” filed May 8, 2006, US copending design application No. 29 / 246,765, the disclosure of which is hereby incorporated by reference. Incorporated into the book.

  This specification further claims priority to “DUAL GRIP GAME CONTROLLER DEVICE WITH LEDs AND OPTICAL PORTS” filed on May 8, 2006, US co-pending design application No. 29 / 246,766, the disclosure of which Which is incorporated herein by reference.

  This specification further claims priority to “GAME INTERFACE DEVICE WITH LEDS AND OPTICAL PORTS” filed May 8, 2006, US co-pending design application No. 29 / 246,764, the disclosure of which is incorporated herein by reference. Incorporated herein by reference.

  This specification further claims priority to “ERGONOMIC GAME INTERFACE DEVICE WITH LEDS AND OPTICAL PORTS” filed May 8, 2006, US co-pending design application No. 29 / 246,762, the disclosure content of which is referred to. Is incorporated herein by reference.

This application is related to “AUDIO, VIDEO, SIMULATION, AND USER INTERFACE PARADIGMS”, US Provisional Application No. 60 / 718,145, filed September 15, 2005, the disclosure of which is hereby incorporated by reference. Incorporated herein by reference.

  This application is “MAN-MACHINE INTERFACE USING A DEFORMABLE DEVICE” filed July 27, 2002, US Patent Application No. 10 / 207,677; “AUDIO INPUT SYSTEM” filed August 27, 2003, US Patent Application No. 10 / 650,409; “METHOD AND APPARATUS FOR ADJUSTING A VIEW OF A SCENE BEING DISPLAYED ACCORDING TO TRACKED HEAD MOTION” filed on September 15, 2003; US Patent Application No. 10 / 663,236; filed on January 16, 2004 “METHOD AND APPARATUS FOR LIGHT INPUT DEVICE”, US Patent Application No. 10 / 759,782; “METHOD AND APPARATUS TO DETECT AND REMOVE AUDIO DISTURBANCES”, US Patent Application No. 10 / 820,469; 2005, filed April 7, 2004; “METHOD FOR USING RELATIVE HEAD AND HAND POSITIONS TO ENABLE A POINTING INTERFACE VIA CAMERA TRACKING” filed on December 12, US Patent Application No. 11 / 301,673; and “DELAY MATCHING IN AUDIO / VIDEO” filed June 22, 2005 SYSTEM ”, related to US patent application Ser. No. 11 / 165,473, the disclosure of which is incorporated herein by reference. It is.

  The present invention further relates to “System And Method For Obtaining user Information From Voices” filed Apr. 10, 2006, US copending patent application No. 11 / 400,997, the disclosure of which is incorporated herein by reference. Is done.

Claims (21)

A method for obtaining inputs for controlling individual functions during execution of a game program executed in a video game system having an input / output device that transmits and receives data to and from a digital processor, a memory, and a peripheral device. ,
a) Information specifying the current position of the control stick movable by the user of the game controller in relation to the stop position of the control stick, or b) Specifying whether the switch included in the game controller is in operation. The input / output device receiving controller input information including at least one of information from a game controller operable by the user;
The input / output device receiving additional input information from an environment in which the controller is used;
The digital processor processing the controller input information and the additional input information to obtain a combination input, thereby obtaining a combination input for controlling execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor may reflect the increased pitch-up input to the combination input when the control stick is moved backward while the additional input information pitch is increased to a positive value (rising), or A method of reflecting an increased pitch down input to the combination input when the control stick is moved forward while the pitch is reduced to a negative value (down). A method for obtaining inputs for controlling individual functions during execution of a game program executed in a video game system having an input / output device that transmits and receives data to and from a digital processor, a memory, and a peripheral device. ,
a) Information specifying the current position of the control stick movable by the user of the game controller in relation to the stop position of the control stick, or b) Specifying whether the switch included in the game controller is in operation. The input / output device receiving controller input information including at least one of information from a game controller operable by the user;
The input / output device receiving additional input information from an environment in which the controller is used;
The digital processor processing the controller input information and the additional input information to obtain a combination input, thereby obtaining a combination input for controlling execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns a value of controller input information representing the position of the control stick as rough control information, and assigns a value of additional input information representing the direction of the object that the user can move as fine control information. Ask for the combination input, or
The digital processor assigns a value of the controller input information that specifies whether or not the switch of the game controller is operating as rough control information, and indicates the direction of the object that the user can move as fine control information. Determining the combination input by assigning a value of the additional input information to represent,
The method wherein the combination input represents a value of the rough control information adjusted according to the fine control information. A method for obtaining inputs for controlling individual functions during execution of a game program executed in a video game system having an input / output device that transmits and receives data to and from a digital processor, a memory, and a peripheral device. ,
a) Information specifying the current position of the control stick movable by the user of the game controller in relation to the stop position of the control stick, or b) Specifying whether the switch included in the game controller is in operation. The input / output device receiving controller input information including at least one of information from a game controller operable by the user;
The input / output device receiving additional input information from an environment in which the controller is used;
The digital processor processing the controller input information and the additional input information to obtain a combination input, thereby obtaining a combination input for controlling execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns a value of the controller input information that specifies whether or not the switch of the game controller is operating as rough control information, and indicates the direction of the object that the user can move as fine control information. Determining the combination input by assigning a value of the additional input information to represent,
The method wherein the combination input represents a value of the rough control information adjusted according to the fine control information. A method for obtaining inputs for controlling individual functions during execution of a game program executed in a video game system having an input / output device that transmits and receives data to and from a digital processor, a memory, and a peripheral device. ,
a) Information specifying the current position of the control stick movable by the user of the game controller in relation to the stop position of the control stick, or b) Specifying whether the switch included in the game controller is in operation. The input / output device receiving controller input information including at least one of information from a game controller operable by the user;
The input / output device receiving additional input information from an environment in which the controller is used;
The digital processor processing the controller input information and the additional input information to obtain a combination input, thereby obtaining a combination input for controlling execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns the value of the additional input information that represents the direction of the object that the user can move as rough control information, and assigns the value of the controller input information that represents the position of the control stick as fine control information. To obtain the combination input,
The method wherein the combination input represents a value of the rough control information adjusted according to the fine control information. A method for obtaining inputs for controlling individual functions during execution of a game program executed in a video game system having an input / output device that transmits and receives data to and from a digital processor, a memory, and a peripheral device. ,
a) Information specifying the current position of the control stick movable by the user of the game controller in relation to the stop position of the control stick, or b) Specifying whether the switch included in the game controller is in operation. The input / output device receiving controller input information including at least one of information from a game controller operable by the user;
The input / output device receiving additional input information from an environment in which the controller is used;
The digital processor processing the controller input information and the additional input information to obtain a combination input, thereby obtaining a combination input for controlling execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns a value of the controller input information that specifies whether or not the switch of the game controller is operating as fine control information, and indicates the direction of the object that the user can move as rough control information. Determining the combination input by assigning a value of the additional input information to represent,
The method wherein the combination input represents a value of the rough control information adjusted according to the fine control information. The digital processor obtains the combination input by additively combining the value represented by the controller input information and the value represented by the additional input information, and as a result, the combination input is obtained from the controller input information. or method according to any one of claims 1 to 5, characterized in that presenting a signal having a value smaller than one single value of the value of the additional input information to the game program. The digital processor outputs, as the combination input, a signal having a smoothed value with a gentler temporal change than the single value of either the controller input information or the additional input information to the game program. the method according to any one of claims 1 to 5, characterized in that presented. The digital processor outputs, as the combination input, a high-resolution signal whose time changes more rapidly than the single value of either the controller input information or the additional input information and the signal content is increased to the game program. the method according to any one of claims 1 to 5, characterized in that presenting Te. Said additional input information, The method according to any one of claims 1 to 5, characterized in that it comprises an acoustic information obtained from the acoustic transducer in the environment. The controller input information, The method according to any one of claims 1 to 5 wherein the pressure-sensitive button mounted on the game controller, characterized in that it comprises information identifying whether in operation. The additional input information is i) information obtained from an imaging device in the environment, ii) information from an inertial sensor associated with at least one of a game controller or a user, or iii) information obtained from an acoustic transducer in the environment. of a method according to any one of claims 1 to 5, characterized in that it comprises at least one information. The additional input information includes information obtained from an imaging device in the environment, information from an inertial sensor associated with at least one of the game controller or the user, and information obtained from an acoustic transducer in the environment. 6. A method according to any one of claims 1 to 5 , characterized in that: A video game system having an input / output device for transmitting / receiving data to / from a digital processor, a memory, and a peripheral device, and an input for controlling individual functions being executed in a game program executed in the game system A game system for obtaining
In the input / output device, a) information specifying a current position of a control stick movable by a user of the game controller in relation to a stop position of the control stick, or b) a switch included in the game controller is operating. Receiving controller input information including at least one of information specifying whether or not from a game controller operable by the user,
It said digital processor processes the said controller input information and additionally subscribed force information to get the combination input for controlling the execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor may reflect the increased pitch-up input to the combination input when the control stick is moved backward while the additional input information pitch is increased to a positive value (rising), or When the control stick is moved forward while the pitch is reduced to a negative value (down), the increased pitch down input is reflected in the combination input. A video game system having an input / output device for transmitting / receiving data to / from a digital processor, a memory, and a peripheral device, and an input for controlling individual functions being executed in a game program executed in the game system A game system for obtaining
In the input / output device, a) information specifying a current position of a control stick movable by a user of the game controller in relation to a stop position of the control stick, or b) a switch included in the game controller is operating. Receiving controller input information including at least one of information specifying whether or not from a game controller operable by the user,
It said digital processor processes the said controller input information and additionally subscribed force information to get the combination input for controlling the execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns a value of controller input information representing the position of the control stick as rough control information, and assigns a value of additional input information representing the direction of the object that the user can move as fine control information. Ask for the combination input, or
The digital processor assigns a value of the controller input information that specifies whether or not the switch of the game controller is operating as rough control information, and indicates the direction of the object that the user can move as fine control information. Determining the combination input by assigning a value of the additional input information to represent,
The combination input represents a value of the rough control information adjusted according to the fine control information. A video game system having an input / output device for transmitting / receiving data to / from a digital processor, a memory, and a peripheral device, and an input for controlling individual functions being executed in a game program executed in the game system A game system for obtaining
In the input / output device, a) information specifying a current position of a control stick movable by a user of the game controller in relation to a stop position of the control stick, or b) a switch included in the game controller is operating. Receiving controller input information including at least one of information specifying whether or not from a game controller operable by the user,
It said digital processor processes the said controller input information and additionally subscribed force information to get the combination input for controlling the execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns the value of the additional input information that represents the direction of the object that the user can move as rough control information, and assigns the value of the controller input information that represents the position of the control stick as fine control information. To obtain the combination input,
The combination input represents a value of the rough control information adjusted according to the fine control information. A video game system having an input / output device for transmitting / receiving data to / from a digital processor, a memory, and a peripheral device, and an input for controlling individual functions being executed in a game program executed in the game system A game system for obtaining
In the input / output device, a) information specifying a current position of a control stick movable by a user of the game controller in relation to a stop position of the control stick, or b) a switch included in the game controller is operating. Receiving controller input information including at least one of information specifying whether or not from a game controller operable by the user,
It said digital processor processes the said controller input information and additionally subscribed force information to get the combination input for controlling the execution of the game program,
The combination input includes individual fused inputs for controlling each individual function during execution of the game program;
The digital processor fuses at least a portion of the individual fused inputs with the controller input information associated with a particular individual function and the additional input information associated with a particular individual function. Seeking
The additional input information includes at least one of inertial sensor information obtained by an operation of an inertial sensor mounted on the game controller or direction information indicating a direction of an object movable by the user,
The object movable by the user includes at least one of the game controller or an object attached to a body of the game controller,
The additional input information includes information indicating a direction of an object that the user can move,
The digital processor obtains the combination input by fusing a value of controller input information representing a position of the control stick and a value of the additional input information representing a direction of an object movable by the user,
The digital processor assigns a value of the controller input information that specifies whether or not the switch of the game controller is operating as fine control information, and indicates the direction of the object that the user can move as rough control information. Determining the combination input by assigning a value of the additional input information to represent,
The combination input represents a value of the rough control information adjusted according to the fine control information. The digital processor uses the first information of the controller input information or the additional input information as correction of the input to the game program, and at least one of the second information of the controller input information or the additional input information. 6. A method according to any preceding claim, further comprising the step of modifying the control over the function activated in response. 6. The method according to claim 1, wherein the inertial sensor is attached to the game controller and is at least one of an accelerometer and a gyroscope. The object movable by the user is at least one of the game controller or an object attached to the game controller main body, and the additional input information includes information indicating a direction of the object movable by the user. A method according to any of claims 1 to 5, characterized in that 6. The method according to claim 1, wherein the additional input information includes information indicating at least one of pitch, yaw, and roll. The digital processor obtains the combination input by fusing a value of controller input information indicating a position of a control stick and a value of the additional input information indicating a direction of an object movable by the user. 21. A method according to any of claims 18-20.

Images