JP2009112829A - Method and system for applying gearing effects to input based on one or more of visual, acoustic, inertia, and mixed data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and a system enabling higher user interactivity in game playing. <P>SOLUTION: The method and apparatus for interactive interfacing with a computer game system 102 are provided. The computer game system 102 includes a video capture device 105 to capture image data. An input device 215 is displayed on the video capture device 105, and the input device 215 has a plurality of light sources 212 which are modulated. The position of the input device 215 and communication data analyzed by the computer game system based on the captured image data and the state of the plurality of light sources 212 are thus transmitted. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The video game industry has experienced many changes over the years. As computing power expands, video game developers have created game software that takes advantage of this increased computing power as well.

To this end, video game developers have been coding games that employ advanced arithmetic and mathematics to create extremely realistic gaming experiences.
Examples of game platforms are Sony PlayStation or Sony PlayStation 2 (PS2), each sold in the form of a game console. As is well known, game consoles are designed to be connected to a monitor (usually a television) and allow user interaction with a handheld controller. The game console is a specialized processing hardware such as a CPU, a graphic synthesizer for graphic operations with a large amount of processing, a vector device for performing geometry conversion, and other connecting hardware, that is, glue hardware, firmware And designed with software etc.
In addition, the game console is designed with an optical disc tray for receiving a game compact disc for local play by the game console. Also possible is an online game where the user can play against other users on the Internet or interactively play with other users.

  Game software and hardware manufacturers have continued to innovate to enable more interactive features as the game continues to attract players' interest. However, in practice, the way users interact with games has not changed dramatically over the years. In general, a user is still playing a computer game using a hand-held controller or interacting with a program using a mouse pointing device. In view of the foregoing, there is a need for a method and system that enables more advanced user interactivity in game play.

  In general, the present invention meets this need by providing a method, system and apparatus that enables dynamic user interactivity with a computing system. In one embodiment, the computing system is executing a program and the program is interactive. The program may be, for example, a video game that defines interactive objects and features. Interactivity includes providing the user with the ability to adjust gearing elements that adjust the degree to which processing is performed. In one embodiment, the gearing may be set to a relative movement between the amount of movement of the input device versus the amount of movement performed by the object or feature of the computer program.

  In another embodiment, gearing can be set for computer program features, and detection from the input device may be based on inertial analyzer processing. The inertial analyzer tracks input devices for inertial activity and communicates that information to the program. The program then processes the output from the inertial analyzer so that the output can be multiplied by a gearing amount. The amount of gearing determines the degree or ratio at which the program calculates operations. An operation can take any number of forms, and examples of operations can include noise, variable noise, movement performed by an object, or generation of a variable. If the output is a variable, this variable (eg, a multiplier, etc.) may be used to complete the process. This allows the process to consider the amount of gearing. The amount of gearing may be set in advance by the user, set dynamically, or can be adjusted as required.

  In one embodiment, tracking can be performed by an acoustic analyzer. The acoustic analyzer is configured to receive an acoustic signal from the input device and can convey the amount of gearing that is being applied to the command or interaction being performed. The acoustic analyzer may be in the form of a computer program segment or may be specially formed in a circuit designed to process acoustic signal information. Therefore, the acoustic signal information is set dynamically by the program or gearing data set by the user via the input device (by selecting buttons, voice commands, etc. on the controller) upon request. Can be included.

  In one embodiment, input device tracking may be done via an image analyzer. As described below, the image analyzer may comprise a camera that captures an image of the space in which the user as well as the input device is located. In this example, the image analyzer determines the position of the controller so that the processing program features perform their respective actions. The program may be a game and the feature may be an object that is controlled by an input device. Further, the image analyzer is configured to mix the position data with the input gearing value. The gearing value may be given dynamically by the user during execution or may be set by the program depending on the activity of the execution session. The gearing input sets a relative impact on some processing by the computer program based on input gestures and actions by the user. In one embodiment, gearing translates commands and actions from users and user input devices into program features. Program features may not be visible objects, but may include adjustment of variables used to calculate some parameters, predictions, or transformations of either speech, vibration or image movement. Thus, gearing adds an additional meaning for controlling the interactivity provided between programs and program features. In yet another embodiment, a mixer analyzer is provided. This mixer analyzer is designed to provide a hybrid effect on game features. For example, a mixer analyzer handles input from a combination of an image analyzer, an acoustic analyzer, an inertial analyzer, and the like. Thus, in one embodiment, the mixer analyzer receives several gearing variables. These variables can be mixed and synthesized to produce hybrid results, commands, or interactions with program features. Similarly, program features should be broadly understood to include visual and non-visual objects, variables used to process movement, adjustments to audible responses, and the like.

  In one particular example, at least a portion of the amount that translates a movement by the user's controller into a movement or action on a game feature relates to the set or determined gearing. Gearing may be set dynamically or pre-set for the game, or may be adjusted by the user during game play. In addition, this response can be associated with video game objects or features to provide another level of user interactivity and an enhanced experience.

  It should be understood that the present invention can be implemented in many ways, including as a process, apparatus, system, device or method. Several inventive embodiments of the present invention are described below.

  In one embodiment, a method for interactively interfacing with a computer game system is provided. The computer game system includes a video capture device that captures image data. In the above method, an input device is displayed on a video capture device, the input device having a plurality of light sources to be modulated, and a computer based on the position of the input device, the captured data analysis and the state of the plurality of light sources Communication data interpreted by the game system is transmitted. Further, in the above method, the movement, that is, movement, movement, and movement of the object of the computer game executed by the computer game system is defined. Thereby, the movement of the object is associated with the movement, movement or movement at the position of the input device detected from the captured image data. Next, in the above method, gearing between the movement of the computer game object and the movement or movement at the position of the input device is determined. Gearing defines the ratio between input device movement and object movement. The gearing may be set dynamically by the game or the user, or may be preset by the user arranged according to software or a gearing algorithm.

  In another embodiment, a method for interactively interfacing with a computer game system is disclosed. The computer game system includes a video capture device that captures image data. In the above method, an input device is displayed on the video capture device. Thereby, the position of the input device and the communication data interpreted by the computer game system based on the captured image data analysis are transmitted. Next, in the above method, the movement of the object of the computer game executed by the computer game system is defined, and the movement of the object is further associated with the movement of the input device detected from the captured image data. Next, in the above method, the gearing between the movement of the computer game object versus the movement of the input device is defined. Gearing defines a user adjustable ratio between input device movement and object movement.

  In yet another embodiment, a computer readable medium is provided that includes program instructions for interactively interfacing with a computer game system. The computer game system includes a video capture device that captures image data. The computer readable medium includes program instructions for detecting a display of the input device to the video capture device. The input device communicates communication data that is interpreted by the computer game system based on the location of the input device and the captured image data analysis. Further, the computer readable medium includes program instructions that define the movement of an object of a computer game executed by the computer game system, and the movement of the object is associated with the movement of the input device detected from the captured image data. Further, the computer readable medium includes program instructions that define a gearing between the movement of the computer game object and the movement of the input device, the gearing being adjustable by the user between the movement of the input device and the movement of the object. The right ratio. The ratio adjustable by the user can be changed dynamically during game play, before a game play session, or during certain action events during the game.

  In yet another embodiment, a system is provided that enables dynamic user interactivity between user actions and actions performed by computer program objects. The system includes a computing system, a video capture device coupled to the computing system, and a display that receives input from the computing system. In addition, the system includes an input device that interfaces with a computer program executed by the computing system. The input device has a gearing control that defines a ratio between the associated input device and the movement of the controlled object. An object is defined by a computer program and executed by a computing system. The movement of the input device is identified by the video capture device. Further, the movement of the object displayed on the display has a set gearing value set by the gearing control of the input device. In this embodiment, the gearing associated with the input device may be applied to a general application, and is not necessarily applied to a game.

  In yet another embodiment, an apparatus is provided that enables dynamic user interactivity between a user action and an action performed by a computer program object. The apparatus includes an input device that interfaces with a computer program executed by the computing system. The input device has a gearing control that defines a ratio between the movement of the input device associated with the movement of the controlled object. The object is defined by a computer program and executed by the computing system, the movement of the input device is identified by the video capture device, and the movement of the object is set by the gearing control of the input device. It is displayed on the display as having a ring value.

  In another embodiment, computer program features can be geared and detection from an input device can be based on inertial analyzer processing. The inertial analyzer tracks input devices for inertial activity and communicates that information to the program. The program then handles the output from the inertial analyzer so that the output can be geared. The amount of gearing determines the degree or ratio at which the program calculates operations. An operation can take any number of forms, examples of which can include noise, variable noise, movement performed by an object, or generation of a variable.

  If the output is a variable, this variable (eg, a multiplier, etc.) may be used to complete the process. This allows the process to consider the amount of gearing. The amount of gearing may be set in advance by the user, set dynamically, or can be adjusted as required. In one embodiment, tracking can be performed by an acoustic analyzer. The acoustic analyzer is configured to receive an acoustic signal from the input device and can convey the amount of gearing that is being applied to the command or interaction being performed. The acoustic analyzer may be in the form of a computer program segment or may be specially formed in a circuit designed to process acoustic signal information. Therefore, the acoustic signal information is set dynamically by the program or gearing data set by the user via the input device (by selecting buttons, voice commands, etc. on the controller) upon request. Can be included. In yet another embodiment, a mixer analyzer is provided.

This mixer analyzer is designed to provide a hybrid effect on game features. For example, a mixer analyzer handles input from a combination of an image analyzer, an acoustic analyzer, an inertial analyzer, and the like.
Thus, in one embodiment, the mixer analyzer receives several gearing variables. These variables can be mixed and synthesized to produce hybrid results, commands, or interactions with program features. Similarly, program features should be broadly understood to include visual and non-visual objects, variables used to process operations, adjustments to audible responses, and the like. The advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.

  The present invention will be readily understood by reading the following detailed description in conjunction with the accompanying drawings, in which: In the drawings, the same reference numerals are assigned to the same structural elements. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without some or all of these details. In some instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.

  The techniques described herein can be used to provide actively geared input for interaction with a computer program. Changing gearing can be generally defined as an input that can vary in magnitude and / or time in a broad sense. The degree of gearing is communicated to the computing system. The degree of gearing is subjected to the process performed by the computing system. For example, a process can be imaged as a bucket of fluid with input and output. This fluid bucket is a process running on the system. Thus, gearing controls the form of processing performed by the computing system. In one example, the gearing can control the rate of exiting the fluid bucket for an input quantity that can be thought of as a drop of fluid entering the bucket. Thus, filling and drainage rates are dynamic and this drainage rate is affected by gearing. Therefore, the gearing can be adjusted or timed so as to adjust the change value flowing in the program such as the game program.

  Furthermore, gearing affects counters such as movement data counters and controls actions by the processor, i.e. actions performed by game elements, objects, players, characters, and the like. Taking this example as a more specific calculation example, the speed at which the fluid exits is the speed at which it is controlled or controlled at some input and gearing depending on the features of the computer program. is there. The computer program features may be objects, processes, variables, or predetermined / custom algorithms, characters, game players, mice (two-dimensional or three-dimensional), and the like. Processing results that can be changed by gearing can be transmitted to the observer in any number of ways.

  Such methods include a method of visually transmitting on a display screen, a method of transmitting audibly via sound, a method of transmitting by vibrate sound by touch, a method of transmitting by a combination of these, or simply a game And the method of communicating by modifying the reaction of the process to the interactive element of the program. Input can be obtained by tracking performed through (1) image analysis, (2) inertial analysis, (3) acoustic analysis, or hybrid mix analysis of (1), (2), and (3). Various examples have been given for image analysis and applied gearing, but tracking is not limited to video methods, and various methods, specifically inertial analysis, acoustic analysis, and a mix of these. And that it can be done by a suitable analyzer.

  In various embodiments, a computer or game system having a video camera (eg, image analysis) processes the image data and is performed in a focal zone or in a predetermined location (which may be in front of the video camera). Identify various actions. In general, such actions can include moving or rotating an object in a three-dimensional space, or actuating various controls such as buttons, dials, and joysticks. In addition to these techniques, the technique of the present invention further provides an additional function of adjusting a scaling factor, referred to herein as gearing, for input to one or more corresponding actions or program features of the display screen. Adjust the sensitivity. For example, the action on the display screen may be an object that can be the focus of a video game. In addition, an object can be a program feature, such as a variable, multiplier, or calculation, represented as audio, vibration, an image on a display screen, or a combination of these, and with a geared output. Expressed in combination with other expressions.

In another embodiment, computer program features can be geared and input device detection may be based on processing by an inertial analyzer. The inertial analyzer tracks input devices for inertial activity and communicates that information to the program.
The program then handles the output from the inertial analyzer so that the output can be geared. The amount of gearing determines the degree or ratio at which the program calculates operations. An operation can take any number of forms, examples of which include noise, variable noise, object movement, or variable generation, calculation by a program that outputs visual and / or audible results. it can. If the output is a variable, this variable may be used to complete the process. This allows the process to consider the amount of gearing. The amount of gearing may be set in advance by the user, dynamically set, or can be adjusted as required.

  Uses various types of inertial sensors to provide information about six degrees of freedom (eg, transformations in the X, Y, and Z directions (eg, acceleration) and rotation about the X, Y, and Z axes) can do. Examples of suitable inertial sensors that provide information about six degrees of freedom include accelerometers, one or more single axis accelerometers, mechanical gyroscopes, ring laser gyroscopes, or combinations of two or more thereof. be able to.

  Signals from the sensors are analyzed to determine controller movement and / or direction during video game play in accordance with the inventive method. Such a method may be implemented as a series of processors capable of executing program code instructions stored on a processor readable medium and executed on a digital processor. For example, a video game system can include one or more processors. Each processor may be a digital processor device, such as a microprocessor commonly used in video game consoles or custom designed multiprocessor cores. In one embodiment, the processor may implement an inertial analyzer through execution of processor readable instructions. Some of the instructions may be stored in memory.

  In other forms, the inertial analyzer may be implemented in hardware, for example, as an application specific integrated circuit (ASIC) or as a digital signal processor (DSP). Such analyzer hardware may be provided on the controller or console, or may be provided at a remote location. In a hardware implementation, the analyzer can be connected via an external signal, eg from a processor, or via a network, the Internet, a short-range radio connection, a broadband radio, Bluetooth, or a local network, eg via USB cable, Ethernet May be programmable in response to external signals from remote sources.

  The inertial analyzer may comprise or implement instructions to analyze the signal generated by the inertial sensor and utilize information regarding the position and / or orientation of the controller. The inertial sensor signal may be analyzed to determine information regarding the position and / or orientation of the controller. This position and / or orientation information may be utilized with the system during video game play.

  In one embodiment, the game controller may include one or more inertial sensors that can provide position and / or direction information to the processor via inertial signals. The direction information may include angle information such as controller tilt, roll or yaw. As described above, and by way of example, the inertial sensor may include any number of accelerometers, gyroscopes or tilt sensors, or any combination thereof. In one embodiment, the inertial sensor includes a tilt sensor configured to detect the direction of the joystick controller relative to the tilt and roll axis, and a first accelerometer configured to detect acceleration along the yaw axis, And a second accelerometer configured to detect angular acceleration relative to the yaw axis. The accelerometer may be implemented, for example, as a MEMS device that includes a mass attached by one or more springs and a sensor that detects the displacement of the mass in one or more directions. A signal from the sensor that depends on the displacement of the mass may be used to determine the acceleration of the joystick controller. Such a technique is realized by a game program stored in a memory and executed by a processor or an instruction from a general program.

  In one example, an accelerometer suitable as an inertial sensor may be a single mass that is telescopically connected to the frame at 3 to 4 points, such as with a spring. The pitch axis and the roll axis exist on a plane that intersects the frame provided in the joystick controller. As the frame (and joystick controller) rotates about the pitch and roll axes, the mass is displaced under the influence of gravity and the springs expand and contract in a manner that depends on the pitch and / or roll angle. The displacement of the mass is detected and converted into a signal depending on the pitch amount and / or the roll amount. Angular acceleration around the yaw axis or linear acceleration along the yaw axis generates characteristic patterns of spring expansion and contraction or mass movement, which are detected and signaled according to the amount of angular acceleration or linear acceleration. Is converted to Such an acceleration device can measure tilt, roll angular acceleration about the yaw axis, and linear acceleration along the yaw axis by tracking the movement of the mass and the stretching force of the spring. There are many different ways to track the position of the mass and / or the forces acting on it, including resistive strain gauge materials, optical sensors, magnetic sensors, Hall effect elements, piezoelectric elements, capacitive sensors, etc. It is.

  Furthermore, the light source may provide a telemetry signal to the processor, for example, in a pulse code, amplitude modulation or frequency modulation format. Such a telemetry signal may represent which joystick button is pressed and / or how hard the button is pressed. The telemetry signal may be encoded into an optical signal by, for example, pulse encoding, pulse width modulation, frequency modulation, or light intensity (amplitude) modulation. The processor may decode the telemetry signal from the optical signal and execute a game command in accordance with the decoded telemetry signal. The telemetry signal may be decoded from an analysis of a joystick image obtained with an image capture device. In other forms, the apparatus may comprise an independent optical sensor dedicated to receiving telemetry signals from the light source.

  The processor uses the inertial signal from the inertial sensor together with the light signal from the light source detected by the image capture device and / or the sound source location and sound source characteristic information from the acoustic signal detected by the microphone array, Information about the user's location and / or direction may be inferred. For example, the source location of an “acoustic radar” and its characteristics can be used with a microphone array to track a moving sound source while the movement of the joystick controller is tracked independently (through inertial sensors and / or light sources). In acoustic radar, a calibrated listening area is selected at run time, and sound originating from sound sources other than the calibrated listening area is removed. The calibrated listening area may include a listening area corresponding to the volume at the focus or field of view of the image capture device.

  In one embodiment, tracking may be performed by an acoustic analyzer. The acoustic analyzer is configured to receive an acoustic signal from an input device, and can transmit a command to be executed and a gearing amount applied to an interaction. The acoustic analyzer may be in the form of a computer program segment or may be specially formed on a circuit designed to process acoustic signal information. Therefore, the acoustic signal information is set dynamically by the program or gearing data set by the user via the input device (by selecting buttons, voice commands, etc. on the controller) upon request. Can be included. For example, a speech analyzer is a US patent application “SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING” filed May 4, 2006, the disclosure of which is incorporated herein by reference (inventor: Xiadong Mao, Richard L. Marks and Gary, M. Zalewski, agent reference number: SCEA04005JUMBOUS). These analyzers can be configured using a mapping chain. The mapping chain can be exchanged by the game during game play so that settings for the analyzer and mixer can be made.

  In one embodiment, the tracking of the input device may be by an image analyzer. As described below, the image analyzer may comprise a camera that captures an image of the space in which the user as well as the input device is located. In this example, the image analyzer determines the position of the controller so that the processing program features perform their respective actions. The program may be a game and the feature may be an object that is controlled by an input device. Further, the image analyzer is configured to mix the position data with the input gearing value. The gearing value may be given dynamically by the user during execution or may be set by the program depending on the activity of the execution session. The gearing input sets a relative impact on some processing by the computer program based on input gestures and actions by the user. In one embodiment, gearing translates commands and actions from users and user input devices into program features. Program features may not be visible objects, but may include adjustment of variables used to calculate some parameters, predictions, or transformations of either speech, vibration or image movement. Thus, gearing adds an additional meaning for controlling the interactivity provided between programs and program features.

  In yet another embodiment, a mixer analyzer is provided. This mixer analyzer is designed to provide a hybrid effect on game features. For example, a mixer analyzer handles input from a combination of an image analyzer, an acoustic analyzer, an inertial analyzer, and the like. Thus, in one embodiment, the mixer analyzer receives a number of gearing variables. These variables are then mixed and synthesized to provide a hybrid result, command, or interaction with the program features. Similarly, program features should be broadly understood to include visual and non-visual objects, variables used to process operations, adjustments to audible responses, and the like.

  FIG. 1 illustrates an interactive game configuration 100 according to one embodiment of the present invention. The interactive game configuration 100 includes a computer 102 (also referred to herein as a “console”) that can be connected to a display screen 110. An image capture device 105 is placed on the display screen 110 and can be connected to the computer 102. The computer 102, in one embodiment, may be a gaming system console that allows a user to play games with the controller 108 and interface with video games. Computer 102 can also be connected to the Internet so that it can play interactive online games. The image capture device 105 is shown placed on the display screen 110, but if the image capture device 105 can capture an image that is substantially in front of the display screen 110, it is placed in another nearby location. It will be appreciated that While the techniques for capturing these movements and interactions may vary, exemplary techniques are described in UK Patent Application Publication No. 0304024.3 (WO 20042004) filed on February 21, 2003, respectively. / 000693 pamphlet) and UK Patent Application Publication No. 0304022.7 (WO 2004/000703), each of which is incorporated herein by reference.

  In one embodiment, the image capture device 105 may be as simple as a standard webcam or equipped with advanced technology. The image capture device 105 can capture an image, digitize the image, and send the image data to the computer 102. In some embodiments, logic circuitry for performing digitization may be incorporated within the image capture device, and in other embodiments, the image capture device 105 may be analog for digitization. A video signal may be transmitted to the computer 102. In any case, the image capture device 105 can capture a color image or a black and white image of any object in front of the image capture device 105.

  FIG. 2 illustrates an exemplary computer interaction process using an image capture and processing system. The computer 102 receives image data from an image capture device 105 that generates an image from the field of view 202. The field of view 202 includes a user 210 who operates a toy airplane object 215. The object 215 includes a plurality of LEDs 212 that are visually recognizable from the image capture device 105. The LED 212 provides position information and posture information of the object 215. In addition, the object 215 includes one or more operating elements such as buttons, triggers, dials, etc. to generate computer input. In addition, voice commands can be used. In one embodiment, the object 215 includes circuitry 220 that includes logic that modulates the LED 212 to transmit data to the computer 102 via the image capture device 105. The data includes encoded commands that are generated in response to the modulation of the operating means.

  When the object 215 is moved and rotated by the image capture device 105 in a visually recognizable three-dimensional space, the position of the LED represented in the image data is as described with respect to FIGS. 3-6 below. To the coordinates in the three-dimensional space. These coordinates describe position information regarding x, y, and z coordinates in addition to attitude information regarding α, β, and γ values. This coordinate is sent to the computer application. This application may be a plane game where a toy plane is displayed on the display 110 as a real or animated plane 215 ′, and the plane 215 ′ will perform various stunts in response to the modulation of the toy plane 215. It can be carried out. In another form, the user 210 can also handle the controller 108 (shown in FIG. 1) instead of the toy airplane 215, and the movement of the controller 108 can be tracked, and a command to move the object on the display screen. Captured.

  In one embodiment, the user 210 may further select to change or modify the degree of interactivity with the animated airplane 215 '. The degree of interactivity can be changed by allowing the user 215 to adjust the “gearing” component. This gearing component adjusts the amount by which movement by the user's controller 108 (or toy airplane 215) is associated with movement by the animated airplane 215 '. Reactions associated with animated airplanes 215 '(eg, video game objects) in response to gearing that is dynamically set or pre-set for the game or adjusted during game play by the user 210 Can change to give another level of user interactivity and an enhanced experience. Details regarding gearing are described below in connection with FIGS.

  FIG. 3 is a block diagram of an exemplary user input system for interacting with objects on a graphic display that can be used to implement embodiments of the present invention. As shown in FIG. 3, the user input system includes a video capture device 300, an input image processor 302, an output image processor 304, and a video display device 306. Video capture device 300 may be any device capable of capturing a sequence of video images, and in one embodiment is a digital video camera (such as a webcam) or similar image capture device.

  Video capture device 300 may be configured to provide a depth image. In this specification, the terms “depth camera” and “three-dimensional camera” refer to any camera that can acquire distance information, that is, depth information, in addition to two-dimensional pixel information. For example, the depth camera can acquire distance information using controlled infrared illumination. Another exemplary depth camera is a stereo camera pair, which uses two reference cameras to determine distance information by triangulation. Similarly, the term “depth sensing device” refers to any type of device capable of obtaining distance information in addition to two-dimensional pixel information.

  Thus, the camera 300 can provide the ability to capture and map the third dimension in addition to the normal two-dimensional video image. Similar to a normal camera, a depth camera captures two-dimensional data of a plurality of pixels that make up a video image. These values are pixel color values and are typically the red, green, and blue (RGB) values of each pixel. In this way, the object captured by the camera is displayed on the monitor as a two-dimensional object. However, unlike conventional cameras, depth cameras also capture the z component of the scene, which represents the depth value of the scene. Since the depth value is normally assigned to the z-axis, the depth value is often referred to as the “z value”.

  In operation, a z value is captured for each pixel in the scene. Each z value represents the distance from the camera to the object corresponding to the associated pixel in the scene. In addition, the maximum detection range may define a boundary where the depth value is not detected. In various embodiments of the present invention, this maximum range of surfaces can be used to provide user-defined object tracking. For this reason, each object can be tracked in three dimensions by using a depth camera. As a result, the computer system of the embodiment of the present invention can create an enhanced three-dimensional interactive environment for the user using the z-value together with the two-dimensional pixel data. For details of depth analysis, see US patent application Ser. No. 10 / 448,614 filed May 29, 2003, “System and Method for Providing a Real”. -time three dimensional interactive environment) ”. This document is incorporated herein by reference.

  A depth camera may be used according to one embodiment, but should not be construed as needed to identify the position of an object as well as the location of coordinates in three-dimensional space. For example, in the scenario depicted in FIG. 2, the distance between the object 215 and the camera 105 can be estimated by measuring the distance between the leftmost LED 212 and the rightmost LED 212. In the image generated by the image capture device 105, the closer the distance between the LEDs 212 is, the farther the object 215 is from the camera 105. Therefore, the z-axis coordinate can be estimated fairly accurately from a two-dimensional image generated by a general digital camera.

  Returning to FIG. 3, the input image processor 302 converts the captured video image of the control object (such as a depth image) into a signal that is sent to the output image processor. In one embodiment, the input image processor 302 is programmed to separate the control object by depth information from the background of the captured video image and generate an output signal that depends on the position and / or movement of the control object. . The output image processor 304 can be programmed to change the translational and / or rotational movement of objects displayed on the video display device 306 in response to signals received from the input image processor 302. These and other aspects of the invention may be implemented by one or more processors that execute software instructions. According to one embodiment of the invention, one processor performs both input image processing and output image processing. However, as shown in the figure, for ease of explanation, the operation processing is described as being divided into an input image processor 302 and an output image processor 304. It should be noted that the present invention should not be construed as limited to a particular processor configuration (such as multiple processors). The plurality of processing blocks in FIG. 3 are only shown for convenience of explanation.

  FIG. 4 is a simplified block diagram of a computer processing system configured to implement the various embodiments of the invention described herein. The processing system may be an embodiment of a computer-based entertainment system that includes a central processing unit (CPU) 424 coupled to a main memory 420 and a graphics processing unit (GPU) 426. CPU 424 is also coupled to an input / output processor (IOP) bus 428. In one embodiment, the GPU 426 includes an internal buffer for high speed processing of pixel based graphics data. Further, the GPU 426 processes the image data and transmits it to a standard television signal such as NTSC or PAL for transmission to a display device 427 connected to the outside of the entertainment system or its components. Or the function may be provided. In another embodiment, the data output signal may be provided to a display device other than a television monitor, such as a computer monitor, LCD (Liquid Crystal Display) device or other type of display device.

  The IOP bus 428 connects the CPU 424 to various input / output devices or other buses and devices. The IOP bus 428 is connected to an input / output processor memory 430, a controller 432, a memory card 434, a universal serial bus (USB) port 436, an IEEE 1394 (also called Firewire interface) port 438, and a bus 450. The bus 450 connects several other components of the system to the CPU 424, such as operating system (OS) ROM 440, flash memory 442, sound processing unit (SPU) 444, optical disc control. 4 and a hard disk drive (HDD) 448. In one aspect of this embodiment, the video capture device may be directly connected to the IOP bus 428 and transmits to it via the CPU 424, which uses data from the video capture device, The value used by the GPU 426 to generate the graphic image is changed or updated.

  Further, various embodiments of the present invention may use various image processing configurations and techniques, including US patent application Ser. No. 10 / 365,120 filed Feb. 11, 2003, “ There are those described in “METHOD AND APPARATUS FOR REAL TIME MOTION CAPTURE”. This document is incorporated herein by reference in its entirety. The computer processing system can be implemented with a CELL® processor. FIG. 5 is a block diagram showing the configuration of various components of a video game console adapted for use with an operation object that functions as an alternative input device according to an embodiment of the present invention. The exemplary game console 510 includes a multiprocessor unit (MPU) 512 for controlling the entire console 510, a main memory 514 that can be used to store various program operations and data, and floating-point vector operations required for geometric processing. Between the image processor 520, the MPU 512 or the vector arithmetic unit 516, which generates data based on the control from the vector arithmetic unit 516 and the MPU 512, and outputs a video signal to the monitor 110 (CRT or the like). A graphic interface (GIF) 522 that performs arbitration on the transmission bus, an input / output port 524 that enables transmission and reception of data to and from peripheral devices, a flash memory that executes control of the kernel, and the like. It has been provided with an internal OSD functional ROM (OSDROM) 526 and real time clock 528 having a calendar function, and timer functions.

  The main memory 514, vector arithmetic unit 516, GIF 522, OSDROM 526, real time clock (RTC) 528, and input / output port 524 are connected to the MPU 512 via the data bus 530. Also connected to the bus 530 is an image processing unit 538 which is a processor that decompresses compressed moving images and texture images and thereby develops image data. For example, the image processing unit 538 has a function of decoding and expanding a bit stream, decoding a microblock, performing inverse discrete cosine transform, color space transform, vector quantization, etc. according to the MPEG2 or MPEG4 standard format. .

  The audio system includes an audio processing unit (SPU) 571 that generates music and other acoustic effects based on instructions from the MPU 512, an audio buffer 573 for recording waveform data by the SPU 571, and music generated by the SPU 571 and other The speaker 575 may output a sound effect. The speaker 575 may be incorporated as a part of the monitor 110, or may be provided as a separate audio line output terminal for attaching the external speaker 575.

  There is also provided a communication interface 540 that is connected to the bus 530 and is an interface having functions of inputting / outputting digital data and inputting digital contents according to the present invention. For example, in order to implement an online video game application, user input data can be transmitted to or received from a server terminal on the network via the communication interface 540. An optical device such as a CD-ROM having an input device 532 (also referred to as a controller) for inputting data (key input data, coordinate data, etc.) to the console 510 and various programs and data (that is, data relating to objects, texture data, etc.) A disk device 536 for reproducing the contents of the disk 569 is connected to the input / output port 524.

  The present invention further includes a digital video camera 105 that is connected to an input / output port 524. The input / output port 524 may be implemented by one or more input interfaces, such as a serial interface or a USB interface, and the digital video camera 190 may provide USB input or any other conventional interface suitable for use with the camera 105. It can be used advantageously.

  The image processor 520 includes a rendering engine 570, an interface 572, an image memory 574, and a display controller 576 (such as a programmable CRT controller). The rendering engine 570 executes processing for rendering predetermined image data in the image memory via a memory interface 572 and a rendering command sent from the MPU 512. The rendering engine 570 conforms to the NTSC system or the PAL system, and more specifically, for example, at a rate exceeding 10 to several tens of times at intervals of 1/60 seconds to 1/30 seconds, 320 × 240 pixels or 640 × A function capable of drawing 480-pixel image data in real time is provided.

  A bus 578 may be connected between the memory interface 572 and the rendering engine 570, and a second bus 580 may be connected between the memory interface 572 and the image memory 574. The bit widths of the first bus 578 and the second bus 580 are each 128 bits, for example, and the rendering engine 570 can execute high-speed rendering processing on the image memory. The image memory 574 employs a unified storage structure, and in this structure, for example, the texture rendering area and the display rendering area can be set to the same area.

  The display controller 576 is configured to write the texture data acquired from the optical disk 569 by the optical disk device 536 or the texture data created in the main memory 514 into the texture rendering area of the image memory 574 via the memory interface 572. The image data rendered in the display rendering area of the image memory 174 can be read out via the memory interface 572, output to the monitor 110, and displayed on the screen.

  FIG. 6 is a block diagram illustrating functional blocks for tracking and determining a pixel group corresponding to a user input device while the user operates the user input device according to an embodiment of the present invention. It should be understood that the functions represented by the blocks are implemented by software executed by the MPU 512 of the game console 510 of FIG. Furthermore, not all the functions represented by the blocks in FIG. 6 are used in each embodiment. First, pixel data input from the camera is sent to the game console 510 via the input / output port interface 524, whereby the process described below is executed on the game console 510. First, when each pixel of the image is sampled on a raster basis, for example, a color separation processing step S201 is executed. Thus, the color of each pixel is determined, and the image is divided into various two-dimensional parts having different colors. Next, depending on the embodiment, the color transition localization step S203 is executed. As a result, an area where different color portions are adjacent to each other is determined in more detail, and the position of an image where a clear color transition occurs is specified. Next, the step of geometric processing S205 is executed. In this step, depending on the embodiment, an edge detection process or area statistics (AS) calculation is performed to determine the lines, curves and / or polygons corresponding to the edges of the object of interest. Defined algebraically or geometrically.

In step S207, the three-dimensional position and orientation of the object are calculated using an algorithm. This algorithm is described below with respect to the preferred embodiment of the present invention. In order to improve the quality, Kalman filtering processing step S209 is performed on the three-dimensional position and orientation data. This process is performed by estimating the position where the object is located at a given point in time, and it is unlikely that it will occur. It is for removing. Another reason for performing Kalman filtering is that the camera 105 generates an image at 30 Hz, but a typical display operates at 60 Hz, so Kalman filtering can cause a lack of data to control the operation of the game program. To fill up. The smoothing of discrete data by Kalman filtering is well known in the field of computer vision and will not be described in detail here. FIG. 7 shows a schematic block diagram of an exemplary image processing system 700 that maps movement of an object 705 to a spatial volume 702 captured by the image capture device 105. In this example, the object 705 is a three-dimensional space 702, moving the distance x _1, the image processing system 700 interprets the video images captured of the object 705 to identify the movement of the object 705, then the corresponding An action is generated on the display screen 110.

  Specifically, the image capture device 105 includes a digital image sensor that generates image data representing light that has formed an image that is incident on the sensor after passing through a lens, as is well known in the art. Furthermore, the image capture device 105 may include an analog video camera that generates an analog signal representing an image formed by light. In the latter case, the analog signal converts the image into a digital representation, which is then processed by the recognition device 710. Image data representing a continuous two-dimensional image in the three-dimensional space 702 is sent to the recognition device 710. In one embodiment, the recognizer 710 performs various processing steps for identification of the object 705, as described above in connection with FIG. The position of the object 705 is sent to the mapper 712. For example, the absolute coordinates of the object 705 in the three-dimensional space 702 are calculated and transmitted to the mapper 712. The coordinate in the x-axis direction and the coordinate in the y-axis direction can be determined from the position of the object displayed in each image. The coordinates of the object in the z-axis direction can be estimated from the dimensions of the object. That is, the closer the object 705 is to the image capture device 105, the larger the object is displayed on the image. Thus, when an object is displayed in an image, dimensions such as the diameter of the object are used to determine the distance from the image capture device 105, i.e., z-axis coordinates.

  In addition to the position information, the recognition device 710 identifies the command received from the object 705. The command is interpreted from transmission / deformation of the object 705, sound, light emission, and the like. The command received from the object 705 is interpreted by the recognition device, and data corresponding to the received command is transmitted to the application 714. Application 714 may be a game application or other requested computer application. Otherwise, it can accept user input from the image capture device 105. In one embodiment, the mapper 712 inputs absolute coordinates from the recognizer 710 and associates these coordinates with output coordinates that are scaled by the amount of gearing.

In another embodiment, the mapper 712 receives continuous coordinate information from the recognizer 710 and converts the change in coordinate information into a vector movement of the object 705. For example, if the object 705 moves a distance x ₁ between time t ₁ and time t ₂ , a vector x ₁ , 0,0 is generated and sent to the application 714. Times t ₁ to t ₂ may be time intervals between successive frames of video generated by the image capture device 105. The mapper 712 may scale the vector according to a scaling algorithm, for example, by multiplying the vector by a gearing amount. In another embodiment, each coordinate is multiplied by a corresponding gearing factor such as G _x , G _y and G _z . Thus, as displayed on the display 110, the mobile is the distance _{x 2} corresponding to the virtual object 705 ', not the _{x 1.}

  The application 714 changes the gearing amount according to the command 713 received from the recognition device 710 or changes the gearing amount according to the normal operation of the software that transmits the gearing data to the mapper 712 and changes the gearing amount. The gearing data is transmitted to the mapper 712 according to a user command, various events, or an operation mode of the application 714. Therefore, the gearing amount may be changed in real time according to a user command or may be controlled by software. As a result, the mapper 712 transmits the output vector of the motion of the object 705 to the application 714. This output is changed with respect to the change in the position of the object 705 in the space 702 and the gearing amount. In one embodiment, application 714 is a video game that converts the output vector into a corresponding action and the converted action is displayed on display 110.

  FIG. 8 shows an exemplary application of the image processing system 700 of FIG. The computer system 102 includes an image capture device 105 that captures a scene 810. The scene 810 includes a user 802 that has grasped an object 804 that is recognized by the recognition device (FIG. 7). This application program is a checker game in this example, recognizes a command from the object 804, and picks up or drops the checker 808 on the checker board 801. When the user moves the object 805 in front of the image capture device 105, the computer 102 processes the image to identify the movement of the object 804 and converts the movement into a movement of the virtual object 805 ′ on the display 110. The distance that the virtual object 805 ′ moves relative to the real object 805 is determined by the gearing amount 806. In this example, the gearing amount is displayed as display 110 “3”. In one embodiment, the gearing amount is selectable by the user. If the gearing amount is large, the movement of the actual object 805 that the checker 805 ′ moves a specific distance on the display 110 may be small.

  9 and 10 illustrate an example controller 900 that interacts with the image capture device 105 (FIG. 1). The controller 900 includes an interface 902 having a plurality of interface devices including various buttons and joysticks. The controller described here may be wired or wireless. Technologies such as WiFi, Bluetooth, infrared, voice, light, etc. can be used to interface with a computer such as a game console. In one embodiment, the controller 900 has an LED array 905. The LED array can be configured in various layouts, such as a 2 × 2 stack in which each LED is placed at the apex of an imaginary rectangular or square binding box. . By tracking the position and deformation of the binding box as it is projected onto the image plane created by the image capture device, the change and deformation can be analyzed with a video analyzer and the position and orientation information of the controller can be decoded.

  The LED array 905 can emit infrared light or visible light. Image capture device 105 (FIG. 1) can identify LED array 905 as described in connection with various embodiments of the present invention. For example, for each controller, for example, the players 1 to 4 are allocated using the switch 910, so that the user can select from the player numbers 1 to 4. Each selection of player number corresponds to a unique LED pattern or modulation from which the LED array 905 is emitting light. For example, in the case of player 1, the first, third and fifth LEDs are lit. Such player information may be encoded and repeatedly transmitted over a predetermined period between a plurality of video frames. It may be desirable to use an interleaving scheme such that the controller or device LEDs switch between tracking and transmission modes. In tracking mode, all LEDs can be lit during the first part of the cycle.

  In the transmission mode, information can be modulated by the LED during the second part of the cycle. For a predetermined period of time, the LED transmits tracking information and communication information to a video analyzer or appropriate device that can receive the signal. In the transmission mode, the LED may encode information representing the player's ID. The period and operation cycle can be selected to correspond to the tracking speed, lighting conditions, number of controllers, and the like. By interleaving communication and tracking information, the video capture device can be given the appropriate information to calculate the tracking parameters for each controller and to distinguish between controllers. Such a distinction can be used in a video analyzer to isolate each physical controller when monitoring and tracking position and orientation and other metrics of controller movement. In transmission mode, other information such as command or status information may be transmitted by the controller or device LED according to known encoding and modulation schemes. On the receiving side, a video analyzer coupled to the video capture device may track and synchronize with the state of the LEDs and decode information and controller movement. In the transmission mode cycle, it is known that a high bandwidth can be obtained by modulating data between frames.

  As the user interacts with the interface 902, one or more LEDs of the LED array 905 may be modulated and / or discolored. For example, when the user moves the joystick, the LEDs may change brightness or transmit information. Intensity or color changes are monitored by the computer system and provided to the game program as intensity values. In addition, each button can be mapped to a change in color or intensity of one or more LEDs in LED array 905. When the controller 900 is moved in 3D space and rotated in either roll, yaw or pitch direction, the image capture device 105, along with the computer system 102, identifies this change and moves in the image plane. A two-dimensional vector can be generated to describe or a three-dimensional vector to describe movement in a three-dimensional space. The vector may be provided as a series of coordinates describing relative movement and / or absolute position relative to the image capture device 105. As will be apparent to those skilled in the art, movement of the image capture device 105 in a plane (image plane) perpendicular to the line of sight can be identified by an absolute position within the image capture zone. In contrast, the movement of the controller approaching the image capture device 105 can be identified by the LED array appearing to expand.

  Since these LEDs 905 are formed in a rectangular shape, the movement of the controller 900 on three axes and the rotation around each axis can be detected. Although only four LEDs are shown, this is for illustrative purposes only, and any number of LEDs can be used in any configuration. When the controller 900 pitches back and forth, the distance between the upper and lower LEDs approaches, but the distance between the left and right LEDs does not change. Similarly, when the controller yaws left and right, the left and right LEDs appear to approach, but the distance between the upper and lower LEDs does not change. Controller roll motion can be detected by specifying the orientation of the LED in the image plane. As the controller approaches the image capture device 105 along the line of sight of the image capture device 105, all LEDs appear to approach. Finally, controller movement along the image plane can be tracked by identifying the position of the LED on the image plane, thereby identifying movement along each of the x and y axes.

  In addition, the controller 900 includes a speaker 915 that generates audible sound or ultrasound. The speaker 915 generates sound effects that enhance interactivity or communicates commands from the interface 902 to a computer system having a microphone or other element that accepts transmissions.

  FIG. 11 illustrates an exemplary application of the controller 900 of FIGS. In this application, the driving simulation interprets the rotation of the controller 900 as the rotation of the steering wheel of the virtual vehicle. When a user (not shown) rotates the controller 900 as indicated by arrow 1105, the virtual handle 900 ′ rotates on the display 110 as indicated by arrow 1105 ′. In one embodiment, the amount of gearing that determines the amount of rotation of the handle 900 'for each degree of rotation of the controller 900 can be selected by the user, as described above in connection with FIG. In another embodiment, the amount of gearing is controlled by software, as shown in the exemplary graph 1200 of FIG. In this example, the gearing amount is changed with respect to the distance from the central portion of the controller 900, that is, the vertical direction. As a result, the virtual handle 900 ′ can be rotated up to 540 degrees simply by rotating the controller 900 by 90 degrees. By keeping the gearing amount low at a position close to 0 degrees (center portion), high controllability can be maintained for high-speed driving that normally does not require significant steering wheel rotation. When the controller 900 is rotated away from the center portion, the gearing amount increases as shown in the graph 1200, and it is possible to cope with a sharp curve that is usually required at a low speed.

  FIG. 13 shows another exemplary controller 1300 having a handle 1305 operated by a user. In this case, the rotation of the handle 1305 and the button operation of the interface 1302 are interpreted by the controller 1300 that transmits data via the LED 1310.

  FIG. 14 shows an exemplary application of the controller 1300. In this example, the application is a driving simulation, which receives a command issued in response to the rotation of the handle 1305 and interprets this command as a rotation corresponding to the virtual handle 1305 of the display 110. The amount of gearing that scales the rotation of the handle 1305 to the corresponding rotation of the virtual handle 1305 ′ can be changed according to user interaction with the interface 1302 (FIG. 13) or under control by software.

  FIG. 15 shows an example graph 1500 depicting an example change in gearing amount over a predetermined time period. In one example, the change in gearing is set dynamically by the user during game play. Further, as shown in the graph 1500, the gearing transitions smoothly or abruptly over a long time step, a short time step, or a combination thereof. Thus, gearing can be set or changed by the game for a predetermined period of time during the game session, thereby providing a more realistic interactive experience. Further, by allowing the user to control the gearing, it is possible to perform another dimension control that exceeds the predetermined control found in the conventional game.

  FIG. 16 illustrates another exemplary application of an image processing system that responds to user interaction. In this example, the user 1602 interacts with a baseball simulation by swinging a toy baseball bat 1605 that the image processing system can recognize as an input object. When the toy baseball bat is swung, the user and / or software controls the amount of gearing and manipulates the speed and distance of the virtual baseball bat 1605 '. In one embodiment, the bat includes a number of buttons that can be pressed during game play, which can be changed to change the gearing. In another embodiment, the user can be preset or programmed in combination with the gearing level to be applied during the bat swing.

FIG. 17 shows a graph 1700 depicting exemplary gearing amounts along the length of the swing, each at different times t ₁ -t ₅ . This gearing amount is set by the user before swinging the bat, or dynamically set by the game according to the position of the bat captured by the camera 105. Similarly, FIG. 17 shows how the gearing changes over a given period and is kept constant or gradually increased over a specific time interval. In the graph 1700, gearing is set to be higher at between time _t 1 ~t _3. Thus, the bat swing is more powerful when it comes into contact with the ball. Further, the gearing is relaxed during the time t _{3 to} t ₄ after the bat contacts the ball. In one embodiment, each time is predicted by a user or determined by a computer.

  In one example, the user swings several times and the computer defines a number of example time slots that correspond to the user's actual swing ability. The user can then uniquely assign specific gearing for each time interval, depending on how much he / she wants to influence game interactivity. When gearing is set, the movement of the bat 1605 by the user is associated with the movement of the bat 1605 '(for example, a game object). Similarly, gearing is preset by the game during another action, set by the user during the game, and further adjusted in real time during game play.

  FIG. 18 illustrates another exemplary application of an image processing system that is responsive to user interaction. In this example, a user (not shown) interacts with the football simulation by throwing with a toy football ball and pressing the actuator (actuator) to release the ball. Of course, a controller may be used instead of a toy football ball. The virtual player 1802 operates a virtual football ball in response to user interaction. In one embodiment, the actuator causes the LED to lighten or change the color recognized by the image processing system as a command sent to the football simulation application. The user can control certain actions on the field, such as selected recipient actions, after releasing the ball. In one embodiment, releasing the ball allows the application to send gearing data to the mapper (FIG. 7) and change the amount of gearing to distinguish, for example, finer movement or movement of the input object. .

  FIG. 19 shows an exemplary graph 1900 showing the change in the amount of gearing after the ball is “released”. Although graph 1900 is illustrated as a simple graph, it should be understood that gearing can be any profile depending on the object of the game.

  FIG. 20 shows a flowchart 200 illustrating an exemplary procedure for receiving user input to a computer program. The procedure begins as shown in start block 2002 and proceeds to operation 2004 where the position of the control input is identified. The control input may be the position of the input object in the three-dimensional space as described above with reference to FIG. 7 or the position of the user interface device as described above with reference to FIG. This position is expressed as one representative value or a plurality of values such as a vector. This procedure proceeds to operation 2006 after the position of the control input is specified.

  In operation 2006, it is determined whether the position has changed. If there is no change in position, the procedure returns to operation 2004. In one embodiment, operation 2004 is delayed until a new frame of image data is received from the image capture device. If it is determined in operation 2006 that the position has been changed, the procedure proceeds to operation 2008.

  In operation 2008, a movement vector is calculated. This movement vector may have any number of dimensions. For example, if the movement is an input object in a three-dimensional space, the movement vector explains the operation as a three-dimensional vector. However, if the movement is a one-dimensional control input such as a handle, the movement vector is a one-dimensional vector that describes the amount of rotation of the handle. After determining the movement vector, the procedure proceeds to operation 2010.

  In operation 2010, the movement vector is multiplied by the current gearing amount to determine the input vector. The current gearing amount is a scalar amount or a multidimensional value. If the gearing amount is a scalar amount, all dimensions of the movement vector are multiplied by the same amount. If the gearing amount is a multidimensional value, each dimension of the movement vector is multiplied by the corresponding gearing amount dimension. The gearing amount changes in response to user input, and the gearing amount is under software control. Therefore, the current gearing amount changes each time. After multiplying the movement vector by the current gearing amount, the procedure proceeds to operation 2012.

  In operation 2012, a new position of the virtual object is calculated using the input vector obtained in operation 2010. This new position may be a camera position or a position of an object such as a virtual handle. Virtual objects are not displayed on the display screen. When the new position of the virtual object is calculated, the procedure proceeds to operation 2014 and data representing the new position is sent to the application program. The procedure ends as indicated by end block 2016. However, it should be noted that this flow is merely exemplary in nature and other forms are possible.

  In one embodiment, in operation, the input device as well as movement of the input device is detected. The movement of the input device is determined by the camera that is capturing the input device. The gearing value is applied by user control, pre-setting or pre-programmed setting. If the gearing value is set by the user, the gearing value is selected, for example, by causing the user to press a button on an input device (eg, a controller). Depending on the amount of gearing, movement control is associated with a computer game object. If the user is using an input device to control a game action figure, the gearing that is set or controlled to be set is that the movement of the input device will move the action figure of the computer game. It has an influence on how it is mapped. Thus, gearing and gearing changes can dynamically apply reactions associated with objects that may be part of a computer game.

  In various embodiments, the image processing functions described above for determining an intensity value, a controller player number, and the posture and / or position of one or more input objects including the controller are performed in a process running on a computer system. May be. A computing system, referred to herein as an application program, may be executing a main process, which may be a game application, requesting or receiving data obtained from image or sound processing. Such data includes the player number of the controller, the posture and / or position of one or more input objects including the controller, the operation of the controller, and the like.

  In various embodiments, the process performing image and / or audio processing functions may be a video camera or video / audio surveillance device driver, which is implementation specific as is generally known. And providing data to the main process via any type of interprocess communication, as is known and understood in the art. The process of performing image or sound processing may be executed on the same processor that is executing the game software or other application program, or may be executed on a different processor. Further, for image or sound processing and game functions, for example, a procedure call may be used to make a common process within the same process. For this reason, in this specification, input vectors and other information may be referred to as “provided to the program”. However, in the present invention, one process includes both an image processing function and a game function. It should be understood that providing such data to process routines using procedure calls or other software functions is also included.

  In addition, these functions may be divided into different processes so that one or more processes executed on a common processor core or multiple processor cores perform image and / or audio processing as described herein. Another process may perform the game function. The present invention may be used as described herein to provide other user input mechanisms, other mechanisms for tracking the angular direction of sound, and / or mechanisms, machines for actively or passively tracking the position of objects. It may be used with a mechanism that uses a visual field, or a combination of these. The tracked object may have auxiliary controls or buttons that manipulate feedback to the system. Such feedback includes, but is not limited to, light emission from the light source, sound distortion means or other suitable transmitters and modulators, as well as buttons, pressure pads, and the like. These can affect its transmission or modulation, coding state and / or commands exchanged with the device being tracked. The invention is implemented by the configuration of other computer systems, such as game consoles, game computers or computing devices, portable devices, microprocessor systems, microprocessor-based programmable consumer electronics, minicomputers, mainframe computers, etc. May be.

  The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a network. For example, online game systems and software may be used. In view of the above embodiments, it should be understood that the present invention may use various computer-implemented operations that use data stored in a computer system. These operations require physical manipulation of physical quantities. This physical quantity typically takes the form of an electrical or magnetic signal that can be manipulated, stored, transferred, combined, compared, etc., but is not necessarily limited thereto. Furthermore, the operations performed are often referred to as generation, identification, determination or comparison. Any of the operations described herein that form part of the present invention are useful mechanical operations. The present invention also relates to a device or apparatus for performing these operations. This device may be specially made for a desired purpose, such as the carrier network described above, or a general-purpose computer, selectively operated by a computer program stored in the computer, or It may be configured. In particular, various general-purpose machines may be used in combination with a computer program described in accordance with the teachings of this specification, or it may be more convenient to create a specialized machine for performing a desired operation. .

  The invention may also be embodied as computer readable code on a computer readable medium. The computer readable medium may be any data storage device that can store data which can be thereafter read by a computer system. Examples of computer readable media include hard disk, network attached storage (NAS), read only memory, random access memory, FLASH base memory, CD-ROM, CD-R, CD-RW, DVD, magnetic tape and other optical Type data storage devices and non-optical data storage devices. The computer readable medium may also be distributed over a computer system coupled to a network so that the computer readable code is stored and executed in a distributed fashion.

  Furthermore, while gearing has been described in the context of video games, it should be understood that gearing can be applied to any environment controlled by a computer. In one example, gearing is associated with a computer input device that allows information to be interacted, selected or entered. By applying different gearing in various input operations and interactive operations, a further degree of operation that is not normally seen in an environment with a set control setting is possible. Thus, as defined herein, the gearing embodiments should be given a wide range of applications.

  Once the gear ring is determined, this gear ring can be applied to a gesture, which is communicated to the computer program. As described above, tracking of gestures and input devices can be realized by image analysis, inertia analysis, or audible sound analysis. Examples of gestures include throwing an object such as a ball, swinging a bat or golf club, pushing a hand pump, opening or closing a door or window, turning a handle or other vehicle control, a fighter making a punch, Polishing, waxing and wiping, house paint, vibration, chatting, roll, football throwing, baseball throwing, turning knob movement, 3D / 2D mouse movement, scrolling, movement with known profile, recording Any possible movement, back and forth movement along any vector, i.e. tire inflation along any direction in space, movement along the path, movement with exact stop and start times, noise floor Any time based on user actions that can be recorded, tracked, and repeated within a spline, etc. There, but are not limited to these.

  Each of these gestures can be pre-recorded from route data and stored as a time-based model. Thus, gearing can be applied to any one of these gestures depending on the degree of gearing set by the user or program. Although the invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the embodiments are illustrative and not limiting and the invention is not limited to the details described herein, but the appended claims and their equivalents. May be changed within.

Explanatory drawing of the interactive game structure which has an image capture device. FIG. 3 is an illustration of exemplary computer interaction using an image processing system. 1 is a block diagram of an example user input system that interacts with objects on a graph display. 1 is a simplified block diagram of a computer process system configured to implement an embodiment of the invention described herein. FIG. The block diagram which shows the structure of the various components of the video game console adapted to use together with the operation object which functions as an alternative input device. The block diagram which shows the functional block for tracking and discriminating the pixel group corresponding to a user input device during operation by a user. 1 is a schematic block diagram of an exemplary image processing system. FIG. 8 is an explanatory diagram of an exemplary application of the image processing system of FIG. 7. FIG. 2 is a plan view and top view of an exemplary controller for interacting with an image capture device. FIG. 2 is a plan view and top view of an exemplary controller for interacting with an image capture device. FIG. 11 is an illustration of an exemplary application for the controller of FIGS. 9 and 10. FIG. 12 is an explanatory diagram showing controlled gearing amounts of exemplary software of the application shown in FIG. 11. FIG. 3 is an illustration of an exemplary controller having a handle. FIG. 14 is an illustration of an exemplary application of the controller of FIG. FIG. 4 is an explanatory diagram of an exemplary graph showing an exemplary change in the amount of gearing during a predetermined period. FIG. 6 is an illustration of another example application of an image processing system that is responsive to user interaction. FIG. 17 is an explanatory diagram of a graph showing exemplary gearing amounts at different times along the length of the baseball bat swing of the application shown in FIG. 16. FIG. 6 is an illustration of another example application of an image processing system that is responsive to user interaction. FIG. 19 is an explanatory diagram of a graph showing an exemplary change in the amount of gearing after “releasing” the football of the application shown in FIG. 18; 6 is a flowchart illustrating an exemplary procedure for receiving user input into a computer program.

Explanation of symbols

100 game configuration 102 computer system 105 image capture device 108 controller 110 display 174 image memory 215 object 300 camera 302 input image processor 304 output image processor 306 video display device 420 main memory 430 input / output processor memory 432 controller 434 memory card 436 port 438 port 442 Flash memory 450 Bus 510 Console 514 Main memory 516 Vector arithmetic unit 520 Image processor 524 Input / output port 524 Input / output port 530 Data bus 532 Input device 536 Optical disk device 538 Image processing unit 540 Communication interface 1200 Graph 1300 Controller 1302 Interface 1305 Handle 1605 baseball bat

Claims (20)

A method of interactively interfacing with a computer game system,
Providing an input device for transmitting acoustic data to be interpreted by the computer game system;
Identifying an action associated with the acoustic data provided by the input device performed by the computer game system;
Including gearing between the action performed by the computer game system and the acoustic data received from the input device;
The gearing scales adjustments to affect the action performed by the computer game system, and the gearing is dynamically adjustable based on user settings, or A method defined by a program executed by the computer game system to make dynamic adjustments during its execution.   The action includes (a) setting a movement amount according to an object of a computer game executed by the computer game system, (b) setting a variable for executing the process, and (c) processing that affects sound or vibration. A method of interactively interfacing with a computer game system as recited in claim 1, including one of: setting a rate of change; or (d) setting a rate of change of movement of a graphical object.   The computer game system of claim 1, further comprising: analyzing the acoustic data at least periodically; and reapplying the action at least periodically using the changes updated by the gearing. To interactively interface with.   If it is determined that the gearing has been updated, apply the updated gearing before the next action or stepwise for a predetermined period of time while continuing to process the action; A method for interactively interfacing with a computer game system according to claim 3.   The adjusted gear ring is (a) phased, (b) incrementally, or (c) smoothly applied when the gear ring is adjusted to affect the action. A method of interactively interfacing with the computer game system of claim 3.   The computer game system according to claim 1, wherein the acoustic data is analyzed to identify coordinate information of the input device, and the coordinate information is converted into vector data by the association and applied to the action. How to interface interactively.   The method of interactively interfacing with a computer game system according to claim 6, wherein the vector data is scaled in response to adjustment of the gearing.   7. The method of interactively interfacing with a computer game system according to claim 6, wherein the vector data is scaled by multiplying the vector data by a gearing amount defined by the gearing.   The method of interactively interfacing with a computer game system according to claim 6, wherein the input device includes a plurality of light sources used to detect the position of the input device and communication data. A system that enables dynamic user interactivity between user actions and actions performed by a computer program,
A computing system; and an acoustic analyzer coupled to or executed by the computing system;
An input device that interfaces with the computer program executed by the computing system, the input device being applied to the acoustic data generated by the input device and the computer program analyzed by the acoustic analyzer With a gearing control that determines the scaling between and
The change of the gearing control is set before the interactivity with the computer program object, during the interactivity, or after the interactivity, and the change is applied to the computer program. A system that plays a role in conveying different reactions by the object each time an action is performed.   11. The system that enables dynamic user interactivity between a user action and an action performed by the computer program according to claim 10, wherein the computer program is an interactive application.   12. The system that enables dynamic user interactivity between a user action and an action performed by a computer program according to claim 11, wherein the interactive application is a video game.   12. The system that enables dynamic user interactivity between a user action and an action performed by a computer program according to claim 11, wherein the input device is a controller or handheld object.   14. The system for enabling dynamic user interactivity between a user action and an action performed by a computer program according to claim 13, wherein the controller or handheld object includes a light emitting diode for position detection.   14. The user action and computer program of claim 13, wherein the controller or handheld object includes a speaker for transmitting audio or ultrasonic signals that cause the computing system to perform the action. A system that enables dynamic user interactivity between actions.   The user action and computer program of claim 13, wherein the controller or handheld object includes a microphone that receives input from a user to communicate with the computing system to cause an action by the computer program. A system that allows dynamic user interactivity between actions performed by. A device that enables dynamic user interactivity between user actions and actions performed by a computer program,
Gearing that includes an input device that interfaces with the computer program executed by a computing system, the input device defining a scaling value between acoustic data of the input device associated with an action applied by the computing system The gearing control is triggered by the computer program in response to a change caused by a button or before, during or after the interactivity with the computer program A device that can be changed dynamically each time it is changed.   18. The dynamic user interactivity between a user action and an action performed by a computer program object according to claim 17, wherein the input device is a controller or handheld object that includes a light emitting diode for position detection. A device that enables.   19. The user action and computer program object of claim 18, wherein the controller or handheld object includes a speaker for transmitting audio or ultrasonic signals that cause the computing system to perform the action. A device that allows dynamic user interactivity between actions performed.   The user action and computer program of claim 19, wherein the controller or handheld object includes a microphone that receives input from a user to communicate with the computing system to cause an action by the computer program. A device that allows dynamic user interactivity between actions performed by different objects.

Images