KR100985694B1 - Selective sound source listening in conjunction with computer interactive processing - Google Patents

Abstract

While interacting with a computer program, methods and apparatus are provided for capturing images and sounds. The apparatus includes an image capture unit configured to capture one or more image frames. Sound capture units are also provided. The sound capture unit is configured to identify one or more sounds. The sound capture unit processes the sound to generate data that can be analyzed to determine the focus zone by substantially excluding sound outside the focus zone. In this way, the sound captured and processed for the focus zone is used for interaction with the computer program.

Selective sound source, image sound capture device, sound source, sound

Description

{SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING}

 The present invention relates to selective sound source listening associated with computer interaction processing.

The video game industry has seen many changes over the years. As computing power has expanded, video game developers have written game software that accepts this increase in computing power. For this purpose, video game developers have coded games with sophisticated computations and mathematical calculations to create highly realistic gaming experiences.

Examples of gaming platforms may be Sony Playstation or Sony Playstation 2 (PS2), each of which is sold in the form of a game console. As is well known, the game console is designed to be connected to a monitor (usually a television) and allows user conversation via a handheld controller. The game console is designed with a specialized processing hardware including a CPU, a graphics synthesizer to handle strong graphics operations, a vector unit to perform geometric transformations and other glue hardware, firmware and software. The game console is designed to further include an optical disc tray for receiving a game compact disc for local play through the game console. Online games are also available for users to play interactively with or against other users over the Internet.

As game complexity draws players' interest, game and hardware manufacturers continue to innovate to enable additional interactivity. In practice, however, the way users interact with games has not changed dramatically over the years.

In view of the above, there is a need for a method and system that further improves user interaction with gameplay.

Broadly speaking, the present invention meets this need by providing an apparatus and method for facilitating interaction with a computer program. In one embodiment, the computer program is a game program, but there is no limitation thereto, and the apparatus and method may be applicable in any computer environment capable of triggering control, taking sound input as input or enabling communication. You can find it. More specifically, if sound is used to trigger a control or input, embodiments of the present invention will allow the input of a particular sound source to be filtered, excluding the filtered input or focusing from a sound source outside of interest. It is configured not to fit. In a video game environment that depends on the selected sound source, the video game responds to a particular response after processing the sound source of interest, without distortion or noise of other sounds outside of the interest. Usually, the game play environment will be exposed to many background noises such as music, movement of others and objects. If the sound outside of the interest is substantially filtered, the computer program may better respond to the sound of interest. The response may be in any form such as command, initiation of movement, selection, change in game status or state, unlocking of features, and the like.

In one embodiment, an apparatus is provided for capturing images and sounds while interacting with a computer program. The apparatus includes an image capture unit configured to capture one or more image frames. Sound capture units are also provided. The sound capture unit is configured to identify one or more sounds. The sound capture unit processes the sound to generate data that can be analyzed to determine the focus zone by substantially excluding sound outside the focus zone. In this way, the sound captured and processed for the focus zone is used for interaction with the computer program.

In yet another embodiment, a method for selective sound listening while interacting with a computer program is disclosed. The method includes receiving input from one or more sound sources to two or more sound source capture microphones. And the method includes determining a delay path from each of the sound sources and identifying a direction for each received input of each of the one or more sound sources. And the method includes filtering sound that is not in the identified direction of the focus zone. The focus zone is configured to provide a sound source for interaction with the computer program.

In yet another embodiment, a game system is provided. The game system provides an image-sound capture device configured to interface with a computing system that enables the execution of an interactive computer program. The image capture device includes video capture hardware that can be arranged to capture video from the focus area. An array of microphones is provided to capture sound from one or more sound sources. Each sound source is identified and combined in a direction relative to the image-source capture device. The focus zone in combination with the video capture hardware is configured to be used to identify one of the sound sources in the proximal direction of the focus zone.

In general, the interactive sound identification and tracking is applicable to interfacing with any computer program on any computing device. If the sound source is identified, the content of the sound source may be further processed to control a feature or object created by a trigger, driving, instruction or computer program.

Other aspects or advantages of the present invention will become apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, which illustrate examples of principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention may be best understood with more advantages by reference to the detailed description of the invention in conjunction with the accompanying drawings.

1 illustrates a game environment in which a video game program may be executed for interaction with one or more users, in accordance with an embodiment of the present invention.

2 shows an example of a three-dimensional diagram of an image-sound capture device, according to one embodiment of the invention.

3A and 3B illustrate logic for processing sound paths and outputting selected sound sources in different microphones designed to receive input, according to one embodiment of the invention.

4 illustrates a computing system for interfacing with an image-sound capture device for processing an input sound source, in accordance with an embodiment of the present invention.

5 illustrates an example of a plurality of microphones used to increase the direction identification precision of a particular sound source, according to one embodiment of the invention.

6 illustrates an example in which sound is identified at a particular spatial volume using a microphone in different planes, according to one embodiment of the invention.

7 and 8 illustrate exemplary method operations that may be addressed for identification of exclusion of sound sources and unfocused sound sources, according to one embodiment of the invention.

The present invention discloses a method and apparatus for facilitating identification of certain sound sources and filtering of unwanted sound sources when sound is used as an interactive tool with a computer program.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without some or all of these specific details. In other instances, well known processing steps will not be described in detail in order not to obscure the present invention.

1 illustrates a gaming environment 100 in which a video game program may be interacted with one or more users, in accordance with an embodiment of the present invention. As depicted, player 102 is visible in front of monitor 108 including display 110. The monitor 108 is interconnected with the computing system 104. The computing system may be a standard computer system, a game console or a portable computer system. In certain examples, not limited to a particular brand, the game console may be produced by Sony computer entertainment, Inc., Microsoft, or other manufacturer.

The computing system 104 is coupled with the image-sound capture device 106. The image-sound capture device 106 includes a sound capture unit 106a and an image capture unit 106b. The player 102 may communicate with the game feature 112 on the display 110. The video game played is one whose input is provided at least in part by the player 102 via the image capture unit 106b and the sound capture unit 106a. As described, the player 102 may move his hand to select the interaction icon 114 on the display 110. The translucent image of the player 102 is captured by the image capture unit 106b and projected onto the display 110. Thus, the player 102 may know where to move his hand to select an icon or interface with the game feature 112. The techniques for capturing such movements and interactions may be different, but exemplary techniques are described in the UK applications GB0304024.3 (PCT / GB / 2004/000693) and GB0304022.7 (PCT /, respectively, filed February 21, 2003). GB2004 / 000703), each of which will be used as a reference.

In this example, the interaction icon 114 is an icon that allows the player to select "swing" so that the game feature 112 can shake the object with the handle. In addition, the player 102 may provide voice commands that may be captured by the sound capture unit 106a and processed by the computing system 104 to provide interaction with the executed video game. As shown, the sound source 116 has a voice command "Jump!" to be. The sound source 116a will be captured by the sound capture unit 106a and processed by the computing system 104 to cause the game feature 112 to jump. Voice recognition may be used to enable identification of the voice command. Alternatively, the player 102 may be connected to the Internet or a network, but may also communicate with remote users who are directly or partly involved in game interaction.

According to one embodiment of the invention, the sound capture unit 106a is configured to include at least two microphones that allow the computing system 104 to select sound coming from a particular direction. By allowing the computing system 104 to filter out directions that are not in the center of the game play (or focus), sound transitions in game environment 100 may interfere with or disrupt the game execution when a specific command is provided by player 102. I will not let you. For example, the game player 102 can tap his foot, which can cause tap noise, which is a non-language sound 117. Such sound may be captured by sound capture unit 106a but the sound coming from the player's foot 102 may be filtered because it is not in the focus area of the video game.

As described below, the focus zone is more preferably identified by an active image area which is the focus point of the emission capture unit 106b. Alternatively, the focus zone may be manually selected from the selection of zones provided to the user after the initialization step. Continuing with the example of FIG. 1, game observer 103 may provide a sound source 116b that disrupts processing by the computing system during interactive game play. However, the game observer 103 is not in the active image area of the image capturing unit 106b, and therefore, the sound coming from the game observer 103 direction is incorrect for the sound source coming from the player 102 and the command from the sound source 116b such that the computing system 104 is sound source 116a. It will be filtered to avoid confusion.

The image-sound capture device 106 includes an image capture unit 106b and a sound capture unit 106a. The image-sound capture device 106 may more preferably digitally capture an image frame and send the image frame to computing system 104 for subsequent processing. An example of the image capture unit 106b is a web camera that is commonly used when a video image is captured and digitally transmitted to a computing device for communication via a network such as a lower storage device or the Internet. Other types of image capture devices may operate analog or digitally as long as the image data is digitally processed to enable identification and filtering. In one preferred embodiment, the digital processing that enables the filtering is performed on software after input data is received. The sound capture unit 106a includes a pair of microphones (microphone 1 and microphone 2). The microphone is a standard microphone that can be integrated into the housing constituting the image-sound capture device 106.

3A depicts sound capture unit 106a when encountering sound source 116 coming from sound A and sound B. FIG. As shown, sound A will project an audible sound and will be sensed by microphone 1 and microphone 2 along sound path 201a and sound path 201b. Sound B will be projected towards microphone 1 and microphone 2 through sound paths 202a and 202b. As depicted, the sound path for sound A will be of different length, thus providing a relative delay when compared to sound paths 202a and 202b. The sound from each of Sound A and Sound B will be processed using a standard triangulation algorithm so that the direction selection can take place in box 216 shown in FIG. 3B. The sound coming from microphones 1 and 2 is buffered in buffers 1 and 2 210a and 210b, respectively, and will pass through delay lines 212a and 212b. In one embodiment, the buffering and delay process will be controlled by software even though hardware can be designed to handle the operation well. Based on triangulation, the direction selection 216 will trigger the identification and selection of any of the sound sources 116.

The sound coming from microphone 1 and microphone 2 respectively will be added in box 214 before being output as the output of the selected sound source. In this way, sound coming from directions other than the active image area can be filtered so that such sound sources do not interfere with processing by computer system 104 or communicate with other users playing interactive video games over a network or the Internet. have.

4 illustrates a computing system 250 that may be used in connection with an image-sound capture device 106 in accordance with an embodiment of the present invention. The computing system 250 includes a processor 252 and a memory 256. The bus 254 interconnects the processor and memory 256 with the image-sound capture device 106. The memory 256 will include at least a portion of the interactive program 258 and will also include optional sound source listening logic or code 260 for processing the received sound source data. Based on the focus zone being identified by the image capture unit 106b, the external sound source of the focus zone will be selectively filtered by executing the optional sound source listening logic 260 (e.g., stored at least in part in the processor and memory 256). By). While the computing system appears in a simplified form, it is emphasized that any hardware configuration can be used that allows the hardware to process instructions to achieve processing of the incoming sound and to allow for selective listening.

Computing system 250 is also interconnected with display 110 via a bus. In this example, the focus zone is identified by the image capture unit focused towards sound source B. Sound from other sound sources, such as sound source A, will essentially be filtered by the optional sound source listening logic 260 when the sound is captured by sound capture unit 106a and transmitted to computing system 250.

In one particular embodiment, the player may participate in the Internet or network video game competition with other users whose primary audible experience is through the speaker. The speaker may be part of the computing system or may be part of the monitor 108. Therefore, it is assumed that the local speaker is generating the sound source A as shown in FIG. In order that sound source A does not receive feedback from the local speaker to the contested user, the optional sound source listening logic 260 filters the sound of sound source A such that the contested user is not provided with feedback of his or her own sound or voice. something to do. During interfacing with video games, it is possible to provide this filtering for interactive communication over the network, while advantageously avoiding negative feedback during the process.

5 depicts an example in which the image-source capture device 106 includes at least four microphones (microphones 1 through 4). Thus, the sound capture unit 106a can be triangulated to have a better granularity and to identify the position of the sound sources 116 (A and B). In other words, by providing an additional microphone, it is possible to more accurately define the position of the sound source, and thus it is possible to delete and filter out sound sources that are outside of interest or which interfere with gameplay or interaction with the computing system. . As shown in FIG. 5, sound source 116 (B) is the sound source of interest as identified by video capture unit 106b. Continuing with the example of FIG. 5, FIG. 6 shows how sound source B is identified in spatial volume.

The spatial volume where sound source B is located will define a focus volume 274. By identifying the focus volume, it is possible to eliminate or filter out noise that is not within a particular volume (eg, not directly in that direction). To facilitate the selection of the focus volume 274, the image-sound capture device 106 will preferably include at least four microphones. At least one microphone will be placed in a plane different from the three microphones. By holding one of the microphones in plane 271 and the other microphone in plane 270 of the image-sound capture device 106, it is possible to define a spatial volume.

As a result, noise from others in the vicinity (as shown at 276a and 276b) will be filtered because it does not lie within the spatial volume defined at volume focus 274. In addition, noise that may be generated outside the spatial volume, such as speaker 276c, may also be filtered because it occurs outside the spatial volume.

7 shows a flowchart in accordance with an embodiment of the present invention. The method begins at step 302 where input is received from one or more sound sources at two or more sound capture microphones. In one example, the two or more sound capture microphones are integrated into the image-sound capture device 106. Alternatively, the two or more sound capture microphones may be part of a second module / housing that interfaces with the image capture unit 106b. Alternatively, the sound capture unit 106a may include a number of sound capture microphones, which may be placed in a specific location designed to capture sound from a user capable of interfacing with the computing system.

The method proceeds to step 304 where a delay path for each sound source is determined. Examples of delay paths are defined by the 201 and 202 sound paths in FIG. 3A. As is well known, the delay path defines the time it takes for a sound wave to travel from a sound source to a particular microphone placed to capture the sound. Based on the delay it takes for the sound to travel from the particular sound source 116, the microphone can use a standard triangulation algorithm to determine the delay and the approximate location at which the sound emanates.

The method proceeds to step 306 where a direction for each of the received one or more sound source inputs is identified. That is, the direction of sound originating from the sound source 116 is identified relative to the position of the image-sound capture device that includes the sound capture unit 106a. Based on the identified direction, the sound source not in the identified direction of the focus area (or volume) is filtered in step 308. It is possible to use an unfiltered sound source for interaction with a computer program as shown in step 310 by filtering the sound source that did not originate from a direction near the focus area.

For example, the interactive program may be a video game in which the user may interact with a person who may be an enemy of the person of the video game or the first player of the video game. The enemy player may be in a local or remote location and may communicate with the first user via a network such as the Internet. In addition, the video game may be played between a large number of users in a group configured to interactively challenge each other's skills in a particular contest associated with the video game.

FIG. 8 shows a flow chart depicting step 320 depicted in an image-sound on device, separate from the software execution step of performing the input received at step 340. Thus, if an input from one or more sound sources is received at two or more sound capture microphones in step 302, the method proceeds to step 304 where a delay path for each of the sound sources is determined in software. As mentioned above, a direction for each of the received inputs based on the delay path is identified in step 306 for each of one or more sound sources.

In this regard, the method moves to step 312 where the identified direction in the vicinity of video capture is determined. For example, as shown in FIG. 1, video capture can be targeted in the active image area. Thus, the vicinity of the video capture may be within the active image area (or volume), and any direction relative to the sound source within or near this active area will be determined. Based on this determination, the method proceeds to step 314 where the direction not near the video capture is filtered. Thus, disturbances, noise and other external inputs that may interfere with the first player's video game play will be filtered out of the processing performed by software running during game play.

In conclusion, the first user may interact with the video game, interact with other users of the video game actively using the video game, or may be logged in or associated with a transaction for the same video game of interest. It can communicate with other users via a network that can be. Such video game communication, interactivity and control will not be disturbed by observers who are not intended to interact, communicate or participate in external noise and / or a particular game or interaction program.

It should be appreciated that the embodiment depicted above can also be applied to online game applications. That is, the embodiment described above may occur in a server that transmits a video signal to a plurality of users via a distributed network such as the Internet, allowing players to communicate with each other in remote noisy locations. It should also be appreciated that the above described embodiments must be performed through hardware or software implementation. That is, the functional embodiments discussed above can be synthesized to define a microchip having logic configured to perform functional tasks for each module associated with the noise cancellation scheme.

Again, selective filtering of the sound source can be applied to other applications such as telephones. In an environment using a telephone, there is the first person (sender) who wants to have a conversation with a third party (recipient). However, while communicating, there may be other people around you who are talking or making noise. The telephone set towards the first person (eg by the direction of the handset) can make the sound coming from the mouth of the first user into the focus area, thus allowing only the first user a choice for listening. . Thus, the selective listening enables substantial filtering of voice or noise unrelated to the original person, and thus can receive clearer communication from the first person using the telephone at the receiving end.

Additional techniques may include other electronic devices that may benefit from taking sound as input for control or communication. For example, the user can control the car's settings with voice commands while avoiding noise from other passengers that may confuse the user's commands. Other applications may include computer controlled applications such as browsing applications, document processing or communications. By enabling this filtering, it is possible to more effectively issue voice or sound commands without disturbing the ambient sound. As such, any electronic device is possible.

In addition, embodiments of the present invention may be applied to various applications, and the scope of the claims should be construed to include any application that may benefit from the above embodiments.

For example, in similar applications, it may be possible to filter sound sources using sound analysis. If sound analysis is used, it is possible to use only one microphone. The sound captured by the single microphone can be digitally analyzed to determine which voice or sound is of interest. In some environments, such as games, it is possible for an initial user to record his or her voice once to train a system that identifies a particular voice. In this way, the exclusion of other voices or sounds will be easy. As a result, since the filtering is based on one sound tone and / or frequency, it is not necessary to identify the direction.

When direction and volume are taken into account, all of the above mentioned advantages associated with sound filtering are equally applicable.

In addition to the above embodiments, it should be understood that the invention may employ various computer-implemented tasks, including data stored on computer systems. These tasks may include tasks that require physical manipulation of physical quantities. Although not required, these quantities usually take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared and manipulated in other ways. In addition, the manipulations performed are related to terms such as production, identification, determination or comparison.

The invention described above may be practiced with other computer system configurations, including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be used to distribute computing environments where tasks are performed by remote processing devices that are linked through a communications network.

The invention may also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device capable of storing data that can be read by a computer system, including electromagnetic wave carriers. Examples of such computer-readable media include hard drives, NAS (Network Attached Storage), ROMs, RAMs, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. . The computer readable medium may be distributed over a network coupled with a computer system such that the computer readable code may be stored and executed in a distributed fashion.

Although the invention has been described in detail for purposes of clarity of understanding, it is apparent that some modifications and variations are possible within the scope of the appended claims. Accordingly, this embodiment is exemplary and not to be considered as limiting. And the invention is not limited to the details described above, but may be modified within the scope and equivalents of the appended claims.

The invention is not limited to the details described above, but may be modified within the scope and equivalents of the appended claims.

Claims (26)

An image capture unit configured to capture one or more image frames; And A sound capture unit configured to identify one or more sound sources, the sound capture unit processing the sound to generate data that can be analyzed to determine the focus zone by substantially excluding sound outside the focus zone, Sound captured and processed for the focus zone is used for interaction with a computer program. The method of claim 1, The sound capture unit includes an array of microphones, wherein the array of microphones is set to receive sound from one or more sound sources, and a path in which the sound travels from the one or more sound sources to each of the microphones is sound of each of the microphones. A device for capturing images and sounds when interacting with a computer program characterized by defining a path. The method of claim 2, The sound paths of different lengths provide a specific delay that enables the calculation of the direction of each of the one or more sound sources with respect to the capture device of the image and sound. . The method of claim 1, And a computing system for interfacing with the image and sound capture device, The computing system includes a processor and A memory configured to store at least a portion of the computer program and an optional sound source listening code, The optional sound source listening code is capable of identifying one or more sound sources according to a focus zone. The method of claim 1, And said sound capture unit comprises at least four microphones, wherein any one of said four microphones is not coplanar with the other microphones. The method of claim 5, Wherein the four microphones are used to define a spatial volume. The method of claim 6, And the spatial volume is a focus volume for listening when interacting with the computer program. The method of claim 7, wherein The computer program is a game program, characterized in that for capturing an image and sound when interacting with the computer program. The method of claim 1, The computer program is a game program, characterized in that for capturing an image and sound when interacting with the computer program. 10. The method of claim 9, Wherein said image capture unit is a camera and said sound capture unit comprises an arrangement of two or more microphones. Receiving input from one or more sound sources at two or more sound source capture microphones; Determining a delay path from each of the sound sources; Identifying a direction for each of the received inputs of each of the one or more sound sources; Filtering a sound source that is not in the identified direction of the focus zone, wherein the focus zone supplies a sound source for interaction with the computer program. The method of claim 11, Wherein said computer program is a game, said game receives interactive input from both image data and sound data, said sound data coming from said sound source in said focus zone. . The method of claim 11, Wherein said at least two sound capture microphones comprise at least four microphones, at least one of said four microphones being in a plane different from said other microphones. The method of claim 13, Identifying a direction for each of the received inputs of each of the one or more sound sources includes processing a triangulation algorithm, And said triangulation algorithm determines a direction relative to a position at which said at least two sound source capture microphones receive input from said at least one sound source. The method of claim 14, Buffering input received from one or more sound sources associated with the two or more sound source capture microphones; And Delaying the received and buffered input; The filtering may further include selecting one of the sound sources. And the selected sound source output is a summation of sounds from each of the sound source capture microphones. An image-sound capture device configured to interface with a computing system that enables the execution of an interactive computer game, The image-sound capture device includes video capture hardware that can be positioned to capture video from a focus area and An array of microphones for capturing sound from one or more sound sources, Each sound source is identified and associated with a direction relative to the image-sound capture device, wherein the focus zone associated with the video capture hardware is configured to be used to identify one of the sound sources in the direction in the focus zone. Game system. The method of claim 16, And the video capture hardware receives video data to enable shape and interactivity of the computer game. The method of claim 16, The sound source in the focus zone enables interaction with the computer game or voice communication with other game users. The method of claim 18, A sound source outside of the focus zone is filtered by interaction with the computer game. A sound capture unit for capturing sound from one or more sound sources; A processor and memory for processing and receiving the sound, The processor is configured to execute instructions for identifying one of the sound sources in the focus zone, and the sound from the identified sound source is processed to enable interactive input with the computer program. Sound capture device. 21. The method of claim 20, And instructions for identifying one of the sound sources use triangulation to identify the direction of each of the sound sources. 21. The method of claim 20, And a command to identify one of the sound sources uses sound frequencies to identify each of the sound sources. 21. The method of claim 20, And wherein said mutual input is one of a communication with a program or a communication with a third party. 21. The method of claim 20, And the input used for the interactive input interfaces with the shape of a computer game. 21. The method of claim 20, And the interaction input interfaces with an electronic device. delete

Images