KR101250610B1 - User recognition system for controlling audio using stereo camera - Google Patents

Abstract

The user recognition sound control system 100 using the stereo camera according to the present invention includes a display unit 110 for displaying a game content image and one or more speakers for outputting sound according to the game content displayed on the display unit 110. The sound output unit 120, the image acquisition unit 130 for acquiring the user image by the stereo camera, the face position detection unit 140 for detecting the face position of the user from the image acquisition unit 130, the face position Sound for adjusting the position of the speaker or the output angle of the speaker according to the ear position predictor 150 for predicting the user's ear position from the face position detected by the detector 140 and the user's ear position predicted by the ear position predictor It includes a signal controller 160.

Description

USER RECOGNITION SYSTEM FOR CONTROLLING AUDIO USING STEREO CAMERA}

The present invention relates to a user recognition sound control system using a stereo camera and a method for adjusting sound in the system. In particular, the present invention relates to a sound control system that outputs an optimal sound by recognizing the ear position of a user's face with a stereo camera, and a method of adjusting sound in the system.

In the conventional home theater system or game system, the position of the user (listener) is limited to a certain area because the position of the speaker is fixed. Therefore, when the user changes the position, there is a problem that the output sound is different according to the place

In particular, the game machine that recognizes the user's motion among the game consoles used in connection with the digital TV requires the user's motion, but there is a problem that the user must perform the motion only in a limited area for optimal sound effects.

If you change the position of the speaker manually according to the user's operation, there is an inconvenience that you can feel the sound and the sense of reality as close as possible to the sound source.

A user recognition sound control system using a stereo camera according to the present invention and a method for adjusting sound in the system aims to solve the following problems.

First, we want to output the optimal sound effect to the user playing the game.

Second, to detect the position of the dynamic moving user to provide the optimum sound effect that the user can feel according to the position.

Third, even if a user who listens to the sound through a home theater or an audio system routinely moves, the sound quality is to be maintained.

The solution to the problem of the present invention is not limited to those mentioned above, and other solutions not mentioned can be clearly understood by those skilled in the art from the following description.

A user recognition sound control system using a stereo camera according to the present invention includes a display unit for displaying a game content image, a sound output unit including one or more speakers for outputting sound according to the game content displayed on the display unit, and a stereo camera. An image acquisition unit for acquiring an image, a face position detection unit for detecting a user's face position in an image acquired by the image acquisition unit, an ear position prediction unit for predicting a user's ear position at a face position detected by the face position detection unit, and an ear position And a sound signal adjusting unit for adjusting the position of the speaker or the output angle of the speaker according to the position of the ear predicted by the predictor.

The image acquisition unit according to the present invention finds a depth of deviation between a deviation error corrector correcting a deviation error occurring in an image captured by two cameras and a match point between images captured by two cameras in an image in which the error is corrected in the deviation error correction part. A depth map generator for generating a map image and a location determiner for identifying a position of a user in a depth map generated in the depth map generator.

The image acquisition unit according to the present invention further includes a noise removal unit for removing noise from the depth map image generated by the depth map generator.

The ear position predicting unit according to the present invention is characterized by estimating the positions of both ears by grasping the nose position which is the center of the face in the image of the face position detected by the face position detecting unit.

The user recognition sound control system according to the present invention may further include a sound information database unit for storing the optimum sound information data that should reach both ears of the user according to the type and sound intensity of the sound output from the speaker.

The sound signal adjusting unit according to the present invention searches for sound information matching the type and sound intensity of the sound output from the sound output unit in the sound information database unit, and the position or output angle of the speaker of the sound output unit based on the found sound information. It characterized in that to adjust.

Method for adjusting the sound in the user recognition system using a stereo camera according to the present invention in the step S1, the sound is output through one or more speakers according to the game content, in the step S2, S2 step of obtaining a user image using the stereo camera The position of the speaker or the output angle of the speaker depending on the ear position predicted in steps S3 and S4 where the position of the user's face is detected in the face position detected in the step S3 and the S4 and S4 stages that are detected in the acquired user image S5 step is adjusted.

In the step S2 according to the present invention, the matching point between the images taken by the two cameras is found in the image in which the deviation error is corrected in the step S2-1 and the step S2-1 in which the deviation error occurred in the image photographed by the two cameras is corrected. Steps S2-2 in which the depth map is generated and step S2-3 in which the user position is identified in the depth map generated in step S2-2 are included.

Step S2 according to the present invention may further include step S2-4 in which noise is removed from the depth map after step S2-2.

In the step S4 according to the present invention, the position of both ears is predicted based on the nose position, which is the center of the face, in the image of the face position detected in the step S3.

The method for adjusting sound in the user recognition system according to the present invention includes the step S0 in which the optimal sound information data that needs to reach both ears of the user is previously stored in the sound information database according to the type of sound and the intensity of the sound output from the speaker. It may further include.

In step S5 according to the present invention, the sound information matching the type and sound intensity of the sound output in step S1 is searched in the sound information database unit, and the position of the speaker or the output angle of the speaker is adjusted based on the found sound information. It is characterized by.

A user recognition sound control system using a stereo camera according to the present invention and a method for adjusting sound in the system has the following effects.

First, even if a user who receives music or sound effects through a home theater or an audio system moves in the room, the user's optimal sound is maintained.

Second, the optimal sound effect is maintained even when the user's movement is used as an interface in a home theater or a console game connected to a TV or when the user moves while using a specific interface device.

Third, even when the user's face is rotated to change the position of the ear, the optimum sound effect that can be heard by the left and right ears is maintained.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a block diagram schematically showing the configuration of a user recognition sound control system using a stereo camera according to the present invention.
2 is a view showing an example of the configuration of the sound control system of the present invention.
3 is a flowchart illustrating a procedure of a method for adjusting sound in a user recognition system using a stereo camera according to the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, but merely for distinguishing one component from other components. Only used as For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is to be understood that the present invention means that there is a part or a combination thereof, and does not exclude the presence or addition possibility of one or more other features or numbers, step operation components, parts or combinations thereof.

Hereinafter, a user recognition sound control system 100 using a stereo camera and a method of adjusting sound in the system will be described in detail with reference to the accompanying drawings.

Prior to the detailed description of the drawings, it is to be clear that the division of the components in the present specification is only divided by the main function of each component. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. Each of the components to be described below may additionally perform some or all of the functions of other components in addition to the main functions of the components, and some of the main functions of each of the components are different. Of course, it may be carried out exclusively by. Therefore, the presence or absence of each component described through this specification should be functionally interpreted, and for this reason, the configuration of the components according to the wireless power transmission apparatus of the present invention is limited to the extent that the object of the present invention can be achieved. Clearly, it can be different.

1 is a block diagram schematically illustrating a configuration of a user recognition sound control system 100 using a stereo camera according to the present invention.

The user recognition sound control system 100 using the stereo camera according to the present invention includes a display unit 110 for displaying a game content image and one or more speakers for outputting sound according to the game content displayed on the display unit 110. The sound output unit 120, the image acquisition unit 130 for acquiring the user image by the stereo camera, the face position detection unit 140 for detecting the face position of the user from the image acquisition unit 130, the face position Sound for adjusting the position of the speaker or the output angle of the speaker according to the ear position predictor 150 for predicting the user's ear position from the face position detected by the detector 140 and the user's ear position predicted by the ear position predictor It includes a signal controller 160. The sound control system 100 has a display unit 110 on which game content images are displayed. A sound control system 100 for a case where a user enjoys a game by connecting a display device and a conson game machine.

In another aspect of the present invention, the user recognition sound control system 100 using the stereo camera includes a sound output unit 120 including one or more speakers for outputting sound, and an image acquisition unit 130 for acquiring a user image by the stereo camera. The ear position predictor 140 detects the face position of the user from the image acquired by the image acquisition unit 130, and the ear position predictor predicts the ear position of the user from the face position detected by the face position detector 140 ( 150) and an acoustic signal controller 160 for adjusting the position of the speaker or the output angle of the speaker according to the position of the user's ear predicted by the ear position predictor. A sound control system 100 for listening to sound effects or music in a home theater or audio device.

The sound output unit 120 includes various types of sound systems, including one or more speakers. It includes all forms that can be implemented with one or more speakers, from 2 channels to surround channels.

The image acquisition unit detects a matching point between the deviation error correction unit 131 for correcting the deviation error occurring in the images captured by the two cameras, and the image captured by the two cameras in the image in which the error is corrected in the deviation error correction unit 131. Depth map generator 132 for finding and generating a depth map image and a location determiner 133 for identifying the position of the user in the depth map generated in the depth map generator 132.

The depth map generated by the depth map generators 132 and 110 has distance information on the photographed object (target). That is, it has information about where it is from the camera used for shooting.

The image acquirer may further include a noise remover 134 that removes noise from the depth map image generated by the depth map generator 132. Noise reduction may be used in a variety of methods known to those skilled in the art.

The ear position predictor 150 detects the nose position, which is the center of the face, in the image of the face position detected by the face position detector 140 to predict the positions of both ears. Of course, the depth map image can also identify the ear position itself.

However, in order to adjust the position or output angle of the speaker while detecting the user's position in real time in the dynamic movement of the user, the user's position detection must be processed quickly.

Accordingly, the face region may be detected in the depth map image, the ear may be located at the left and right ends on the center of the detected face region, and the sound may be adjusted. In addition, if the position of the nose which is the center of the face area is known, the position of the nose and the position of the ear existing on the horizontal line can be detected relatively quickly and accurately.

The relative position of the ear, depending on the position of the ear or nose in the facial area, varies from person to person. Therefore, for accurate sound output, it is desirable to store ear position information in advance and use it for ear position prediction. In other words, the user's face area can be scanned or photographed with a camera before the game is played to determine the exact ear position in the contour of the face, the relative position of the ear relative to the nose position, or the relative position of the ear relative to the eye position. It is desirable to store in the system.

Detection is possible using various image processing methods available to those skilled in the art of facial contour detection, nose detection or eye detection.

The user recognition sound control system 100 may further include a sound information database unit for storing the optimal sound information data that should reach both ears of the user according to the type and sound intensity of the sound output from the speaker.

For example, when listening to music through audio, it may be desirable to arrange the speaker and / or change the left and right angles or the vertical angle of the speaker according to the type or volume of the music. When playing a game with a console game machine connected to the display device, it is preferable that the sound effect according to the type of game or the content of the game is changed based on the location of the user who participated in the game. As a result, a specific location of the speaker, an output angle of the speaker, and whether or not the speaker moves (sound output while moving) are stored in the sound information database unit for the optimal sound effect.

In case of moving the speaker, it is to reflect this when a specific object moves away from or close to the user's standard among images watched by a game or home theater. A similar effect is possible just by changing the output of the sound effect, but it may be desirable to move the speaker itself for a more dynamic sound effect.

The sound signal controller 160 retrieves sound information matching the type and sound intensity of the sound output from the sound output unit 120 in the sound information database unit, and outputs the sound output unit 120 based on the found sound information. Adjust the speaker's position or output angle.

Furthermore, in the case of a game, sound information related to speaker control may be previously stored in data stored in a memory card or a DVD form that is game content. In this case, the user only needs to know the location of the user and outputs sound effects according to the stored sound information.

Since the sound signal controller 160 needs to change the position of the speaker or change the output direction (angle) of the speaker, the sound signal controller 160 should include a mechanism for moving or changing the output angle. In order to move the speaker, it is desirable to have a driving device such as a wheel or the like movable on the rail device or the speaker itself. To rotate the speaker, the speaker itself or a case housing the speaker must be equipped with a device for rotation. Alternatively, instead of physically rotating the speaker, an acoustic control device having an effect similar to that of the speaker rotated may be used.

2 is a view showing an example of the configuration of the sound control system of the present invention. 2 is a scene in which a user plays a game with a console game machine connected to a digital TV (display unit 110). The console game machine is connected through the display 110 or directly connected to the sound output unit 120 including a plurality of speakers to output sound.

Although FIG. 2 assumes a method of playing a game by capturing motion with a camera, the present invention is also applicable to a traditional game method in which a user plays through an interface device such as a remote controller. When the user moves during the game or the user's movement itself acts as an interface, the user's movement is detected and the optimum sound is output.

In particular, it recognizes the position of the user's face, predicts the ear position, and adjusts the position of the speaker and the output angle of the speaker. 2 illustrates a case in which the speaker moves, and the main speaker located next to the display 110 rotates. However, the speaker can be rotated at various angles such as up and down as well as up and down.

Hereinafter, a method of adjusting sound in a user recognition system using a stereo camera will be described. The overlapping contents of the user recognition sound control system 100 using the stereo camera will be described briefly.

3 is a flowchart illustrating a procedure of a method for adjusting sound in a user recognition system using a stereo camera according to the present invention.

Method for adjusting the sound in the user recognition system using a stereo camera according to the present invention in the step S1, the sound is output through one or more speakers according to the game content, in the step S2, S2 step of obtaining a user image using the stereo camera The position of the speaker or the output angle of the speaker depending on the ear position predicted in steps S3 and S4 where the position of the user's face is detected in the face position detected in the step S3 and the S4 and S4 stages that are detected in the acquired user image S5 step is adjusted.

In step S2, the depth map is generated by finding the matching point between the images taken by the two cameras in the step S2-1 in which the deviation error occurred in the images captured by the two cameras is corrected, and in the image in which the deviation error is corrected in the S2-1 step. In step S2-2 and the depth map generated in the step S2-2 comprises a step S2-3 where the user location is identified.

Furthermore, it is preferable that step S2 further includes step S2-4 in which noise is removed from the depth map after step S2-2. Step S2-4 is preferably performed after step S2-2 in the image processing process.

In step S4, the positions of both ears may be predicted based on the position of the eye or the nose position which is the center of the face in the image of the face position detected in step S3. Various other methods may predict ear position.

The method of adjusting the sound in the user recognition system may further include a step S0 in which the optimal sound information data that needs to reach both ears of the user is previously stored in the sound information database according to the type of sound output from the speaker and the sound intensity. desirable. As described above, the sound information optimal for the movie or game is stored in advance in a DVD title, a game pack (memory) driven by a home theater, a console game machine, or a DVD title, and can be read and used through the system. have.

When the sound information database unit is used, in step S5, sound information matching the type and sound intensity of the sound output in step S1 is searched in the sound information database unit, and the position of the speaker or the output of the speaker based on the found sound information. The angle is adjusted.

The embodiments and drawings attached to this specification are merely to clearly show some of the technical ideas included in the present invention, and those skilled in the art can easily infer within the scope of the technical ideas included in the specification and drawings of the present invention. Modifications that can be made and specific embodiments will be apparent that all fall within the scope of the present invention.

100: user recognition sound control system using stereo camera
110: display unit 120: sound output unit
130: image acquisition unit 131: deviation error correction
132: depth map generator 133: position grasping unit
134: noise removing unit 140: face position detection unit
150: ear position predictor 160: sound signal control unit

Claims (12)

A display unit displaying a game content image;
A sound output unit including one or more speakers to output sound according to game contents displayed on the display unit;
Image acquisition unit for obtaining a user image with a stereo camera;
A face position detector for detecting a face position of a user from the image acquired by the image acquisition unit;
An ear position predictor for predicting a user's ear position at the face position detected by the face position detector; And
A sound signal adjusting unit for adjusting the position of the speaker or the output angle of the speaker according to the position of the ear of the user predicted by the ear position prediction unit,
The ear position predictor predicts the positions of both ears based on the position of the eye or the nose position which is the center of the face in the image of the face position detected by the face position detector.
A user recognition sound control system using a stereo camera, characterized in that the ear is located on the center of the detected face area or the left and right ends of the nose position. The method of claim 1,
The image acquisition unit
A deviation error correction unit for correcting a deviation error occurring in an image captured by two cameras;
A depth map generator for generating a depth map image by finding a matching point between images captured by two cameras in the image in which the error is corrected in the deviation error correction unit; And
And a location determiner configured to determine a location of a user from a depth map generated by the depth map generator. The method of claim 2,
The image acquisition unit
And a noise removal unit for removing noise from the depth map image generated by the depth map generator. delete The method of claim 1,
The user recognition sound control system
And a sound information database unit for storing optimum sound information data which should reach both ears of the user according to the type and sound intensity of the sound output from the speaker. The method of claim 5,
The sound signal adjusting unit
Search for sound information matching the type and sound intensity of the sound output from the sound output unit in the sound information database unit, and adjust the position or output angle of the speaker of the sound output unit based on the searched sound information. User recognition sound control system using stereo camera. Step S1 of outputting sound through at least one speaker according to game content;
Step S2 of obtaining a user image using a stereo camera;
Step S3 of detecting a face position of the user from the user image acquired in step S2;
Step S4 of predicting the position of the user's ear at the face position detected in the step S3; And
S5 step of adjusting the position of the speaker or the output angle of the speaker according to the ear position predicted in the step S4,
In the step S4, the position of both ears is predicted based on the position of the eye or the nose position which is the center of the face in the image of the face position detected in the step S3.
A method of controlling sound in a user recognition system using a stereo camera, characterized in that the ear is predicted to be located on the center of the detected face region or at the left and right ends of the nose position. The method of claim 7, wherein
The step S2
Step S2-1 of correcting the deviation error occurring in the images captured by the two cameras;
Step S2-2 of generating a depth map by finding a matching point between images captured by two cameras in the image in which the deviation error is corrected in step S2-1; And
And a step S2-3 in which the user position is identified from the depth map generated in step S2-2. 9. The method of claim 8,
The step S2
And a step S2-4 of removing noise from the depth map after the step S2-2. delete The method of claim 7, wherein
The method for adjusting the sound in the user recognition system
According to the type of sound and the intensity of the sound output from the speaker, the user's recognition system using a stereo camera, characterized in that it further comprises a step S0 is stored in the acoustic information database in advance the optimal sound information data that must reach both ears of the user How to adjust the sound on the The method of claim 11,
The step S5
The sound information matching the type and sound intensity of the sound output in step S1 is searched in the sound information database unit, and the position of the speaker or the output angle of the speaker is adjusted based on the searched sound information. How to adjust sound in user recognition system using stereo camera.

Images