KR101260053B1 - Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture - Google Patents

Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture Download PDF

Info

Publication number
KR101260053B1
KR101260053B1 KR1020110120027A KR20110120027A KR101260053B1 KR 101260053 B1 KR101260053 B1 KR 101260053B1 KR 1020110120027 A KR1020110120027 A KR 1020110120027A KR 20110120027 A KR20110120027 A KR 20110120027A KR 101260053 B1 KR101260053 B1 KR 101260053B1
Authority
KR
South Korea
Prior art keywords
voice
hand gesture
hand
recognized
vehicle control
Prior art date
Application number
KR1020110120027A
Other languages
Korean (ko)
Inventor
이상헌
손명규
김병민
이장우
Original Assignee
재단법인대구경북과학기술원
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 재단법인대구경북과학기술원 filed Critical 재단법인대구경북과학기술원
Priority to KR1020110120027A priority Critical patent/KR101260053B1/en
Application granted granted Critical
Publication of KR101260053B1 publication Critical patent/KR101260053B1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K28/00—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K28/00—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions
    • B60K28/02—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output

Abstract

An intelligent vehicle control apparatus and method using motion and voice are provided. The 3D photographing unit photographs and outputs the user's hand gestures, the microphone receives the user's voice, the motion recognition module recognizes the hand gestures captured by the 3D photographing unit, and the voice recognition module is used for the hand gestures recognized by the motion recognition module. The controller recognizes the voice input through the microphone, and when the user's hand gesture is recognized after the voice is recognized and the user's hand is re-photographed through the 3D photographing unit, the controller checks the vehicle control command corresponding to the recognized voice and the re-photographed hand gesture. .

Figure R1020110120027

Description

Intelligent Vehicle Controlling Apparatus and Method using Fusion Recognition of User's Voice and Hand Gesture}

The present invention relates to an intelligent vehicle control apparatus and method using an operation and a voice, and more particularly to an intelligent vehicle control apparatus and method using an operation and a voice capable of controlling a device of a vehicle using a user's operation and voice. It is about.

Speech Recognition technology allows a computer to analyze a user's voice input through a microphone, extract features, recognize a result that is close to a pre-entered word or sentence as a command, and perform an operation corresponding to the recognized command. It is a technique to do.

When the voice recognition technology is applied to a vehicle, the driver can easily operate a desired vehicle device by issuing a command using only a voice without directly operating the device by using a hand. However, the voice recognition technology applied to the existing vehicle frequently fails to accurately recognize the driver's voice due to the driving of the vehicle or the noise inside or outside the vehicle. As a result, the driver is inconvenient to repeatedly give a command until the module to which the voice recognition technology is applied recognizes the driver's voice. Therefore, there is a need for a technology capable of more quickly and accurately recognizing a driver's command in a vehicle.

Korean Patent Registration No. 10-0948600 (March 12, 2010)

The technical problem to be solved by the present invention is an intelligent vehicle control apparatus and method using the operation and voice that can recognize the driver's motion and voice and perform a corresponding command even without the driver manually operating the internal device. To provide.

According to an embodiment of the present invention, the 3D photographing unit for photographing and outputting the user's hand gesture; A microphone for receiving a voice of the user; A motion recognition module for recognizing a hand motion taken by the 3D photographing unit; A voice recognition module for recognizing a voice input through the microphone based on a hand gesture recognized by the motion recognition module; And a controller configured to check a vehicle control command corresponding to the recognized voice and the re-photographed hand gesture when the hand gesture of the user who is re-photographed through the 3D photographing unit is recognized after the voice is recognized. And an intelligent vehicle control apparatus using voice.

The controller may control the voice recognition module to recognize a voice input through the microphone when the hand gesture recognized by the motion recognition module is a predetermined mode switch hand gesture.

If the hand gesture recognized by the motion recognition module is a hand gesture for mode switching requesting vehicle control, the controller may switch from a standby mode to a command mode operating based on the user's voice and hand gesture.

The motion recognition module obtains a difference image of a 2D image of the hand motion photographed in the standby mode when the depth image of the hand motion photographed in the standby mode is less than a first threshold value. When the obtained difference image is less than a second predetermined threshold value, it may be determined that the hand gesture is the hand gesture for mode switching.

When the user's voice is recognized after the switch to the command mode, the controller enters an operation mode corresponding to the recognized voice, and when the re-photographed hand gesture is recognized, the controller enters an operation mode corresponding to the voice. And a vehicle control command mapped to the re-photographed hand gesture.

The operation mode is a mode for controlling the operation of one of the devices mounted on the vehicle, and the devices may include at least one of a radio, an air conditioner, a mobile phone, a music player, a navigation, a window, and a side mirror.

The user's hand gesture recognized by the motion recognition module may be a shaking motion of shaking the user's hand from side to side, a right operation of moving the user's hand to the right, a left operation of moving the user's hand to the left, and a movement of the user. It may include at least one of an upward operation of moving the hand up and a downward operation of moving the hand of the user down.

On the other hand, according to another embodiment of the present invention, photographing the user's hand gestures and outputting the 3D image; Recognizing the hand gesture from the output 3D image; Recognizing a voice input through a microphone based on the recognized hand gesture; Re-photographing the hand gesture of the user after the voice is recognized to recognize the re-photographed hand gesture; When the re-photographed hand gesture is recognized, confirming a vehicle control command corresponding to the recognized voice and the re-photographed hand gesture may include providing an intelligent vehicle control method using an operation and voice. .

The recognizing of the voice may be performed when the hand gesture recognized in the recognizing the hand gesture is a predetermined mode switching hand gesture.

If the hand gesture recognized in the step of recognizing the hand gesture is a hand switch for mode switching requesting vehicle control, switching from a standby mode to a command mode operating based on the user's voice and hand gesture.

The recognizing of the hand gesture may include comparing the depth image of the hand gesture photographed in the standby mode with a predetermined first threshold value, and if the depth image is less than the first threshold value, 2D of the hand gesture photographed in the standby mode. Obtaining a difference image of the image; And comparing the obtained second image with a second predetermined threshold value and determining that the hand gesture is the mode switching hand gesture when the difference image is less than the second threshold value.

When the voice of the user is recognized after the switch to the command mode, the step of entering the operation mode corresponding to the recognized voice; The step of confirming the vehicle control command, entering the operation mode Thereafter, when the re-photographed hand gesture is recognized, an operation mode corresponding to the voice and a vehicle control command mapped to the re-photographed hand gesture may be checked.

According to an embodiment of the present invention, by combining the voice recognition technology and the motion recognition technology to determine the vehicle control command, the driver can control the operation of the desired device in a range that does not disturb the driving even while driving.

In addition, the exemplary embodiment of the present invention may enter the command mode after recognizing the driver's hand gesture in the standby mode, and then determine a vehicle control command corresponding to the driver's voice and the hand gesture. Therefore, according to an embodiment of the present invention, by recognizing the voice after entering the command mode by the hand gesture to solve the existing problem that does not accurately recognize the driver's voice due to the noise of the vehicle passengers or noise generated inside and outside the vehicle. Can be.

In addition, according to an embodiment of the present invention, since a depth image of a hand gesture is also acquired by using a multi-view 3D camera, it is possible to more accurately determine whether the photographed hand gesture is for vehicle control or motion independent of vehicle control. have.

In addition, according to an embodiment of the present invention, since the speech recognition technology and the motion recognition technology are applied in a fusion manner, the vehicle control command can be recognized more accurately.

Further, according to an embodiment of the present invention, the 3D camera for motion recognition is provided at a position where the hand can be easily taken, such as the center fascia of the vehicle, and the microphone is a position where the driver's face is mainly directed, such as the ceiling or the dashboard of the vehicle. By providing it with the driver, the driver can reduce the risk of getting out of sight and operate the vehicle more easily.

1 is a diagram illustrating a vehicle control system including an intelligent vehicle control apparatus using an operation and voice according to an embodiment of the present invention;
2 is a view illustrating various examples of a hand gesture of a driver;
3 is a diagram illustrating an example of a vehicle control command corresponding to a recognized voice and a hand gesture of a driver according to an embodiment of the present invention;
4 is a diagram illustrating an example in which a 3D photographing unit and a microphone are installed in a vehicle among an intelligent vehicle control apparatus using an operation and voice according to an embodiment of the present invention;
5 is a flowchart illustrating an intelligent vehicle control method using operation and voice recognition according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features, and advantages of the present invention will become more readily apparent from the following description of preferred embodiments with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. Rather, the embodiments disclosed herein are provided so that the disclosure can be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

1 is a diagram illustrating a vehicle control system including an intelligent vehicle control apparatus 100 using an operation and voice according to an embodiment of the present invention.

Referring to FIG. 1, a vehicle control system may include an intelligent vehicle control apparatus 100, a vehicle control module 200, and a vehicle device 300 using motion and voice.

The vehicle device 300 is a device mounted inside or outside the vehicle, and may include various devices such as a radio 310, an air conditioner 320, a navigation 330, a mobile phone, a music player, a window, or a side mirror. . The driver or passenger riding in the vehicle may control the operation of the vehicle device 300 by using the hand gesture and the voice. This will be described later in detail with reference to the intelligent vehicle control apparatus 100 using operation and voice.

The vehicle control module 200 controls the overall operation of the vehicle and may be basically mounted in the vehicle. For example, when the driver manually adjusts the temperature of the air conditioner, the vehicle control module 200 may control the temperature by operating the air conditioner according to the manual adjustment of the driver. 1 illustrates an example in which the device 100 and the vehicle control module 200 are connected through the interface unit 120, but the device 100 may be directly connected to the vehicle device 300.

In an embodiment of the present disclosure, when the vehicle control command corresponding to the hand gesture and the voice is received from the apparatus 100, the vehicle control module 200 may perform an operation corresponding to the received vehicle control command. For example, if the received vehicle control command is to increase the volume of the radio, the vehicle control module 200 may automatically increase the volume of the radio.

The intelligent vehicle control apparatus 100 using an operation and voice according to an embodiment of the present invention receives a hand gesture and a voice from a user, and sends a vehicle control command corresponding to the received hand gesture and the voice to the vehicle control module 200 or the vehicle. It may be delivered directly to the device 300. The user may be a driver or a passenger of the vehicle, hereinafter, the driver will be described as an example.

To this end, the intelligent vehicle control device 100 using motion and voice includes a storage unit 110, an interface unit 120, a 3D (dimension) photographing unit 130, a motion recognition module 140, a microphone 150, It may include a voice recognition module 160 and the control unit 170.

The interface unit 120 may connect the device 100 and the vehicle control module 200 to communicate with each other. The interface unit 120 may be connected to the vehicle control module 200 by a short range wireless communication method such as Bluetooth, or may be connected to the vehicle control module 200 through a wired cable.

The storage unit 110 may store a vehicle control command corresponding to the voice and the hand gesture of the driver.

Table 1 shows an example of a vehicle control command stored in the storage 110.

Recognized voice Recognized hand gesture Vehicle control instruction radio Left action Volume down Right action Volume up Upward motion Channel search Downward motion Channel search below Air conditioner Left action Temperature up Right action Temperature down Upward motion Wind hard Downward motion Wind weakly Music player (CD player) Left action Go to the previous music Right action Go to the next music Upward motion Volume down Downward motion Volume up navigation Left action Move left on the map Right action Move map right telephone Voice of each number or recipient's name instead of hand gesture Attempt to connect the call to the receiver

Referring to [Table 1], the storage unit 110 maps and stores the volume of the radio as a vehicle control command corresponding to the voice of 'radio' and the 'left operation'.

Table 2 shows another example of the vehicle control command stored in the storage 110.

Recognized higher voice Recognized Subvoice Recognized hand gesture Vehicle control instruction radio Volume (sound) Left action Volume down Right action Volume up Channel (selection) Left action Channel search Right action Channel search below Air conditioner Temperature Left action Temperature up Right action Temperature down wind Left action Wind hard Right action Wind weakly Music player (CD player) - Left action Go to the previous music - Right action Go to the next music navigation - Left action Move left on the map - Right action Move map right telephone Each number or
Recipient's Name - Attempt to connect the call to the receiver

Referring to [Table 2], the storage unit 110 controls the radio volume as a vehicle control command corresponding to a first voice input 'radio' and a second voice input 'volume' and a left voice input. Command is stored in the mapping.

The 3D photographing unit 130 may be a 3D camera that photographs a subject as a 3D image by using a multi-view photographing method. For example, the 3D photographing unit 130 may include a left lens and a right lens side by side and simultaneously photograph the same subject to acquire a 2D image and a depth image. The 2D image is an image obtained by a general camera, and the depth image is an image in which the distance between the 3D photographing unit 130 and the subject is displayed in gray scale. When the gray scale is expressed as 0 to 255, the closer the distance between the subject and the 3D photographing unit 130 is, the greater the pixel of the subject may have a gray value.

In the present exemplary embodiment, the 3D photographing unit 130 photographs a driver's hand gesture and outputs an image of the photographed hand gesture as an example, but is not limited thereto. For example, the device 100 may determine the vehicle control command using the movement of the head as well as the hand gesture of the driver. In addition, hereinafter, for convenience of description, the 3D image output from the 3D photographing unit 130 may be mixed with a '2D image and a depth image' or 'photographed hand gesture'.

The motion recognition module 140 may recognize a hand motion photographed by the 3D photographing unit 130. That is, the motion recognition module 140 may recognize the direction in which the hand motion of the driver moves by analyzing the photographed hand motion. To this end, the motion recognition module 140 may include an image matcher 142, a depth value calculator 144, and a hand motion detector 146.

The image matching unit 142 may perform calibration, calibration, and the like on the 2D image and the depth image input from the 3D photographing unit 130 by using an algorithm. Calibration is the operation of correcting the coordinates of the 2D image and the depth image that are shifted due to the lens distortion of the 3D photographing unit 130 and the skew of the image sensor. This is to match the optical axes of the images captured from each other. Since the algorithm applied by the image matching unit 142 may use a well-known technique or an algorithm to be developed in the future, it will not be described in detail. The image matcher 142 may output the 2D image on which the matching is performed to the hand motion detector 146.

The depth calculator 144 may calculate a depth value from the depth image input from the 3D photographing unit 130. For example, the depth value calculator 144 may know the depth value for each pixel from the gray value of each pixel constituting the depth image. The depth calculator 144 may output the average value, the minimum depth value, or the maximum depth value of the depth values of each pixel to the hand motion detector 146 according to a set policy. Hereinafter, one of an average value, a minimum depth value, and a maximum depth value output from the depth value calculator 144 to the hand motion detector 146 is referred to as a “depth value” for convenience of description.

When the depth image of the hand gesture captured in the standby mode is less than a first threshold value, the hand gesture detector 146 may obtain a difference image of the 2D image of the hand gesture captured in the standby mode. The hand gesture detector 146 may output the obtained difference image to the controller 170.

In detail, the hand motion detector 146 may compare the depth value input from the depth value calculator 144 with a first threshold value. This means that when the driver moves his hand in front of the 3D photographing unit 130, if the depth value representing the distance between the hand and the 3D photographing unit 130 is greater than the first threshold value, the photographed hand gesture is a hand gesture irrelevant to the vehicle control command. This is to judge. For example, if the depth value is greater than the first threshold value, it means that a hand gesture generated at a distance greater than the first threshold value is photographed, since it may be a hand gesture accidentally generated by a passenger other than the driver.

When the input depth value is less than the first threshold value, the hand gesture detector 146 determines that the photographed hand gesture is valid, and compares the 2D image with the previously photographed 2D image to obtain a difference image. Since the difference image shows only a portion changed based on the previous image, the hand gesture detector 146 may determine how much the driver's hand has moved from the difference image compared to the previous image.

The hand motion detection unit 146 compares the difference image with a second predetermined threshold value, and if the difference image is less than the second threshold value, the photographed hand motion (that is, a hand motion taken by the driver) is a hand motion for mode switching. It can be recognized as. In addition, the hand motion detector 146 may notify the controller 170 that the hand motion for mode switching, which is the photographed hand motion, was performed. The mode switching hand gesture may be, for example, a shaking gesture in which the driver shakes his hand from side to side. For example, the hand gesture detector 146 may compare the average of the pixel values constituting the difference image and the second threshold value.

Meanwhile, after the device 100 enters the command mode and the driver's voice is recognized, the hand gesture detector 146 may recognize the re-photographed hand gesture when the driver's hand gesture is retaken. In this case, the recognized rephotographed hand gesture may be used to confirm the vehicle control command.

In detail, when the 2D image and the depth image of the re-photographed hand gesture are matched by the image matcher 142, the image matcher 142 outputs the matched 2D image and the depth image to the hand gesture detector 146. The hand gesture detector 146 may detect a portion of the 2D image corresponding to the hand as a region of interest (ROI) by using the 2D image corresponding to the re-photographed hand gesture. Alternatively, the hand motion detector 146 may recognize the center point or posture of the hand by using the depth value of the re-photographed 2D image. The reason for recognizing the ROI or the center of the hand is to recognize a more accurate and realistic hand gesture.

When the driver's hand is detected, the hand gesture detector 146 may segment the re-photographed 2D image to determine a start point and an end point of the moving direction among the 2D images. This is because it is necessary to know exactly where the hand movement starts and ends in order to recognize the hand movement accurately. Assuming a case where the hand recognizes the motion of moving from right to left, the driver moves the hand from left to right and then back to the center point (ie, the point where the hand first started moving to the right). Afterwards, since the driver performs the next operation, the hand gesture detection unit 146 may automatically perform the operation of starting and terminating the part exactly meant by the hand gesture for each hand gesture by the dividing operation.

In addition, the hand motion detector 146 may recognize the direction in which the detected hand moves by using the difference image. As a result, the hand motion detector 146 may recognize the motion of the re-photographed hand motion by combining the hand detection, the segmentation operation, and the moving direction of the hand.

2 is a diagram illustrating various examples of a hand gesture of a driver.

Referring to FIG. 2, the driver's hand motion includes a shaking motion of shaking a hand left and right, a push motion of pushing the hand forward, a right motion of moving the hand to the right, a left motion of moving the hand to the left, and an upward motion of moving the hand up. And a downward motion of moving the hand down. This means that the hand motion detector 146 may recognize at least one of a shaking motion, a right motion, a left motion, an upward motion, a downward motion, and a push motion from the driver's photographed hand motion. The shaking operation is used as a hand switch for mode switching, the push operation is used as a hand gesture for selection, and other operations can be used as a vehicle control hand gesture, which can be changed by the driver's operation or at the manufacturing stage.

The microphone 150 receives a driver's voice. When the microphone 150 continuously receives the driver's voice or is switched to the command mode by the controller 170 and the microphone 150 is in the on state, the microphone 150 receives the voice from the on state. Can be input.

The voice recognition module 160 may recognize a voice input through the microphone 150 based on the hand gesture recognized by the motion recognition module 140. In detail, the voice recognition module 160 controls the vehicle among the voices input by segmenting the voice output from the microphone 150 when the hand gesture recognized by the motion recognition module 140 is a hand gesture for mode switching. The voice associated with the command can be extracted and recognized. For example, when the driver speaks 'radio', the voice recognition module 160 may recognize that the driver speaks 'radio' from the input voice signal. In addition, the voice recognition module 160 may recognize two or more words that are continuously input from the driver. In this case, the first word may be recognized as an upper voice and the second word may be recognized as a lower voice. The speech recognition module 160 may segment and recognize speech using one of various well-known algorithms or an algorithm to be developed in the future.

Meanwhile, the voice recognition module 160 may recognize only the driver's voice according to the driver's operation and may not perform the voice recognition operation on the voice other than the driver. To this end, the voice recognition module 160 may store in advance a frequency or waveform corresponding to the driver's voice, and compare the input voice with a pre-stored frequency or waveform to determine whether the input voice is the driver's voice. have.

The controller 170 may control the overall operation of the apparatus 100 using a control program stored in a memory (not shown) or at least one processor.

In particular, in an embodiment of the present invention, the controller 170 may receive a command by the driver's voice and the hand gesture in the standby mode if the hand gesture recognized by the gesture recognition module 140 is a predetermined mode switch hand gesture. In the command mode, the voice recognition module 160 may be controlled to recognize the voice input through the microphone 150. That is, the controller 170 may be in the standby mode waiting for a driver command input until the driver for mode switching is recognized by the driver.

In detail, when the mode of the device 100 is switched to the command mode operating based on the hand gesture of the driver, the controller 170 turns on the microphone 150, and the motion recognition module 140 and the voice recognition The module 160 may initialize and synchronize the motion recognition module 140 and the voice recognition module 160 to recognize the hand gesture and the voice simultaneously or sequentially.

When the driver's voice is input through the microphone 150 within a predetermined time, the controller 170 controls the voice recognition module 160 to recognize the voice input through the microphone 150, and the driver's voice. If it is recognized, it may enter an operation mode corresponding to the recognized voice. The operation mode is a mode for controlling the operation of the device 300 mounted on the vehicle, and the device 300 may be at least one of a radio, an air conditioner, a mobile phone, a music player, a navigation, a window, and a side mirror as described above. .

In this case, the controller 170 may feed back information to the driver through the speaker (not shown) entering the operation mode.

For example, if the recognized voice is 'radio', the controller 170 may enter an operation mode for controlling 'radio'. Alternatively, when two voices, 'radio' and 'volume,' are sequentially recognized, the controller 170 enters an operation mode for controlling the 'volume of the radio' and feeds back the voice of 'radio volume' to the driver. can do.

After entering the operation mode corresponding to the recognized voice, when the driver's hand gesture is re-photographed by the 3D photographing unit 130, the controller 170 may control the gesture recognition module 140 to recognize the re-photographed hand gesture. Can be. As a result, the motion recognition module 140 may detect the hand using the ROI, and recognize the hand motion to be used for the vehicle control command using the segmentation work and the vehicle image.

When the driver's voice and the driver's hand gesture are re-photographed, the controller 170 may check, from the storage unit 110, the vehicle control command mapped to the operation mode corresponding to the recognized voice and the re-photographed hand gesture. In addition, the controller 170 may transmit the confirmed vehicle control command to the vehicle control module 200 connected through the interface unit 120. The vehicle control module 200 may control the operation of the vehicle device 300 corresponding to the vehicle control command.

3 is a diagram illustrating an example of a vehicle control command corresponding to a recognized voice and a hand gesture of a driver according to an exemplary embodiment of the present invention.

Referring to FIG. 3, radio, air conditioner, telephone, CD, and navigation are recognized upper voices, and volume (sound), channel (tuning), temperature, wind, and numbers corresponding to the upper voices correspond to upper voices. Is the lower voice. The hand gesture mapped to the lower voice may be a hand gesture for controlling an operation corresponding to the lower voice among devices corresponding to the upper voice. For example, when the upper voice called 'radio' and the lower voice called 'volume' are recognized after the switch to the command mode, and the 'left hand gesture' is recognized, the device 100 may detect the radio, volume and the left. As a vehicle control command corresponding to a hand gesture, the vehicle control command for reducing the volume of the radio can be confirmed. Accordingly, the apparatus 100 may request the vehicle control module 200 to decrease the volume of the radio, and the vehicle control module 200 may decrease the volume of the radio 310.

4 illustrates an example in which the 3D photographing unit 130 and the microphone 150 are installed in a vehicle among the intelligent vehicle control apparatus 100 using the operation and the voice according to an embodiment of the present invention.

Referring to FIG. 4, the 3D photographing unit 130 may be attached to the center fascia portion of the vehicle to easily photograph the hand gestures of the driver. The 3D photographing unit 130 may be attached to, for example, a dashboard or a steering wheel portion in addition to the center fascia portion.

In addition, the microphone 150 may be attached to the front surface of the driver's face, such as the ceiling of the vehicle, the steering wheel, or the dashboard.

5 is a flowchart illustrating an intelligent vehicle control method using operation and voice recognition according to an embodiment of the present invention.

The intelligent vehicle control apparatus for performing the intelligent vehicle control method using the operation and voice recognition of FIG. 5 may be the intelligent control apparatus 100 described with reference to FIG. 1 or at least one processor provided in the apparatus 100.

Referring to FIG. 5, the intelligent vehicle control apparatus is in a standby mode for receiving a control command by operation and voice from a driver (S505).

When the driver's hand gesture is photographed by the 3D photographing unit in the standby mode (S510), the intelligent vehicle controller may acquire a 2D image and a depth image corresponding to the hand gesture, and adjust the coordinates by matching the 2D image and the depth image. (S515).

The intelligent vehicle control apparatus may calculate a depth value from the depth image acquired in step S510 (S520). In operation S520, the intelligent vehicle control apparatus may calculate a depth value from gray values of the depth image.

The intelligent vehicle control apparatus may compare the calculated depth value with a predetermined first threshold value TH1 (S525).

As a result of the comparison, if the depth value is less than the first threshold value TH1 (S525-N), the intelligent vehicle control apparatus determines that the hand motion captured in step S510 is related to the vehicle control command, and generates a difference image of the 2D image. It may be (S530). The car image generated in operation S530 may be used to determine how much the driver's hand has moved compared to the previous image.

The intelligent vehicle control apparatus may compare the vehicle image generated in operation S530 with a second predetermined threshold value TH2 (S535).

As a result of the comparison, if the difference image is less than the second threshold value TH2 (S535-Y), the intelligent vehicle control apparatus recognizes that the hand gesture photographed in step S510 is a mode switching hand gesture and may switch to the command mode (S540). ). If the microphone is off after step S540, the intelligent vehicle control device switches the microphone on.

When the driver's voice is input through the microphone within a predetermined time after the switch to the command mode (S545-Y), the intelligent vehicle controller recognizes the input voice and enters the operation mode corresponding to the recognized voice. There is (S550). In operation S550, the intelligent vehicle control device may provide feedback to the driver through the speaker that the operation mode has been entered. In operation S550, the intelligent vehicle control apparatus may recognize the voice through a voice division operation.

When the user enters the operation mode corresponding to the voice, the intelligent vehicle control apparatus may recapture the hand gesture generated by the driver to acquire a 2D image and a depth image corresponding to the hand gesture (S555). The hand gesture that is re-photographed in step S555 is a hand gesture for use in a vehicle control command, not for mode switching.

The intelligent vehicle control apparatus may match the 2D image and the depth image obtained in operation S555 to correct the coordinates (S560).

The intelligent vehicle control apparatus may detect a portion of the 2D image corresponding to the hand as the region of interest and recognize the detected hand movement (S565). In operation S565, the intelligent vehicle control apparatus may recognize the re-photographed hand gesture by combining a hand detected as the ROI, a splitting operation of the 2D image corresponding to the detected hand, and a moving direction of the hand. The hand gesture recognized in operation S565 may be one of a left gesture, a right gesture, an upward gesture, a downward gesture, and a push gesture, but is not limited thereto.

When the hand gesture is recognized in operation S565, the intelligent vehicle control apparatus may check the vehicle control command stored by mapping the voice recognized in operation S550 and the hand gesture recognized in operation S565 (S570). The intelligent vehicle control apparatus may transmit an identified vehicle control command to the vehicle control module to request an operation corresponding to the vehicle control command. As a result, the vehicle control module may control the vehicle device to perform an operation of the vehicle device corresponding to the vehicle control command.

In the above-described embodiment, steps S545 to S550 and steps S555 to S565 may be linked and applied in a fusion manner.

While the present invention has been described with reference to the particular embodiments and drawings, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. This is possible. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: intelligent vehicle control device 200: vehicle control module
300: vehicle device 110: storage unit
120: interface unit 130: 3D imaging unit
140: motion recognition module 150: microphone
160: voice recognition module 170: control unit

Claims (14)

A 3D photographing unit which photographs and outputs a user's hand gesture;
A microphone for receiving a voice of the user;
A motion recognition module for recognizing a hand motion photographed by the 3D photographing unit;
A voice recognition module for recognizing a voice input through the microphone based on a hand gesture recognized by the motion recognition module; And
And a controller for checking a vehicle control command corresponding to the recognized voice and the re-photographed hand gesture when the hand gesture of the user who is re-photographed through the 3D photographing unit is recognized after the voice is recognized. Intelligent vehicle control using voice. The method of claim 1,
The controller controls the voice recognition module to recognize the voice input through the microphone when the hand gesture recognized by the motion recognition module is a predetermined mode switch hand gesture. Device. The method of claim 1,
If the hand gesture recognized by the motion recognition module is a hand gesture for mode switching requesting vehicle control, the controller switches from the standby mode to a command mode operating based on the voice of the user and the hand gesture. Intelligent vehicle control device using The method of claim 3,
The motion recognition module obtains a difference image of the 2D image of the hand motion photographed in the standby mode when the depth image of the hand motion photographed in the standby mode is less than a first predetermined threshold value.
And the controller determines that the hand gesture is the hand gesture for mode switching when the obtained difference image is less than a second predetermined threshold value. The method according to claim 3 or 4,
When the user's voice is recognized after the switch to the command mode, the controller enters an operation mode corresponding to the recognized voice, and when the re-photographed hand gesture is recognized, the controller enters an operation mode corresponding to the voice. And a vehicle control command mapped to the re-photographed hand gesture. The method of claim 5,
The operation mode is a mode for controlling the operation of one of the devices mounted on the vehicle, the operation device characterized in that it comprises at least one of radio, air conditioner, mobile phone, music player, navigation, window and side mirror And intelligent vehicle control apparatus using voice. The method of claim 1,
The user's hand gesture recognized by the motion recognition module may be a shaking motion of shaking the user's hand from side to side, a right operation of moving the user's hand to the right, a left operation of moving the user's hand to the left, and a movement of the user. Intelligent vehicle control apparatus using the operation and voice, characterized in that it comprises at least one of the upward motion to move the hand up and the downward motion to move the user's hand down. Photographing a user's hand gesture and outputting the image in a 3D image;
Recognizing the hand gesture from the output 3D image;
Recognizing a voice input through a microphone based on the recognized hand gesture;
Re-photographing the hand gesture of the user after the voice is recognized to recognize the re-photographed hand gesture;
And checking a vehicle control command corresponding to the recognized voice and the re-photographed hand gesture when the re-photographed hand gesture is recognized. 9. The method of claim 8,
The recognizing of the voice may be performed when the hand gesture recognized in the step of recognizing the hand gesture is a predetermined mode switching hand gesture. 10. The method according to claim 8 or 9,
If the hand gesture recognized in the step of recognizing the hand gesture is a hand gesture for mode switching requesting vehicle control, switching from a standby mode to a command mode operating based on the user's voice and hand gesture; Intelligent vehicle control method using the operation and voice. The method of claim 10,
The step of recognizing the hand-
When the depth image is less than the first threshold value by comparing a depth image of the hand gesture photographed in the standby mode with a first threshold value, a difference image of a 2D image of the hand gesture photographed in the standby mode Obtaining a;
And comparing the obtained second image with a second predetermined threshold value and determining that the hand gesture is the mode switching hand gesture when the difference image is less than the second threshold value. Intelligent vehicle control method using voice. The method of claim 10,
If the voice of the user is recognized after the switch to the command mode, entering the operation mode corresponding to the recognized voice;
The checking of the vehicle control command may include checking an operation mode corresponding to the voice and a vehicle control command mapped to the re-photographed hand gesture when the re-photographed hand gesture is recognized after entering the operation mode. Intelligent vehicle control method using the operation and voice. The method of claim 12,
The operation mode is a mode for controlling the operation of one of the devices mounted on the vehicle, wherein the device comprises at least one of a radio, an air conditioner, a mobile phone, a music player, a navigation, a window and a side mirror Intelligent vehicle control method using motion and voice. 9. The method of claim 8,
The user's hand gesture recognized in the step of recognizing the hand gesture may include a shaking operation of shaking the user's hand from side to side, a right operation of moving the user's hand to the right, a left operation of moving the user's hand to the left, and Intelligent vehicle control method using the operation and voice, characterized in that it comprises at least one of the upward motion to move the user's hand up and the downward motion to move the user's hand down.
KR1020110120027A 2011-11-17 2011-11-17 Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture KR101260053B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020110120027A KR101260053B1 (en) 2011-11-17 2011-11-17 Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020110120027A KR101260053B1 (en) 2011-11-17 2011-11-17 Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture

Publications (1)

Publication Number Publication Date
KR101260053B1 true KR101260053B1 (en) 2013-05-06

Family

ID=48665345

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020110120027A KR101260053B1 (en) 2011-11-17 2011-11-17 Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture

Country Status (1)

Country Link
KR (1) KR101260053B1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160084951A1 (en) * 2014-09-18 2016-03-24 Hyundai Motor Company System and method for recognizing a motion by analyzing a radio signal
KR101650769B1 (en) * 2015-05-28 2016-08-25 미디어젠(주) The vehicle-mounted voice recognition system by using gesture recognition
DE102016213313A1 (en) 2016-07-21 2018-01-25 Bayerische Motoren Werke Aktiengesellschaft Device and method for supporting a user
KR20230001968A (en) 2021-06-29 2023-01-05 혜윰기술 주식회사 Voice and gesture integrating device of vehicle

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110050589A1 (en) 2009-08-28 2011-03-03 Robert Bosch Gmbh Gesture-based information and command entry for motor vehicle
WO2011036618A2 (en) 2009-09-22 2011-03-31 Pebblestech Ltd. Remote control of computer devices

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110050589A1 (en) 2009-08-28 2011-03-03 Robert Bosch Gmbh Gesture-based information and command entry for motor vehicle
WO2011036618A2 (en) 2009-09-22 2011-03-31 Pebblestech Ltd. Remote control of computer devices

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160084951A1 (en) * 2014-09-18 2016-03-24 Hyundai Motor Company System and method for recognizing a motion by analyzing a radio signal
US9869760B2 (en) * 2014-09-18 2018-01-16 Hyundai Motor Company System and method for recognizing a motion by analyzing a radio signal
KR101650769B1 (en) * 2015-05-28 2016-08-25 미디어젠(주) The vehicle-mounted voice recognition system by using gesture recognition
DE102016213313A1 (en) 2016-07-21 2018-01-25 Bayerische Motoren Werke Aktiengesellschaft Device and method for supporting a user
WO2018015165A1 (en) 2016-07-21 2018-01-25 Bayerische Motoren Werke Aktiengesellschaft Device and method for assisting a user
KR20230001968A (en) 2021-06-29 2023-01-05 혜윰기술 주식회사 Voice and gesture integrating device of vehicle

Similar Documents

Publication Publication Date Title
KR101459441B1 (en) System and method for providing a user interface using finger start points shape recognition in a vehicle
US9576189B2 (en) Method and apparatus for controlling vehicle using motion recognition with face recognition
CN108473109B (en) Seamless vehicle access system
US7702130B2 (en) User interface apparatus using hand gesture recognition and method thereof
KR102061925B1 (en) Depth based context identification
CN110114825A (en) Speech recognition system
JP4311190B2 (en) In-vehicle device interface
US20100039224A1 (en) Biometrics information matching apparatus, biometrics information matching system, biometrics information matching method, person authentication apparatus, and person authentication method
KR101438615B1 (en) System and method for providing a user interface using 2 dimension camera in a vehicle
KR101260053B1 (en) Intelligent vehicle controlling apparatus and method using fusion recognition of user's voice and hand gesture
KR101459445B1 (en) System and method for providing a user interface using wrist angle in a vehicle
JP2009284442A (en) Apparatus and method for person identification
KR20140072734A (en) System and method for providing a user interface using hand shape trace recognition in a vehicle
KR102266354B1 (en) Apparatus and Method for Authenticating Biometric Information for Multi Preference
KR101808714B1 (en) Vehicle Center Fascia Control Method Based On Gesture Recognition By Depth Information And Virtual Touch Sensor
JP2007216920A (en) Seat controller for automobile, seat control program and on-vehicle navigation device
JP2018206073A (en) Remote operation system
KR101677641B1 (en) User recognition apparatus and method thereof
KR20130046759A (en) Apparatus and method for recogniting driver command in a vehicle
US11097745B2 (en) Driving support method, vehicle, and driving support system
KR102537879B1 (en) Active Control System of Dual Mic for Car And Method thereof
TWI737588B (en) System and method of capturing image
WO2022242589A1 (en) Method and apparatus for controlling rear-view mirror of vehicle, and vehicle and storage medium
JP2020075570A (en) Vehicle control device
KR101976498B1 (en) System and method for gesture recognition of vehicle

Legal Events

Date Code Title Description
A201 Request for examination
E701 Decision to grant or registration of patent right
GRNT Written decision to grant
FPAY Annual fee payment

Payment date: 20160401

Year of fee payment: 4

FPAY Annual fee payment

Payment date: 20170328

Year of fee payment: 5

FPAY Annual fee payment

Payment date: 20180406

Year of fee payment: 6

FPAY Annual fee payment

Payment date: 20190325

Year of fee payment: 7