US20200211539A1 - Home appliance and method for voice recognition thereof - Google Patents
Home appliance and method for voice recognition thereof Download PDFInfo
- Publication number
- US20200211539A1 US20200211539A1 US16/726,031 US201916726031A US2020211539A1 US 20200211539 A1 US20200211539 A1 US 20200211539A1 US 201916726031 A US201916726031 A US 201916726031A US 2020211539 A1 US2020211539 A1 US 2020211539A1
- Authority
- US
- United States
- Prior art keywords
- home appliance
- voice data
- noise
- data
- voice
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/20—Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
- G10L19/26—Pre-filtering or post-filtering
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L21/0232—Processing in the frequency domain
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/78—Detection of presence or absence of voice signals
- G10L25/84—Detection of presence or absence of voice signals for discriminating voice from noise
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/223—Execution procedure of a spoken command
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
- G10L2021/02165—Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
Definitions
- Apparatuses and methods consistent with the disclosure relate to a home appliance and a method for voice recognition thereof, and more particularly, to a home appliance which estimates noise using information on noise that occurs in another home appliance and reduces the estimated noise to increase a voice recognition rate, and a method for voice recognition thereof.
- Home appliances may perform various functions according to control commands from users. Recently, home appliances employ a voice recognition function to receive a control command through a user voice, as well as receiving a control command via an input device such as a keypad, a remote controller, and the like.
- home appliances to which a method of activating a voice recognition function based on a specific start command (e.g., Bixby) when a user speaks the specific start command has been increasingly adopted.
- a specific start command e.g., Bixby
- Embodiments of the disclosure overcome the above disadvantages and other disadvantages not described above. Also, the disclosure is not required to overcome the disadvantages described above, and an embodiment of the disclosure may not overcome any of the problems described above.
- the disclosure provides a home appliance which estimates noise using information on noise produced in another home appliance and reduces the estimated noise to increase a voice recognition rate, and a method for voice recognition thereof.
- a home appliance includes: a communication device configured to communicate with another home appliance; a microphone configured to receive a voice from a user; and a processor configured to perform signal processing on first voice data obtained from the microphone and perform voice recognition using the signal-processed first voice data, wherein the processor generates noise data using second voice data received from the other home appliance and performs the signal processing on the first voice data using the generated noise data.
- a method for voice recognition of a home appliance includes: obtaining first voice data from a microphone; performing signal processing on the obtained first voice data; and performing voice recognition using the signal-processed first voice data, wherein the performing of signal processing includes: receiving second voice data from another home appliance; generating noise data using the received second voice data; and performing signal processing on the first voice data using the generated noise data.
- a recording medium storing a program for performing a method for voice recognition, in which the method includes: requesting voice data from a first home appliance and a second home appliance; receiving first voice data and second voice data respectively obtained from the first home appliance and the second home appliance based on the request; generating noise data using the received second voice data; performing signal processing on the first voice data using the generated noise data; and transmitting the signal-processed first voice data to the first home appliance.
- various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium.
- application and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code.
- computer readable program code includes any type of computer code, including source code, object code, and executable code.
- computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.
- ROM read only memory
- RAM random access memory
- CD compact disc
- DVD digital video disc
- a “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals.
- a non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.
- FIG. 1 illustrates a block diagram of a voice recognition system according to an embodiment of the disclosure
- FIG. 2 is a block diagram illustrating a simple configuration of a first home appliance according to an embodiment of the disclosure
- FIG. 3 is a block diagram illustrating a specific configuration of a first home appliance according to an embodiment of the disclosure
- FIG. 4 is a block diagram illustrating a simple configuration of a second home appliance according to an embodiment of the disclosure
- FIG. 5 is a block diagram illustrating a simple configuration of a server according to an embodiment of the disclosure.
- FIG. 6 is a diagram illustrating a noise removing method according to a first embodiment
- FIG. 7 is a diagram illustrating a noise removing method according to the first embodiment
- FIG. 8 is a diagram illustrating a noise removing method according to a second embodiment
- FIG. 9 is a diagram illustrating a noise removing method according to the second embodiment.
- FIG. 10 is a diagram illustrating a noise removing method according to the second embodiment
- FIG. 11 is a diagram illustrating a noise removing method according to a third embodiment
- FIG. 12 is a diagram illustrating a noise removing method according to the third embodiment.
- FIG. 13 is a diagram illustrating a noise removing method according to the third embodiment.
- FIG. 14 is a sequence diagram illustrating a voice recognition method according to the first embodiment
- FIG. 15 is a sequence diagram illustrating a voice recognition method according to the first embodiment
- FIG. 16 is a sequence diagram illustrating a voice recognition method according to the first embodiment
- FIG. 17 is a flowchart illustrating a voice recognition method according to the second embodiment
- FIG. 18 is a flowchart illustrating a voice recognition method according to the second embodiment
- FIG. 19 is a flowchart illustrating a voice recognition method of a first home appliance according to an embodiment of the disclosure.
- FIG. 20 is a flowchart illustrating a voice recognition method of a second home appliance according to an embodiment of the disclosure.
- FIG. 21 is a flowchart illustrating a voice recognition method of a server according to an embodiment of the disclosure.
- FIGS. 1 through 21 discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged system or device.
- FIG. 1 is a diagram illustrating a voice recognition system according to an embodiment of the disclosure.
- a voice recognition system 1000 includes a first home appliance 100 , a second home appliance 200 , and a server 300 .
- the first home appliance 100 and the second home appliance 200 are electric machine tools used in a home, and include a robot cleaner, a vacuum cleaner, an electric range, a gas range, an electric wave oven, a range hood, a washing machine, a dryer, a refrigerator, a dishwasher, an air conditioner, and the like.
- the first home appliance 100 and the second home appliance 200 may perform a voice recognition function and perform a function according to a voice recognition result.
- voice recognition refers to a technique of converting an acoustic signal of an input speech into a word or sentence.
- the first home appliance 100 and the second home appliance 200 may detect a user speaking voice and perform voice recognition on the detected voice. Specifically, when a wake up word (WUW) which is a trigger voice command for activating the voice recognition function is detected, the first home appliance 100 and the second home appliance 200 activate the voice recognition function and perform voice recognition using voice data of a voice input thereafter.
- WUW wake up word
- the second home appliance 200 may be a home appliance that produces large magnitude noise.
- the second home appliance 200 may produce large magnitude noise due to a motor, a fan, or the like included therein.
- the first home appliance 100 may estimate the noise using information on the large magnitude noise produced in the second home appliance 200 and reduce the estimated noise to increase a voice recognition rate.
- the first home appliance 100 may transmit a recording request signal to the second home appliance 200 .
- the first home appliance 100 may receive second voice data obtained based on the recording request signal from the second home appliance 200 .
- the second voice data is voice data regarding noise produced in the second home appliance 200 .
- the first home appliance 100 may obtain first voice data based on the recording request signal.
- the first home appliance 100 may obtain first voice data and second voice data of the voice detected at the same time.
- the first home appliance 100 may generate noise data using the second voice data.
- the noise data refers to sound data corresponding to pure noise of a noise source.
- the first home appliance 100 may generate the noise data by applying the second voice data to a noise path estimation filter. A specific operation of generating noise data will be described later with reference to FIGS. 6 through 10 .
- the first home appliance 100 may signal-process the first voice data using the generated noise data. Specifically, the first home appliance 100 may remove noise included in the first voice data using the generated noise data.
- the first home appliance 100 may perform voice recognition using the first signal-processed voice data.
- the second home appliance 200 may be connected to the first home appliance 100 through wired or wireless communication.
- the second home appliance 200 may receive a recording request signal from the first home appliance 100 .
- the second home appliance 200 may obtain second voice data based on the received recording request signal.
- the obtained second voice data may be transmitted to the first home appliance 100 .
- the recording request signal transmission operation and the signal processing operation on the first voice data of the first home appliance 100 may also be performed by the server 300 .
- the server 300 may be connected to the first home appliance 100 and the second home appliance 200 through wired or wireless communication.
- the server 300 may transmit a recording request signal to the first home appliance 100 and the second home appliance 200 .
- the server 300 may receive first voice data obtained based on the recording request signal from the first home appliance 100 .
- the server 300 may receive second voice data obtained based on the recording request signal from the second home appliance 200 .
- the server 300 may generate noise data using the second voice data. Specifically, the server 300 may generate the noise data by applying the second voice data to the noise path estimation filter.
- the server 300 may signal-process the first voice data using the generated noise data.
- the server 300 may transmit the signal-processed first voice data to the first home appliance 100 .
- the first home appliance 100 is connected to one second home appliance 200 , but it may also be implemented such that the first home appliance 100 is connected to a plurality of home appliances and perform signal processing using information on noise produced in the plurality of home appliances.
- FIG. 1 it is illustrated and described that the server 300 is connected to two home appliances, but it may also be implemented such that the server 300 is connected to three or more home appliances.
- FIG. 1 it is illustrated and described that the server 300 is included in the voice recognition system 1000 , but only the first home appliance 100 and the second home appliance 200 may be implemented without the server 300 .
- FIG. 2 is a block diagram illustrating a simple configuration of a first home appliance according to an embodiment of the disclosure.
- the first home appliance 100 may include a microphone 110 , a communication device 120 , and a processor 130 .
- the microphone 110 may output a user's spoken voice and a surrounding sound as voice data.
- the microphone 110 may deliver the output voice data to the processor 130 .
- the microphone 110 may be disposed on a surface of the housing. Specifically, the microphone 110 may be disposed on a surface of the housing to generate first voice data corresponding to a sound near the first home appliance 100 in order to collect the user's spoken voice.
- an arrangement position of the microphone 110 is not limited to the example described above.
- the first home appliance 100 includes one microphone, but the first home appliance 100 may include two or more microphones when implemented.
- the communication device 120 may be connected to an external device and may receive various data from the external device.
- the communication device 120 may be connected to an external device through a local area network (LAN) and the Internet, as well as through a universal serial bus (USB) port or a wireless communication (e.g., Wi-Fi 802.11a/b/g/n, NFC, Bluetooth).
- LAN local area network
- USB universal serial bus
- the external device may be a PC, a notebook, a smartphone, a server, or the like.
- the communication device 120 may be connected to the second home appliance 200 or the server 300 in the voice recognition system 1000 .
- the communication device 120 may communicate with the second home appliance 200 in the voice recognition system 1000 through Bluetooth communication and communicate with the server 300 in the voice recognition system 1000 through Wi-Fi communication.
- the method for the first home appliance 100 to communicate with the second home appliance 200 or the server 300 is not limited to the example described above.
- the processor 130 controls the first home appliance 100 . Specifically, the processor 130 may control each component of the first home appliance 100 according to a user's control command. For example, when the first home appliance 100 is a refrigerator, the processor 130 may control an operation of a compressor to provide cooled air to a cooling chamber containing food based on a cooling command being received.
- the processor 130 may perform voice recognition using first voice data obtained through the microphone 110 .
- the first voice data may include noise as well as a user's voice, and a voice recognition rate may be lowered by the contained noise.
- the processor 130 may perform preprocessing on the first voice data.
- the preprocessing is a series of signal processing performed before voice recognition and may remove noise included in the voice data.
- the processor 130 may perform voice recognition using the preprocessed first voice data.
- the noise included in the first voice data may be due to the second home appliance 200 disposed near the first home appliance 100 and producing large magnitude noise. Therefore, the processor 130 obtains information on the noise produced in the second home appliance 200 from the second home appliance 200 and preprocesses the first voice data using the obtained information on the noise.
- the processor 130 may control the communication device 120 to detect another connectable home appliance. Specifically, the processor 130 may control the communication device 120 to detect the second home appliance connectable through short-range wireless communication or the second home appliance 200 connected to an access point (AP) to which the first home appliance 100 is connected.
- AP access point
- the second home appliance 200 available for short-range wireless communication with the first home appliance 100 or connected to the AP to which the first home appliance 100 is connected is expected to be disposed near the first home appliance 100 . Therefore, it may be inferred that the noise produced in the second home appliance 200 may affect the voice recognition function of the first home appliance 100 .
- the processor 130 may control the communication device 120 to request transmission of voice data from the second home appliance 200 .
- the processor 130 may control the communication device 120 to transmit a recording request signal for requesting generation and transmission of second voice data, which is voice data of the noise produced in the second home appliance 200 , to the second home appliance 200 .
- the recording request signal may include information on a predetermined time.
- the second home appliance 200 may detect noise using a microphone and generate second voice data at a predetermined time based on the received recording request signal. For example, when information on the predetermined time included in the recording request signal is 3:00:30 p.m., the second home appliance 200 may detect noise using the microphone at 3:00:30 p.m. and generate second voice data.
- the information on the predetermined time may be implemented to include absolute time information as in the example described above and may be implemented to include relative time information such as “x seconds after the recording request signal is received”.
- the recording request signal may include information on the predetermined time length of the second voice data.
- the second home appliance 200 may generate second voice data having a predetermined time length based on the received recording request signal. For example, when the information on the predetermined time length included in the recording request signal is 3 seconds, the second home appliance 200 may generate second voice data having a length of 3 seconds.
- the information that may be included in the recording request signal is not limited to the example described above.
- the processor 130 may receive second voice data generated based on the request transmitted to the second home appliance 200 from the second home appliance 200 .
- the processor 130 may receive the second voice data generated based on the predetermined time information and the predetermined time length information included in the recording request signal from the second home appliance 200 .
- the processor 130 may receive the second voice data regarding the voice detected through the microphone of the second home appliance 200 from 3:00:30 p.m. to 3:00:32 p.m.
- the processor 130 may obtain first voice data based on the request transmitted to the second home appliance 200 . Specifically, the processor 130 may obtain first voice data based on the predetermined time information and predetermined time length information included in the recording request signal transmitted to the second home appliance 200 .
- the processor 130 may obtain the first voice data regarding the voice detected through the microphone 110 from 3:00:30 p.m. to 3:00:32 p.m.
- the first voice data and the second voice data being obtained are generated based on the recording request signal, and thus correspond to voice data detected by the first home appliance 100 and the second home appliance 200 at the same time, respectively.
- the second voice data is voice data for noise directly detected in the second home appliance 200 , and thus may include a noise having a magnitude larger than that included in the first voice data.
- the noise included in the first voice data may be more accurately extracted than when only the first voice data is used.
- the processor 130 may generate noise data which is sound data corresponding to pure noise of a noise source in the second home appliance 200 using the second voice data.
- the processor 130 may use a noise path estimation filter to generate noise data.
- the noise path estimation filter refers to a filter for filtering components other than the noise of the noise source included in the second voice data.
- the noise path estimation filter may be referred to as a filtering algorithm.
- a user's speech component included in the second voice data may be filtered to output noise data.
- a specific operation of generating noise data using the noise path estimation filter will be described later with reference to FIGS. 6 through 10 .
- the processor 130 may use a pre-stored noise path estimation filter or receive a noise path estimation filter from an external device, and generate noise data using the received noise path estimation filter. Meanwhile, a specific operation of receiving the noise path estimation filter from outside and generating noise data will be described later with reference to FIGS. 11 through 13 .
- the processor 130 may perform preprocessing on the first voice data using the noise data. Specifically, the processor 130 may perform preprocessing on the first voice data by removing a component corresponding to noise of the noise source from the first voice data using the noise data generated through the noise path estimation filter.
- the operations of the processor 130 may be repeatedly performed according to a predetermined period. Specifically, the processor 130 may perform a sequential operation of requesting voice data from the second home appliance 200 , receiving second voice data obtained based on the request from the second home appliance 200 , obtaining first voice data corresponding to the request, generating noise data using the received second voice data, and performing preprocessing on the first voice data using the generated noise data, according to a predetermined period.
- the processor 130 may change the preprocessing method for the first voice data according to whether noise is produced in the second home appliance 200 at present.
- the processor 130 may change the preprocessing method for the first voice data according to whether the motor of the second home appliance 200 is driven.
- the processor 130 may control the communication device 120 to determine whether the motor of the second home appliance 200 is driven. If the motor of the second home appliance 200 is driven, the noise produced in the second home appliance 200 is large in magnitude, and thus, preprocessing may be performed on the first voice data using the second voice data according to the method described above.
- voice recognition may be immediately performed using the first voice data without using the second voice data or preprocessing may be performed on the first voice data according to the method of the related art.
- the processor 130 may change the preprocessing method for the first voice data according to whether the noise source is driven.
- the processor 130 may perform voice recognition using the preprocessed first voice data.
- the first home appliance performs the operation of transmitting the recording request signal to the second home appliance 200 , receiving the second voice data from the second home appliance 200 , and performing preprocessing on the first voice data, but it may be implemented such that the server 300 instead of the first home appliance 100 performs the operation described above and the first home appliance 100 simply receives preprocessed first voice data from the server 300 and performs voice recognition. A specific operation thereof will be described later with reference to FIG. 5 .
- FIG. 3 is a block diagram illustrating a specific configuration of a first home appliance according to an embodiment of the disclosure.
- the first home appliance 100 may include the microphone 110 , the communication device 120 , the processor 130 , an input device 140 , a memory 150 , and a display 160 .
- the microphone 110 and the communication device 120 perform the same functions as those of FIG. 2 , and thus redundant descriptions thereof will be omitted. Also, the processor 130 has been described above with reference to FIG. 2 , and thus, the descriptions of FIG. 2 are not repeated and only the contents related to the components added to FIG. 3 will be described below.
- the input device 140 may include a plurality of function keys for the user to set or select various functions supported by the first home appliance 100 . Through this, the user may input various control commands for the first home appliance 100 . For example, when the first home appliance 100 is a washing machine, the user may input a spin-dry command of the washing machine through the input device 140 .
- the memory 150 stores various data for an overall operation of the first home appliance 100 , such as a program for processing or controlling the processor 130 .
- the memory 150 may store a plurality of application programs run in the first home appliance 100 and data and instructions for operating the first home appliance 100 .
- the memory 150 may be accessed by the processor 130 , and data reading/writing/modifying/deleting/updating may be performed by the processor 130 .
- the memory 150 may be implemented not only as a storage medium in the first home appliance 100 but also as an external storage medium, a removable disk including a USB memory, a web server through a network, and the like.
- the memory 150 may store a noise path estimation filter necessary to generate noise data.
- the display 160 may display various types of information provided by the first home appliance 100 . Specifically, the display 160 may display an operation state of the first home appliance 100 or may display a user interface window for selecting a function and an option selected by the user.
- the display 160 may display that the washing machine is performing a spin-dry operation or display an interface window allowing the user to select how many minutes the spin-dry function is to be operated.
- the display 160 may display a result of performing the voice recognition function or may display an interface window so that the user may change a setting for the voice recognition function.
- a beamforming technology of obtaining a plurality of voice signals including a voice and noise through a plurality of microphones and separating the voice and the noise using features that the voice and noise are incident in different directions and frequency spectrums thereof are different is used to remove noise.
- the first home appliance obtains second voice data regarding the corresponding noise from another home appliance that produces large magnitude noise, and performs preprocessing using the obtained second voice data, whereby noise included in the voice data may be accurately removed even though a magnitude of the noise produced from outside is large.
- FIG. 4 is a block diagram illustrating a simple configuration of a second home appliance according to an embodiment of the disclosure.
- the second home appliance 200 may include a microphone 210 , a communication device 220 , a processor 240 , a motor 230 , an accelerometer 250 , an input device 260 , and a memory 270 .
- the microphone 210 is a device that converts a sound into a sound signal and may output a user's spoken voice and surrounding sounds as voice data.
- the microphone 210 may delivery the output voice data to the processor 240 .
- the microphone 210 may be disposed on a surface of the housing. Specifically, in order to collect the user's spoken voice, the microphone 210 may be disposed on the surface of the housing to generate second voice data corresponding to a sound around the second home appliance 200 .
- the microphone 210 may be disposed in the housing. Specifically, to collect a noise sound produced in the second home appliance 200 itself, the microphone 210 may be disposed inside the housing (specifically, near a noise source that produces noise such as a motor) to generate second voice data corresponding to the sound generated by the second home appliance 200 .
- an arrangement position of the microphone 210 is not limited to the example described above.
- the second home appliance 200 includes one microphone but the second home appliance 200 may include two or more microphones when implemented.
- the communication device 220 may be connected to an external device and receive various data from the external device. Specifically, the communication device 220 may be connected to an external device through a local area network (LAN) and the Internet, as well as through a universal serial bus (USB) port or a wireless communication Wi-Fi 802.11a/b/g/n, NFC, Bluetooth) port.
- the external device may be a PC, a notebook, a smartphone, a server, or the like.
- the communication device 220 may be connected to the first home appliance 100 or the server 300 in the voice recognition system 1000 .
- the communication device 220 may communicate with the first home appliance 100 in the voice recognition system 1000 through Bluetooth communication and communicate with the server 300 in the voice recognition system 1000 through Wi-Fi communication.
- the method for the second home appliance 200 to communicate with the first home appliance 100 or the server 300 is not limited to the example described above.
- the motor 230 is disposed in the second home appliance 200 to drive a component related to performing of a function of the second home appliance 200 .
- the motor 230 may rotate a drum containing laundry at a high speed for spin-dry of the laundry.
- vibration and noise may occur while the motor 230 starts the drum.
- the motor 230 may start a refrigerant compressor that generates a refrigerant.
- vibration and noise may be generated while the motor 230 starts the refrigerant compressor.
- the motor 230 may be a noise source when the second home appliance 200 produces large magnitude noise by itself. Therefore, the microphone 210 may be disposed in the vicinity of the motor 230 and detect a sound produced by the motor 230 and generate second voice data corresponding thereto.
- the processor 240 controls the second home appliance 200 . Specifically, the processor 240 may control each component of the second home appliance 200 according to a control command of the user. For example, when the second home appliance 200 is a washing machine, the processor 240 may control the operation of the motor to provide a rotational force to the drum containing the laundry when a spin-dry command is received.
- the processor 240 may receive a request for transmission of second voice data. Specifically, the processor 240 may receive a recording request signal requesting generation and transmission of the second voice data which is voice data regarding noise produced in the second home appliance 200 from the first home appliance 100 or the server 300 connected to the second home appliance 200 .
- the processor 240 may obtain second voice data based on the received request. Specifically, the processor 240 may detect noise using the microphone 210 based on the received recording request signal and obtain second voice data. More specifically, the processor 240 may obtain the second voice data based on information on a predetermined time and predetermined time length information included in the received recording request signal.
- the processor 240 may control the communication device 220 to transmit the obtained second voice data. Specifically, the processor 240 may transmit the obtained second voice data to the first home appliance 100 or the server 300 connected to the second home appliance 200 .
- the second voice data transmitted to the first home appliance 100 or the server 300 is voice data regarding noise produced in the second home appliance 200 , and thus, may be used to perform preprocessing to remove the noise included in the first voice data obtained in the first home appliance 100 .
- the first home appliance 100 or the server 300 may generate noise data which is pure motor noise data regarding the motor of the second home appliance 200 and perform preprocessing to remove the motor noise from the first voice data using the generated motor noise data.
- the processor 240 may obtain the second voice data at each predetermined period when the recording request signal is received, and transmit the obtained second voice data as described above.
- the processor 240 may also provide other types of reference data regarding the noise to the first home appliance 100 or the server 300 .
- the reference data may be data including information of the noise source.
- the information of the noise source may include a magnitude and phase of vibration produced from the noise source, a magnitude and phase of noise produced from the noise source, main frequency information, and the like.
- the processor 240 may generate noise data by obtaining reference data through the accelerometer 250 or by obtaining reference data through a control command input through the input device 260 . A specific operation thereof will be described below.
- the accelerometer 250 is a device for measuring acceleration of an object.
- the accelerometer 250 may be disposed in the vicinity of the motor 230 to measure acceleration of the motor 230 and generate information on the measured acceleration.
- the processor 240 may extract an operating frequency of the motor 230 from the obtained acceleration information and generate reference data using the extracted operating frequency.
- the processor 240 may generate reference data represented by a trigonometric function having a specific size and phase using 50 Hz.
- the input device 260 may include a plurality of function keys allowing the user to set or select various functions supported in the second home appliance 200 . Through this, the user may input various control commands regarding the second home appliance 200 .
- a control command input through the input device 260 may be related to driving of the motor 230 .
- an operating frequency of the motor 230 corresponding to the control command input through the input device 260 may be checked.
- the motor 230 may rotate the drum of the washing machine to perform a spin-dry function. In this case, it may be checked that the operating frequency of the motor 230 corresponding to the spin-dry command is 50 Hz.
- the processor 240 may generate reference data using an operating frequency of the motor 230 corresponding to the control command.
- This may be applied in the same manner to a control command generated by the processor 240 itself according to a situation determination.
- the processor 240 may transmit at least one of second voice data obtained through the microphone 210 and reference data generated using a driving frequency identified from the control command or the acceleration information of the accelerometer 250 to the first home appliance 100 or the server 300 .
- the first home appliance 100 or the server 300 may generate noise data which is sound data corresponding to pure noise of the noise source in the second home appliance 200 using the received reference data.
- the processor 240 may not transmit the above-described second voice data and reference data to the first home appliance 100 or the server 300 , and instead, the processor 240 may generate noise data using the second voice data and the reference data and transmit the generated noise data to the first home appliance 100 or the server 300 .
- a noise path estimation filter may be previously stored in the second home appliance 200 .
- the first home appliance 100 or the server 300 may receive noise data and perform preprocessing to remove noise included in the first voice data using the received noise data.
- the memory 270 stores various data for an overall operation of the second home appliance 200 , such as a program for processing or controlling of the processor 240 .
- the memory 270 may store a plurality of application programs driven in the second home appliance 200 and data and instructions for the operation of the second home appliance 200 .
- the memory 270 may store driving frequency information of the motor 230 corresponding to the control command input through the input device 260 .
- the processor 240 may identify a driving frequency corresponding to the input control command and generate reference data using the identified driving frequency.
- the noise source is illustrated and described as being the motor 230 in FIG. 4 , but the noise source may be a fan or another component, not the motor 230 when implemented.
- FIG. 4 a single motor is illustrated and described, but a plurality of motors may be provided at the time of implementation, a plurality of motors may be provided, and noise may be estimated using reference data for each of the plurality of motors.
- the second home appliance includes both a microphone and an accelerometer, but only the accelerometer may be provided at the time of implementation and reference data may be obtained through the accelerometer. Alternatively, only the microphone may be provided and second voice data may be obtained through the microphone. Alternatively, the second microphone and the accelerometer may not be provided, and reference data may be obtained through an input device.
- FIG. 4 it is illustrated and described that the operation of generating the reference data using the operating frequency of the motor is performed by the processor, but a sine wave signal generator generating a sine wave signal upon receiving operating frequency information may be provided at the time of implementation. In this case, the signal generated by the sine wave signal generator may be used as reference data.
- the second home appliance 200 is a home appliance having a different configuration from that of the first home appliance 100 , but the second home appliance 200 may be a home appliance having the same configuration as the first home appliance 100 at the time of implementation.
- the second home appliance generates reference data regarding the noise of the internal noise source and provides the generated reference data to the first home appliance or the server so that the first home appliance or the server may perform preprocessing to remove noise included in voice data including user's speech by using the reference data.
- FIG. 5 is a block diagram illustrating a simple configuration of a server according to an embodiment of the disclosure.
- the server 300 may include a communication device 310 , a memory 320 , and a processor 330 .
- the server 300 refers to a computer or a program that provides information or services to a client through a network.
- the communication device 310 may be connected to an external device and may receive various data from the external device. Specifically, the communication device 310 may be connected to an external device through a local area network (LAN) and the Internet, as well as through a universal serial bus (USB) port or wireless communication (Wi-Fi 802.11a/b/g/n, NFC, Bluetooth) port.
- the external device may be a PC, a notebook, a smartphone, a server, or the like.
- the communication device 310 may be connected to the first home appliance 100 or the second home appliance 200 in the voice recognition system 1000 .
- the communication device 310 may perform communication with the first home appliance 100 or the second home appliance 200 in the voice recognition system 1000 through Wi-Fi communication.
- the method for the server 300 to communicate with the first home appliance 100 or the second home appliance 200 is not limited to the example described above.
- the memory 320 stores various data for an overall operation of the server 300 such as a program for processing or controlling of the processor 330 .
- the memory 320 may store a plurality of application programs run in the server 300 and data and instructions for the operation of the server 300 .
- the memory 320 may be accessed by the processor 330 , and data reading/writing/modifying/deleting/updating may be performed by the processor 330 .
- the memory 320 may be implemented not only as a storage medium in the server 300 , but also as an external storage medium, a removable disk including a USB memory, a web server through a network, and the like.
- the memory 320 may store device information of a plurality of home appliances. Specifically, the memory 320 may store device information of a plurality of home appliances connected to the server 300 or a plurality of home appliances having a history of connection to the server.
- the device information of the home appliance may include serial number or universally unique identifier (UUID) information, and the information included is not limited to the example described above.
- the memory 320 may store a noise path estimation filter required to generate noise data.
- the memory 320 may store an artificial intelligence model that generates a noise path estimation filter. Meanwhile, a specific operation of generating a noise path estimation filter using an artificial intelligence model will be described later with reference to FIG. 11 .
- the processor 330 controls the server 300 . Specifically, the processor 330 may control each component of the server 300 according to a user's control command.
- the processor 330 may perform preprocessing on the voice data obtained by the first home appliance 100 in place of the first home appliance 100 . Specifically, the processor 330 may receive first voice data corresponding to sound in the vicinity the first home appliance 100 from the first home appliance 100 . In addition, preprocessing may be performed on the received first voice data.
- the processor 330 may perform preprocessing on the received first voice data using information on the noise produced in the second home appliance 200 .
- the processor 330 may receive information on the second home appliance 200 connected to the first home appliance 100 from the first home appliance 100 .
- the processor 330 may receive information on the second home appliance 200 which may be connected to the first home appliance 100 through short-range wireless communication or the second home appliance 200 connected to an access point (AP) to which the first home appliance 100 is connected, from the first home appliance 100 .
- AP access point
- the processor 330 may be connected to the second home appliance 200 . Specifically, the processor 330 may be connected to the second home appliance 200 using the information of the second home appliance 200 received from the first home appliance 100 .
- the processor 330 may control the communication device 310 to request voice data from the first home appliance 100 and the second home appliance 200 . Specifically, the processor 330 may control the communication device 310 to transmit a recording request signal for requesting generation and transmission of voice data to the first home appliance 100 and the second home appliance 200 .
- the recording request signal may include information on a predetermined time and information on a predetermined time length.
- the information included in the recording request signal is not limited to the example described above.
- the processor 330 may receive voice data generated based on the request. Specifically, the processor 330 may receive first voice data obtained based on the recording request signal from the first home appliance 100 . The processor 330 may receive second voice data obtained based on the recording request signal from the second home appliance 200 .
- the processor 330 may generate noise data which is sound data corresponding to pure noise of a noise source in the second home appliance 200 using the received second voice data.
- the processor 330 may receive other types of reference data regarding the noise in addition to the second voice data generated by detecting the noise produced in the second home appliance 200 .
- the processor 330 may receive reference data obtained through the accelerometer of the second home appliance 200 or reference data obtained through a control command input to the second home appliance 200 .
- the reference data may be generated based on noise source information including a magnitude and a phase of vibration produced from the noise source, a magnitude and a phase of noise produced from the noise source, and main frequency information.
- the processor 330 may generate noise data which is sound data corresponding to pure noise of the noise source in the second home appliance 200 using the received reference data.
- the processor 330 may use a noise path estimation filter to generate noise data. Meanwhile, a specific operation of generating noise data using the noise path estimation filter will be described later with reference to FIGS. 6 through 10 .
- the processor 330 may also be implemented to receive the noise data itself generated in the second home appliance 200 , without receiving the second voice data or the reference data from the second home appliance 200 .
- the processor 330 may perform preprocessing on the first voice data using the noise data. Specifically, the processor 330 may perform preprocessing on the first voice data by removing a component corresponding to the noise of the noise source from the first voice data using the noise data generated through the noise path estimation filter.
- the processor 330 may transmit the preprocessed first voice data to the first home appliance 100 .
- the first home appliance 100 may perform voice recognition using the preprocessed first voice data received from the server 300 .
- the server 300 receives the first voice data from the first home appliance which is to perform the voice recognition function, obtains the second voice data or reference data regarding the noise from the second home appliance that produces large magnitude noise, and performs preprocessing on the first voice data using the obtained second voice data or reference data, whereby the noise included in the voice data may be accurately removed even if the noise is large in magnitude.
- FIGS. 6 and 7 are diagrams illustrating a noise removing method according to a first embodiment.
- FIG. 6 is a simple block diagram illustrating a noise removing method according to the first embodiment.
- noise data y is generated using reference data r, voice data d, and a noise estimation algorithm, and noise of voice data d is removed using the generated noise data y.
- the reference data r may be at least one of the second voice data obtained through the microphone 210 of the second home appliance 200 , the acceleration information of the accelerometer 250 of the second home appliance 200 , or reference data generated using a driving frequency checked from the control command.
- the voice data d may correspond to the first voice data obtained by the microphone 110 of the first home appliance 100 .
- the processor 130 of the first home appliance 100 or the processor 330 of the server 300 may generate the noise data y using the noise estimation algorithm and the reference data r. Specifically, the processors 130 and 330 may extract the noise data y, which is sound data corresponding to a noise of the source noise, from the reference data r using the noise path estimation filter information included in the noise estimation algorithm.
- the noise path estimation filter may be implemented as a finite impulse response (FIR) filter or an infinite impulse response (IIR) filter in a time domain.
- the noise path estimation filter may be implemented in the form of a transfer function predetermined for each frequency band in a frequency domain.
- the noise path estimation filter may have a linear structure as in the example described above, but is not limited thereto and may have a non-linear structure.
- the noise path estimation filter may be fixed and used as one noise path estimation filter, and in case that a plurality of noise path estimation filter information are stored in advance, one of the noise path estimation filters may be selected and used to generate the noise data y according to situations.
- the processors 130 and 330 may perform preprocessing on the voice data d by removing a component corresponding to the noise of the noise source included in the voice data d using the generated noise data y.
- the processor 330 of the server 300 may perform preprocessing and transmit noise-removed voice data e to the first home appliance 100 .
- the processor 130 of the first home appliance 100 may perform voice recognition using the voice data e from which noise was removed by performing preprocessing.
- the processor 130 or 330 may update the method of generating the noise data y using the voice data d so that the accurate noise data y may be generated even when noise of the noise source is changed or a surrounding environment is changed.
- the processors 130 and 330 may update the noise path estimation filter using the voice data d including the noise. Details thereof will be described below with reference to FIG. 7 .
- FIG. 7 is a block diagram illustrating a method of removing noise in a frequency domain according to the first embodiment.
- a noise estimation algorithm using a noise path estimation filter H in the frequency domain and performing updating using voice data d including noise may be identified.
- the processors 130 and 330 may convert the reference data r into a frequency domain using fast Fourier transform (FFT).
- FFT fast Fourier transform
- the processors 130 and 330 may convert the noise data Y into the time domain using inverse fast Fourier transform (IFFT).
- IFFT inverse fast Fourier transform
- the processors 130 and 330 may use the converted noise data y to remove noise of the voice data d.
- the processors 130 and 330 may update the noise path estimation filter H using the voice data d with noise mixed therein. Specifically, the processor 130 or 330 may update the noise path estimation filter H using a correlation between the voice data D converted into the frequency domain through the FFT from the voice data d including noise and the converted reference data R.
- a k ⁇ 1th noise path estimation filter is H (k ⁇ 1)
- voice data including subsequent kth noise is d( k )
- kth reference data is r( k )
- voice data converted into kth frequency domain is D( k )
- reference data converted into the kth frequency domain is R( k ).
- f(k) ⁇ sqrt(V(k) ⁇ M(k) 2 )/
- M(k) (1 ⁇ ) ⁇ M(k ⁇ 1)+ ⁇ D (k)
- V(k) (1 ⁇ ) ⁇ V(k ⁇ 1)+ ⁇ D (k) ⁇ D (k) ( ⁇ , ⁇ , and ⁇ are constants).
- the processors 130 and 330 When the k+1th reference data (r (k+1) and voice data (d (k+1) ) are obtained, the processors 130 and 330 generate noise data y (k+1) from next reference data using an updated new noise path estimation filter H (k) and remove noise from the voice data d (k+1) using the generated noise data y (k+1) .
- the method of updating the noise path estimation filter H is not limited to the example described above.
- updating is performed each time the voice data including noise and the reference data are obtained, but it may be implemented such that updating is performed when voice data including a predetermined number of noise and reference data are obtained. In this case, updating may be performed using the voice data including the predetermined number of noise and the reference data together.
- FIGS. 6 and 7 it is illustrated and described that that the processors 130 and 330 update the noise path estimation filter after performing preprocessing on the voice data d including the noise but it may be implemented such that the noise path estimation filter may be updated first, and thereafter, preprocessing may be performed on the voice data d including noise.
- FIGS. 8 through 10 are diagrams illustrating a noise removing method according to a second embodiment.
- FIG. 8 is a simple block diagram illustrating a noise removing method according to a second embodiment.
- noise data y is generated using the reference data r, the noise-removed voice data e, and the noise estimation algorithm, and noise of the voice data d is removed using the generated noise data y.
- the reference data r may be at least one of the second voice data obtained through the microphone 210 of the second home appliance 200 and the reference data generated using a driving frequency identified from the control command or the acceleration information of the accelerometer 250 of the second home appliance 200 .
- the voice data d may correspond to the first voice data obtained from the microphone 110 of the first home appliance 100 .
- the processors 130 and 330 may extract noise data y, which is sound data corresponding to noise of the noise source, from the reference data r using the noise path estimation filter information included in the noise estimation algorithm.
- noise data y which is sound data corresponding to noise of the noise source
- the configuration of the noise estimation filter is the same as that of FIG. 7 , and thus, redundant description thereof will be omitted.
- the processors 130 and 330 may perform preprocessing on the voice data d by removing a component corresponding to the noise of the noise source included in the voice data d using the generated noise data y.
- the processor 330 of the server 300 may perform preprocessing to transmit the noise-removed voice data e to the first home appliance 100 .
- the processor 130 of the first home appliance 100 may perform voice recognition using the noise-removed voice data e by performing preprocessing.
- the processors 130 and 330 may update the method of generating noise data using the noise-removed voice data, rather than the voice data including the noise. Details thereof will be described below with reference to FIGS. 9 and 10 .
- FIG. 9 is a block diagram illustrating a method of removing noise in a time domain according to the second embodiment.
- a noise estimation algorithm using a noise path estimation filter which is an FIR filter, in the time domain and performing updating using noise-removed voice data may be identified.
- the processors 130 and 330 may update the noise path estimation filter h using the noise-removed voice data e. Specifically, after performing preprocessing, the processors 130 and 330 may update the noise path estimation filter h using a correlation between the noise-removed voice data e and the reference data r.
- a k ⁇ 1th noise path estimation filter is h (k ⁇ 1)
- kth noise-removed voice data is e (k)
- kth reference data is r (k)
- ⁇ a constant determined according to systems may be used or ⁇ may be variably used for stability of the algorithm.
- P(k) (1 ⁇ ) ⁇ P(k ⁇ 1)+ ⁇ r(k) T ⁇ r(k)) ( ⁇ and ⁇ are constants).
- the processors 130 and 330 When the processors 130 and 330 obtain next reference data r′ and voice data d′, the processors 130 and 330 generate noise data y′ from the next reference data using an updated new noise path estimation filter h′ and remove noise of the voice data d′ using the noise data y′, thus obtaining noise-removed voice data e′.
- the method of updating the noise path estimation filter h in the time domain is not limited to the example described above.
- updating is performed each time the noise-removed voice data and the reference data are obtained, but it may also be implemented such that updating is performed when the voice data from which a predetermined number of noise was removed and reference data are obtained. In this case, updating may be performed using the voice data from which the predetermined number of noise was removed and the reference data together.
- the noise path estimation filter h may be a filter implemented in the form of a predetermined transfer function for each frequency band in the frequency domain, rather than an FIR filter in the time domain, and the noise path estimation filter h may be updated using the noise-removed voice signal e. Details thereof will be described later with reference to FIG. 10 .
- FIG. 10 is a block diagram illustrating a method of removing noise in a frequency domain according to the second embodiment.
- a noise estimation algorithm using a noise path estimation filter in a frequency domain and performing updating using noise-removed voice data may be identified.
- the processors 130 and 330 may convert the reference data r into a frequency domain using the FFT.
- the processors 130 and 330 may convert the noise data Y into a time domain using IFFT.
- the processor 130 may use the converted noise data y to remove noise of the voice data d.
- the processors 130 and 330 may update the noise path estimation filter H in the frequency domain using the noise-removed voice data e. Specifically, the processors 130 and 330 may update the noise path estimation filter H using a correlation between the voice data E converted into the frequency domain through the FFT from the voice data e and the converted reference data R.
- k ⁇ 1th noise path estimation filter is H (k ⁇ 1)
- kth converted noise-removed voice data is E (k)
- kth reference data converted into the frequency domain is R (k)
- ⁇ a constant determined according to systems may be used or ⁇ may be variably used for stability of the algorithm.
- f(k) ⁇ sqrt(V(k) ⁇
- M(k) (1 ⁇ ) ⁇ M(k ⁇ 1)+ ⁇ D (k)
- V(k) (1 ⁇ ) ⁇ V(k ⁇ 1)+ ⁇ D (k) ⁇ D (k) ( ⁇ , ⁇ , and ⁇ are constants).
- the processors 130 and 330 may generate noise data y (k+1) from the next reference data using the updated new noise path estimation filter H (k) and noise may be removed from the voice data d (k+1) using the generated noise data y (k+1) .
- the method of updating the noise path estimation filter H is not limited to the example described above.
- updating is performed each time the noise-removed voice data and the reference data are obtained, but updating may also be performed when a predetermined number of noise-removed voice data and reference data are obtained. In this case, updating may be performed using the predetermined number of noise-removed voice data and the reference data together.
- FIGS. 8 through 10 it is illustrated and described that the processors 130 and 330 update the noise path estimation filter after performing preprocessing on the voice data d including noise, but it may also be implemented such that the noise path estimation filter is first updated and preprocessing is performed on the voice data d including noise.
- FIGS. 11 through 13 are diagrams illustrating a noise removing method according to a third embodiment.
- the information on the noise path estimation filter may be previously stored in the first home appliance 100 or the server 300 and used to generate noise data as described above.
- the noise path estimation filter may not be pre-stored in the manufacturing stage of the first home appliance 100 but may be generated through an artificial intelligence model after the first home appliance 100 is installed in a home.
- the server 300 may generate a noise path estimation filter suitable for a home appliance connected through an artificial intelligence model.
- the first home appliance 100 or the server 300 may perform noise removal using the generated noise path estimation filter.
- the first home appliance 100 obtains a noise path estimation filter from an external device 400 including an artificial intelligence model.
- FIG. 11 is a sequence diagram illustrating a method of obtaining a noise path estimation filter by the first home appliance 100 .
- the first home appliance 100 may obtain voice data and reference data at a time when there is no user's speech (S 1110 ). Specifically, in order to generate a noise path estimation filter, voice data and reference data in which there is no user's spoken voice and only noise of a noise source are detected are required. Therefore, when voice data determined not to have a user's speech is determined as a result of performing voice recognition, reference data obtained at the same time point as that of corresponding voice data may be checked.
- the first home appliance 100 may transmit the obtained voice data and reference data to the external device 400 .
- the first home appliance 100 may transmit the obtained voice data and reference data as a signal in a time domain or convert it into a frequency domain and transmit the same to the external device 400 .
- the external device 400 or the first home appliance 100 may request reference data at the corresponding time from the second home appliance 200 and obtain the reference data.
- the noise of the noise source of the second home appliance 200 may be different according to a change in an operation mode of the second home appliance 200 or an environment. Therefore, it is needed to generate a noise path estimation filter to be applied to each case.
- the first home appliance 100 may transmit information on an operation mode of the second home appliance 200 or information on a surrounding environment together, when transmitting information to the external device 400 .
- a rotation speed of the motor included in the washing machine may vary according to an operation mode. Therefore, a magnitude or characteristics of noise of the noise source (motor) may vary according to the operation mode.
- the information on each operation mode of the second home appliance 200 and the voice data and the reference data obtained when the second home appliance 200 operates in each operation mode may be transmitted together, so that the external device 400 may generate a noise path estimation filter applicable to each operation mode.
- the external device 400 may calculate a noise path estimation filter using the received voice data and reference data (S 1130 ). Specifically, the external device 400 may obtain a noise path estimation filter using an artificial intelligence model that receives voice data and reference data and outputs a noise path estimation filter corresponding thereto.
- the artificial intelligence model may be a linear regression model.
- the external device 400 may transmit the calculated noise path estimation filter information to the first home appliance 100 (S 1140 ). Meanwhile, when the second home appliance 200 is implemented in such a manner as to obtain second voice data and generate noise data using the obtained second voice data, the noise path estimation filter may be transmitted to the second home appliance 200 .
- the first home appliance 100 may store information of the received noise path estimation filter (S 1150 ).
- the first home appliance 100 may generate noise data using the obtained reference data and noise path estimation filter information, and remove noise of the voice data including user's spoken voice using the generated noise data.
- the first home appliance 100 may perform the voice recognition function using the noise-removed voice data.
- the external device is a device other than the voice recognition system but it may be implemented as a server in the voice recognition system.
- FIG. 11 it is illustrated and described that the voice data and the reference data are transmitted to the external device and the noise path estimation filter generated by an artificial intelligence mode is received from the external device, but the first home appliance may generate a noise path estimation filter using the previously stored artificial intelligence model when implemented.
- a separate device in the home appliance distinguished from the processor may generate a noise path estimation filter using the pre-stored artificial intelligence model, and the processor may use the generated noise path estimation filter.
- the method of generating the noise path estimation filter is not limited to the example described above.
- FIG. 12 is a block diagram illustrating a method of removing noise in a time domain according to the third embodiment.
- a noise removing method using a noise path estimation filter h in a time domain may be identified.
- the processor 130 of the first home appliance 100 may remove noise using the noise path estimation filter h received from the external device 400 .
- the processor 330 of the server 300 may perform noise removal using the generated noise path estimation filter h.
- the processors 130 and 330 may obtain the noise-removed voice data e by removing the noise of the voice data d using the noise data y.
- FIG. 13 is a block diagram illustrating a method of removing noise in a frequency domain according to the third embodiment.
- a noise removing method using a noise path estimation filter H in a frequency domain may be identified.
- the processors 130 and 330 may convert the reference data r into a frequency domain using FFT.
- the processors 130 and 330 may remove noise of the converted voice data D through the FFT using the noise data Y. In addition, the processors 130 and 330 may obtain the noise-removed voice data e by converting the noise-removed voice data E into a time domain using IFFT.
- FIGS. 12 and 13 it is illustrated and described that updating is not performed on the noise path estimation filter, but updating may be performed on the noise path estimation filter according to the updating method described above when implemented.
- FIGS. 14 through 16 are sequence diagrams illustrating a voice recognition method performed without intervention of a server.
- FIG. 14 is a sequence diagram illustrating a voice recognition method performed without intervention of a server according to the first embodiment of the disclosure.
- the first home appliance 100 may transmit a recording request signal to the second home appliance 200 (S 1410 ).
- the recording request signal may include information on a predetermined time and information on a predetermined time length.
- the first home appliance 100 may obtain first voice data corresponding to the recording request signal (S 1420 ). Specifically, the first home appliance 100 may obtain first voice data based on predetermined time information and predetermined time length information included in the recording request signal.
- the second home appliance 200 may obtain second voice data corresponding to the recording request signal (S 1425 ). Specifically, the second home appliance 200 may obtain second voice data based on the predetermined time information and the predetermined time length information included in the recording request signal.
- the second home appliance 200 may transmit the obtained second voice data to the first home appliance 100 (S 1430 ).
- the first home appliance 100 may generate noise data using the received second voice data (S 1440 ). Specifically, the first home appliance 100 may generate noise data which is sound data corresponding to pure noise of a noise source in the second home appliance 200 by applying the received second voice data to the noise path estimation filter.
- the first home appliance 100 may perform preprocessing on the first voice data using the noise data (S 1450 ).
- the first home appliance 100 may update the noise path estimation filter (S 1460 ). Specifically, the first home appliance 100 may update the noise path estimation filter using the second voice data and the first voice data or may update the noise path estimation filter using the second voice data and the preprocessed first voice data. Meanwhile, the method of updating the noise path estimation filter has been described above, and thus, a redundant description thereof will be omitted.
- the first home appliance 100 may perform voice recognition using the preprocessed first voice data (S 1470 ).
- FIG. 15 is a sequence diagram illustrating a voice recognition method performed without intervention of a server according to the second embodiment of the disclosure.
- the first home appliance 100 may transmit a recording request signal to the second home appliance 200 (S 1510 ).
- the first home appliance 100 may obtain first voice data corresponding to the recording request signal (S 1520 ).
- the second home appliance 200 may obtain second voice data corresponding to the recording request signal (S 1525 ).
- the second home appliance 200 may transmit the obtained second voice data to the first home appliance 100 (S 1530 ).
- the first home appliance 100 may update the noise path estimation filter before performing the preprocessing on the first voice data (S 1540 ).
- the first home appliance 100 may generate noise data using the received second voice data (S 1550 ). Specifically, the first home appliance 100 may generate noise data which is sound data corresponding to pure noise of a noise source in the second home appliance 200 by applying the received second voice data to the updated noise path estimation filter.
- the first home appliance 100 may perform preprocessing on the first voice data using the noise data (S 1560 ).
- the first home appliance 100 may perform voice recognition using the preprocessed first voice data (S 1570 ).
- FIG. 16 is a sequence diagram illustrating a voice recognition method performed without intervention of a server according to a third embodiment of the disclosure.
- the first home appliance 100 may transmit a recording request signal to the second home appliance 200 (S 1610 ).
- the first home appliance 100 may obtain first voice data corresponding to the recording request signal. Specifically, the first home appliance 100 may obtain first voice data based on predetermined time information and predetermined time length information included in the recording request signal.
- the second home appliance 200 may obtain second voice data corresponding to the recording request signal (S 1625 ).
- the second home appliance 200 may generate noise data using the second voice data (S 1630 ). Specifically, the second home appliance 200 may generate noise data which is sound data corresponding to pure noise of the noise source in the second home appliance 200 by applying the second voice data to the noise path estimation filter.
- the noise path estimation filter may be a filter previously stored in the second home appliance 200 or generated by an artificial intelligence model of the server 300 or the external device 400 and transmitted to the second home appliance 200 .
- the second home appliance 200 may transmit the generated noise data to the first home appliance 100 (S 1640 ).
- the first home appliance 100 may perform preprocessing on the first voice data using the received noise data (S 1650 ).
- the first home appliance 100 may perform voice recognition using the preprocessed first voice data (S 1660 ).
- the first home appliance 100 may estimate noise using information on the noise produced in the other home appliance without intervention of the server in various manners, and reduce the estimated noise.
- FIGS. 17 and 18 are sequence diagrams illustrating a voice recognition method performed according to intervention of a server.
- FIG. 17 is a sequence diagram illustrating a voice recognition method performed according to intervention of a server according to the first embodiment of the disclosure.
- the server 300 may transmit a recording request signal to the first home appliance 100 (S 1710 ).
- the server 300 may transmit a recording request signal to the second home appliance 200 (S 1715 ).
- the recording request signal may include information on a predetermined time and information on a predetermined time length.
- the first home appliance 100 may obtain first voice data corresponding to the recording request signal (S 1720 ). Specifically, the first home appliance 100 may obtain first voice data based on the predetermined time information and the predetermined time length information included in the recording request signal.
- the second home appliance 200 may obtain second voice data corresponding to the recording request signal (S 1725 ). Specifically, the second home appliance 200 may obtain second voice data based on the predetermined time information and the predetermined time length information included in the recording request signal.
- the first home appliance 100 may transmit the obtained first voice data to the server 300 (S 1730 ).
- the second home appliance 200 may transmit the obtained second voice data to the server 300 (S 1735 ).
- the server 300 may generate noise data using the received second voice data (S 1740 ). Specifically, the server 300 may generate noise data which is sound data corresponding to pure noise of the noise source in the second home appliance 200 by applying the received second voice data to the noise path estimation filter.
- the server 300 may perform preprocessing on the first voice data using the noise data (S 1750 ).
- the server 300 may transmit the preprocessed first voice data to the first home appliance 100 (S 1760 ).
- the server 300 may update the noise path estimation filter (S 1770 ). Specifically, the server 300 may update the noise path estimation filter using the second voice data and the first voice data or update the noise path estimation filter using the second voice data and the preprocessed first voice data. Meanwhile, the method of updating the noise path estimation filter has been described above, and thus, a redundant description thereof will be omitted.
- the first home appliance 100 may perform voice recognition using the preprocessed first voice data (S 1780 ).
- FIG. 18 is a sequence diagram illustrating a voice recognition method performed under intervention of a server according to the second embodiment of the disclosure.
- the server 300 may transmit a recording request signal to the first home appliance 100 (S 1810 ).
- the server 300 may transmit a recording request signal to the second home appliance 200 (S 1815 ).
- the first home appliance 100 may obtain first voice data corresponding to the recording request signal (S 1820 ).
- the second home appliance 200 may obtain second voice data based on the recording request signal (S 1825 ).
- the first home appliance 100 may transmit the obtained first voice data to the server 300 (S 1830 ).
- the second home appliance 200 may transmit the obtained second voice data to the server 300 (S 1835 ).
- the server 300 may update the noise path estimation filter before performing the preprocessing on the first voice data (S 1840 ).
- the operation of updating the noise path estimation filter before performing the preprocessing has been described above with reference to FIG. 15 , and thus, a redundant description thereof will be omitted.
- the server 300 may generate noise data using the received second voice data (S 1850 ). Specifically, the server 300 may generate noise data corresponding to pure noise of the noise source in the second home appliance 200 by applying the received second voice data to the updated noise path estimation filter.
- the server 300 may perform preprocessing on the first voice data using the noise data (S 1860 ).
- the server 300 may transmit the preprocessed first voice data to the first home appliance 100 (S 1870 ).
- the first home appliance 100 may perform voice recognition using the preprocessed first voice data (S 1880 ).
- the server may estimate the noise using the information on the noise produced in the second home appliance in various manners and reduce the estimated noise from the voice data of the first home appliance.
- the server 300 is connected to two home appliances but the server may be connected to two or more home appliances and estimate noise using information on the noise produced in the two or more home appliances when implemented.
- FIG. 19 is a flowchart illustrating a voice recognition method of a first home appliance according to an embodiment of the disclosure.
- first voice data is obtained through a microphone of a first home appliance (S 1910 ).
- a second home appliance which may be connected to the first home appliance through short-range wireless communication or a second home appliance connected to an access point AP to which the first home appliance is connected may be detected.
- the first home appliance may request transmission of voice data from the second home appliance.
- a recording request signal for requesting generation and transmission of second voice data which is voice data regarding noise produced in the second home appliance, may be transmitted to the second home appliance.
- the recording request signal may include information on a predetermined time or information on a predetermined time length. Meanwhile, the information that may be included in the recording request signal is not limited to the example described above.
- Second voice data is received from the second home appliance (S 1920 ). Specifically, the second voice data generated based on the predetermined time information and the predetermined time length information included in the recording request signal may be received from the second home appliance.
- First voice data may be obtained based on the request transmitted to the second home appliance. Specifically, first voice data may be obtained through a microphone of the first home appliance based on the predetermined time information and the predetermined time length information included in the recording request signal transmitted to the second home appliance.
- Noise data is generated using the received second voice data (S 1930 ).
- the noise data refers to sound data corresponding to noise of the noise source.
- the noise data may be obtained by extracting only a component corresponding to the noise of the noise source from the second voice data or filtering remaining components other than the component corresponding to the noise of the noise source.
- the noise data may be generated by filtering the remaining components other than the component corresponding to the noise source included in the second voice data using at least one of a finite impulse response (FIR) filter or an infinite impulse response (IIR) filter in the time domain.
- FIR finite impulse response
- IIR infinite impulse response
- the noise data may extract a component corresponding to the noise of the noise source from the second voice data using a transfer function previously determined for each frequency band on the frequency domain.
- the information on the above-described filter or transfer function may be pre-stored in a home appliance at a manufacturing stage and used to generate noise data, but is not limited thereto.
- information on a filter or a transfer function may be received through communication with the external device, and noise data may be generated using the information on the received filter or transfer function.
- the information on the filter or the transfer function received from the external device may be information obtained using an artificial intelligence model included in the external device.
- Signal processing is performed on the first voice data using the generated noise data (S 1940 ). Specifically, signal processing for removing a component corresponding to noise of a noise source included in the first voice data may be performed using the noise data.
- the noise source of the second home appliance may be a motor disposed in a housing of the second home appliance to perform a predetermined function of the second home appliance.
- the second voice data may be voice data obtained from a microphone of the second home appliance.
- the noise data may be motor noise data of the motor of the second home appliance, and preprocessing on the first voice data may be performed by removing the motor noise from the first voice data.
- a signal processing method for the first voice data may be changed according to whether noise is produced in the second home appliance.
- the preprocessing method regarding the first voice data may be changed depending on whether the motor of the second home appliance is driven.
- preprocessing on the first voice data may be performed using the second voice data according to the method described above.
- voice recognition may be performed directly using the first voice data without using the second voice data or preprocessing may be performed on the first voice data according to the related art method.
- the method of preprocessing on the first voice data may be changed according to whether the noise source is driven.
- Voice recognition is performed using the signal-processed first voice data (S 1950 ).
- the method of generating noise data may be updated using at least one of the first voice data including noise, the signal-processed first voice data, or the second voice data to generate accurate noise data even when noise of the noise source is changed or a surrounding environment is changed.
- the method of generating noise data may be updated using a correlation between at least one of the first voice data and the signal-processed first voice data and the second voice data.
- the operation of updating the method of generating noise data may be performed after signal processing is performed on the first voice data as described above, but alternatively, the operation of updating the method of generating noise data may be first performed before signal processing is performed on the first voice data.
- the voice data of the noise is obtained from another home appliance which produces large magnitude noise and the preprocessing is performed on the obtained noise using the voice data, whereby the noise included in the voice data obtained through the microphone may be accurately removed even though a magnitude of the noise produced from outside is large.
- the voice recognition method as shown in FIG. 19 may also be executed on a home appliance having the configuration of FIG. 2 or 3 or may be executed on a home appliance having another configuration.
- the voice recognition method as described above may be implemented by at least one executable program for executing the voice recognition method as described above, and such an executable program may be stored in a non-transitory readable medium.
- the non-transitory readable medium refers to a medium which stores data semi-permanently and is readable by a device, not a medium storing data for a short time such as a register, a cache, a memory, and the like.
- various applications or programs may be stored and provided in a non-transitory readable medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.
- FIG. 20 is a flowchart illustrating a voice recognition method of a second home appliance according to an embodiment of the disclosure.
- a voice data request signal may be received (S 2010 ).
- a recording request signal requesting generation and transmission of second voice data which is voice data of the noise produced in the second home appliance may be received from a server or a first home appliance connected to the second home appliance.
- Second voice data may be obtained through a microphone of the second home appliance (S 2020 ). Specifically, noise may be detected using the microphone of the second home appliance and the second voice data may be obtained based on the received recording request signal. More specifically, the second voice data may be obtained based on information on a predetermined time and predetermined time length information included in the received recording request signal.
- the obtained second voice data may be transmitted (S 2030 ). Specifically, the obtained second voice data may be transmitted to the server or the first home appliance connected to the second home appliance.
- the second voice data may not be transmitted to the server or the first home appliance connected to the second home appliance, and noise data may be generated using the second voice data and the generated noise data may be transmitted to the first home appliance or the server.
- the home appliance or the server may accurately remove the noise included in the voice data including user's speech.
- the voice recognition method as shown in FIG. 20 may be executed on a home appliance having the configuration of FIG. 4 or may be executed on a home appliance having another configuration.
- the voice recognition method as described above may be implemented by at least one executable program for executing the voice recognition method as described above, and such an executable program may be stored in a non-transitory readable medium.
- FIG. 21 is a flowchart illustrating a voice recognition method of a server according to an embodiment of the disclosure.
- voice data may be requested from a first home appliance and a second home appliance (S 2110 ).
- a recording request signal for requesting generation and transmission of voice data may be transmitted to the first home appliance and the second home appliance.
- the recording request signal may include information on a predetermined time and information on a predetermined time length.
- the information included in the recording request signal is not limited to the examples described above.
- First voice data and second voice data may be received (S 2120 ). Specifically, the first voice data obtained based on the recording request signal may be received from the first home appliance. The second voice data obtained based on the recording request signal may be received from the second home appliance.
- Noise data may be generated using the second voice data (S 2130 ).
- the noise data refers to sound data corresponding to pure noise of a noise source in the second home appliance.
- Signal processing may be performed on the first voice data using the noise data (S 2140 ). Specifically, signal processing for removing a component corresponding to the noise of the noise source included in the first voice data may be performed using the noise data.
- the signal-processed first voice data may be transmitted to the first home appliance (S 2150 ).
- the first voice data is received from the first home appliance which is to perform the voice recognition function
- the second voice data regarding the noise or reference data is obtained from the second home appliance that produces large magnitude noise
- preprocessing is performed on the first voice data using the obtained second voice data or reference data, so that the noise included in the voice data may be accurately removed even if the noise is large in magnitude.
- the voice recognition method as shown in FIG. 21 may be executed on a server having the configuration of FIG. 5 or may be executed on a server having another configuration.
- the voice recognition method as described above may be implemented with at least one executable program for executing the voice recognition method as described above, and such an executable program may be stored in a non-transitory readable medium.
Abstract
Description
- This application claims priority from Korean Patent Application No. 10-2018-0170469, filed on Dec. 27, 2018 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety.
- Apparatuses and methods consistent with the disclosure relate to a home appliance and a method for voice recognition thereof, and more particularly, to a home appliance which estimates noise using information on noise that occurs in another home appliance and reduces the estimated noise to increase a voice recognition rate, and a method for voice recognition thereof.
- Home appliances may perform various functions according to control commands from users. Recently, home appliances employ a voice recognition function to receive a control command through a user voice, as well as receiving a control command via an input device such as a keypad, a remote controller, and the like.
- In particular, home appliances to which a method of activating a voice recognition function based on a specific start command (e.g., Bixby) when a user speaks the specific start command has been increasingly adopted.
- Recently, it is common for a household to have and use a plurality of home appliances. Here, if large magnitude noise is produced in a home appliance, a voice recognition function of another home appliance may not operate smoothly due to the noise.
- Embodiments of the disclosure overcome the above disadvantages and other disadvantages not described above. Also, the disclosure is not required to overcome the disadvantages described above, and an embodiment of the disclosure may not overcome any of the problems described above.
- The disclosure provides a home appliance which estimates noise using information on noise produced in another home appliance and reduces the estimated noise to increase a voice recognition rate, and a method for voice recognition thereof.
- According to an embodiment of the disclosure, a home appliance includes: a communication device configured to communicate with another home appliance; a microphone configured to receive a voice from a user; and a processor configured to perform signal processing on first voice data obtained from the microphone and perform voice recognition using the signal-processed first voice data, wherein the processor generates noise data using second voice data received from the other home appliance and performs the signal processing on the first voice data using the generated noise data.
- According to another embodiment of the disclosure, a method for voice recognition of a home appliance includes: obtaining first voice data from a microphone; performing signal processing on the obtained first voice data; and performing voice recognition using the signal-processed first voice data, wherein the performing of signal processing includes: receiving second voice data from another home appliance; generating noise data using the received second voice data; and performing signal processing on the first voice data using the generated noise data.
- According to another embodiment of the disclosure, a recording medium storing a program for performing a method for voice recognition, in which the method includes: requesting voice data from a first home appliance and a second home appliance; receiving first voice data and second voice data respectively obtained from the first home appliance and the second home appliance based on the request; generating noise data using the received second voice data; performing signal processing on the first voice data using the generated noise data; and transmitting the signal-processed first voice data to the first home appliance.
- Additional and/or other aspects and advantages of the disclosure will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the disclosure.
- Before undertaking the DETAILED DESCRIPTION below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or,” is inclusive, meaning and/or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely.
- Moreover, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms “application” and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code. The phrase “computer readable program code” includes any type of computer code, including source code, object code, and executable code. The phrase “computer readable medium” includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.
- Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.
- The above and/or other aspects of the disclosure will be more apparent by describing certain embodiments of the disclosure with reference to the accompanying drawings, in which:
-
FIG. 1 illustrates a block diagram of a voice recognition system according to an embodiment of the disclosure; -
FIG. 2 is a block diagram illustrating a simple configuration of a first home appliance according to an embodiment of the disclosure; -
FIG. 3 is a block diagram illustrating a specific configuration of a first home appliance according to an embodiment of the disclosure; -
FIG. 4 is a block diagram illustrating a simple configuration of a second home appliance according to an embodiment of the disclosure; -
FIG. 5 is a block diagram illustrating a simple configuration of a server according to an embodiment of the disclosure; -
FIG. 6 is a diagram illustrating a noise removing method according to a first embodiment; -
FIG. 7 is a diagram illustrating a noise removing method according to the first embodiment; -
FIG. 8 is a diagram illustrating a noise removing method according to a second embodiment; -
FIG. 9 is a diagram illustrating a noise removing method according to the second embodiment; -
FIG. 10 is a diagram illustrating a noise removing method according to the second embodiment; -
FIG. 11 is a diagram illustrating a noise removing method according to a third embodiment; -
FIG. 12 is a diagram illustrating a noise removing method according to the third embodiment; -
FIG. 13 is a diagram illustrating a noise removing method according to the third embodiment; -
FIG. 14 is a sequence diagram illustrating a voice recognition method according to the first embodiment; -
FIG. 15 is a sequence diagram illustrating a voice recognition method according to the first embodiment; -
FIG. 16 is a sequence diagram illustrating a voice recognition method according to the first embodiment; -
FIG. 17 is a flowchart illustrating a voice recognition method according to the second embodiment; -
FIG. 18 is a flowchart illustrating a voice recognition method according to the second embodiment; -
FIG. 19 is a flowchart illustrating a voice recognition method of a first home appliance according to an embodiment of the disclosure; -
FIG. 20 is a flowchart illustrating a voice recognition method of a second home appliance according to an embodiment of the disclosure; and -
FIG. 21 is a flowchart illustrating a voice recognition method of a server according to an embodiment of the disclosure. -
FIGS. 1 through 21 , discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged system or device. - Terms used in the description of the various embodiments of the disclosure are briefly described and then the various example embodiments of the disclosure will be described in greater detail.
- The terms used in the example embodiments of the disclosure are general terms which are widely used now and selected considering the functions of the disclosure. However, the terms may vary depending on the intention of a person skilled in the art, a precedent, or the advent of new technology. In addition, in a specified case, the term may be arbitrarily selected. In this case, the meaning of the term will be explained in the corresponding description. Therefore, terms used in the disclosure may be defined based on a meaning of the terms and contents described in the disclosure, not simply based on names of the terms.
- Hereinafter, various embodiments of the disclosure are described with reference to the accompanying drawings. However, it should be appreciated that the disclosure is not limited to a specific embodiment and all modifications, equivalents and/or alternatives thereof also belong to the scope of the disclosure. Descriptions of well-known functions and structures incorporated herein may be omitted to avoid obscuring the subject matter of the disclosure.
- In the disclosure, terms including an ordinal number such as ‘first’, ‘second’, etc. may be used to describe various components, but the components are not to be construed as being limited to the terms. The terms are only used to differentiate one component from other components.
- In the description, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be understood that the terms “comprising”, “including”, “having” and variants thereof specify the presence of stated features, numbers, steps, operations, elements, components, and/or groups thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, elements, components, and/or groups thereof.
- Hereinafter, embodiments will be described in detail with reference to the accompanying tables and drawings such that they can be easily practiced by those skilled in the art to which the disclosure pertains. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the disclosure. In the accompanying drawings, a portion irrelevant to description of the disclosure will be omitted for clarity. Like reference numerals refer to like elements throughout.
- Hereinafter, the disclosure will be described in detail with reference to the accompanying drawings.
-
FIG. 1 is a diagram illustrating a voice recognition system according to an embodiment of the disclosure. - Referring to
FIG. 1 , avoice recognition system 1000 includes afirst home appliance 100, asecond home appliance 200, and aserver 300. - The
first home appliance 100 and thesecond home appliance 200 are electric machine tools used in a home, and include a robot cleaner, a vacuum cleaner, an electric range, a gas range, an electric wave oven, a range hood, a washing machine, a dryer, a refrigerator, a dishwasher, an air conditioner, and the like. - The
first home appliance 100 and thesecond home appliance 200 may perform a voice recognition function and perform a function according to a voice recognition result. Here, voice recognition refers to a technique of converting an acoustic signal of an input speech into a word or sentence. - In addition, the
first home appliance 100 and thesecond home appliance 200 may detect a user speaking voice and perform voice recognition on the detected voice. Specifically, when a wake up word (WUW) which is a trigger voice command for activating the voice recognition function is detected, thefirst home appliance 100 and thesecond home appliance 200 activate the voice recognition function and perform voice recognition using voice data of a voice input thereafter. - Hereinafter, for ease of explanation, it is assumed that the
first home appliance 100 performs a voice recognition function. Here, thesecond home appliance 200 may be a home appliance that produces large magnitude noise. Specifically, thesecond home appliance 200 may produce large magnitude noise due to a motor, a fan, or the like included therein. - In addition, due to the large magnitude noise produced in the
second home appliance 200, the voice recognition function of thefirst home appliance 100 may not operate properly. Therefore, thefirst home appliance 100 may estimate the noise using information on the large magnitude noise produced in thesecond home appliance 200 and reduce the estimated noise to increase a voice recognition rate. - To this end, the
first home appliance 100 may transmit a recording request signal to thesecond home appliance 200. In addition, thefirst home appliance 100 may receive second voice data obtained based on the recording request signal from thesecond home appliance 200. Here, the second voice data is voice data regarding noise produced in thesecond home appliance 200. - In addition, the
first home appliance 100 may obtain first voice data based on the recording request signal. - Accordingly, the
first home appliance 100 may obtain first voice data and second voice data of the voice detected at the same time. - In addition, the
first home appliance 100 may generate noise data using the second voice data. Here, the noise data refers to sound data corresponding to pure noise of a noise source. Specifically, thefirst home appliance 100 may generate the noise data by applying the second voice data to a noise path estimation filter. A specific operation of generating noise data will be described later with reference toFIGS. 6 through 10 . - In addition, the
first home appliance 100 may signal-process the first voice data using the generated noise data. Specifically, thefirst home appliance 100 may remove noise included in the first voice data using the generated noise data. - In addition, the
first home appliance 100 may perform voice recognition using the first signal-processed voice data. - The
second home appliance 200 may be connected to thefirst home appliance 100 through wired or wireless communication. Thesecond home appliance 200 may receive a recording request signal from thefirst home appliance 100. - In addition, the
second home appliance 200 may obtain second voice data based on the received recording request signal. The obtained second voice data may be transmitted to thefirst home appliance 100. - Meanwhile, the recording request signal transmission operation and the signal processing operation on the first voice data of the
first home appliance 100 may also be performed by theserver 300. - Specifically, the
server 300 may be connected to thefirst home appliance 100 and thesecond home appliance 200 through wired or wireless communication. Theserver 300 may transmit a recording request signal to thefirst home appliance 100 and thesecond home appliance 200. - In addition, the
server 300 may receive first voice data obtained based on the recording request signal from thefirst home appliance 100. Theserver 300 may receive second voice data obtained based on the recording request signal from thesecond home appliance 200. - In addition, the
server 300 may generate noise data using the second voice data. Specifically, theserver 300 may generate the noise data by applying the second voice data to the noise path estimation filter. - The
server 300 may signal-process the first voice data using the generated noise data. Theserver 300 may transmit the signal-processed first voice data to thefirst home appliance 100. - Meanwhile, in
FIG. 1 , it is illustrated and described that thefirst home appliance 100 is connected to onesecond home appliance 200, but it may also be implemented such that thefirst home appliance 100 is connected to a plurality of home appliances and perform signal processing using information on noise produced in the plurality of home appliances. - In addition, in
FIG. 1 , it is illustrated and described that theserver 300 is connected to two home appliances, but it may also be implemented such that theserver 300 is connected to three or more home appliances. - In addition, in
FIG. 1 , it is illustrated and described that theserver 300 is included in thevoice recognition system 1000, but only thefirst home appliance 100 and thesecond home appliance 200 may be implemented without theserver 300. -
FIG. 2 is a block diagram illustrating a simple configuration of a first home appliance according to an embodiment of the disclosure. - Referring to
FIG. 2 , thefirst home appliance 100 may include amicrophone 110, acommunication device 120, and aprocessor 130. - The
microphone 110, a device that converts sound into a sound signal, may output a user's spoken voice and a surrounding sound as voice data. Themicrophone 110 may deliver the output voice data to theprocessor 130. - The
microphone 110 may be disposed on a surface of the housing. Specifically, themicrophone 110 may be disposed on a surface of the housing to generate first voice data corresponding to a sound near thefirst home appliance 100 in order to collect the user's spoken voice. In addition, an arrangement position of themicrophone 110 is not limited to the example described above. - In addition, in the disclosure, it is illustrated and described that the
first home appliance 100 includes one microphone, but thefirst home appliance 100 may include two or more microphones when implemented. - The
communication device 120 may be connected to an external device and may receive various data from the external device. Specifically, thecommunication device 120 may be connected to an external device through a local area network (LAN) and the Internet, as well as through a universal serial bus (USB) port or a wireless communication (e.g., Wi-Fi 802.11a/b/g/n, NFC, Bluetooth). Here, the external device may be a PC, a notebook, a smartphone, a server, or the like. - In addition, the
communication device 120 may be connected to thesecond home appliance 200 or theserver 300 in thevoice recognition system 1000. For example, thecommunication device 120 may communicate with thesecond home appliance 200 in thevoice recognition system 1000 through Bluetooth communication and communicate with theserver 300 in thevoice recognition system 1000 through Wi-Fi communication. Meanwhile, the method for thefirst home appliance 100 to communicate with thesecond home appliance 200 or theserver 300 is not limited to the example described above. - The
processor 130 controls thefirst home appliance 100. Specifically, theprocessor 130 may control each component of thefirst home appliance 100 according to a user's control command. For example, when thefirst home appliance 100 is a refrigerator, theprocessor 130 may control an operation of a compressor to provide cooled air to a cooling chamber containing food based on a cooling command being received. - In addition, the
processor 130 may perform voice recognition using first voice data obtained through themicrophone 110. Meanwhile, the first voice data may include noise as well as a user's voice, and a voice recognition rate may be lowered by the contained noise. - Accordingly, the
processor 130 may perform preprocessing on the first voice data. Here, the preprocessing is a series of signal processing performed before voice recognition and may remove noise included in the voice data. In addition, theprocessor 130 may perform voice recognition using the preprocessed first voice data. - Meanwhile, the noise included in the first voice data may be due to the
second home appliance 200 disposed near thefirst home appliance 100 and producing large magnitude noise. Therefore, theprocessor 130 obtains information on the noise produced in thesecond home appliance 200 from thesecond home appliance 200 and preprocesses the first voice data using the obtained information on the noise. - Hereinafter, an operation of performing preprocessing on the first voice data using the information on the noise produced in the
second home appliance 200 will be described in detail. - First, the
processor 130 may control thecommunication device 120 to detect another connectable home appliance. Specifically, theprocessor 130 may control thecommunication device 120 to detect the second home appliance connectable through short-range wireless communication or thesecond home appliance 200 connected to an access point (AP) to which thefirst home appliance 100 is connected. - Here, the
second home appliance 200 available for short-range wireless communication with thefirst home appliance 100 or connected to the AP to which thefirst home appliance 100 is connected is expected to be disposed near thefirst home appliance 100. Therefore, it may be inferred that the noise produced in thesecond home appliance 200 may affect the voice recognition function of thefirst home appliance 100. - In addition, the
processor 130 may control thecommunication device 120 to request transmission of voice data from thesecond home appliance 200. Specifically, theprocessor 130 may control thecommunication device 120 to transmit a recording request signal for requesting generation and transmission of second voice data, which is voice data of the noise produced in thesecond home appliance 200, to thesecond home appliance 200. - Here, the recording request signal may include information on a predetermined time. The
second home appliance 200 may detect noise using a microphone and generate second voice data at a predetermined time based on the received recording request signal. For example, when information on the predetermined time included in the recording request signal is 3:00:30 p.m., thesecond home appliance 200 may detect noise using the microphone at 3:00:30 p.m. and generate second voice data. - Meanwhile, the information on the predetermined time may be implemented to include absolute time information as in the example described above and may be implemented to include relative time information such as “x seconds after the recording request signal is received”.
- In addition, the recording request signal may include information on the predetermined time length of the second voice data. The
second home appliance 200 may generate second voice data having a predetermined time length based on the received recording request signal. For example, when the information on the predetermined time length included in the recording request signal is 3 seconds, thesecond home appliance 200 may generate second voice data having a length of 3 seconds. - Meanwhile, the information that may be included in the recording request signal is not limited to the example described above.
- In addition, the
processor 130 may receive second voice data generated based on the request transmitted to thesecond home appliance 200 from thesecond home appliance 200. Specifically, theprocessor 130 may receive the second voice data generated based on the predetermined time information and the predetermined time length information included in the recording request signal from thesecond home appliance 200. - For example, when the predetermined time information included in the recording request signal is 3:00:30 p.m. and the predetermined time length information is 2 seconds, the
processor 130 may receive the second voice data regarding the voice detected through the microphone of thesecond home appliance 200 from 3:00:30 p.m. to 3:00:32 p.m. - In addition, the
processor 130 may obtain first voice data based on the request transmitted to thesecond home appliance 200. Specifically, theprocessor 130 may obtain first voice data based on the predetermined time information and predetermined time length information included in the recording request signal transmitted to thesecond home appliance 200. - For example, when the predetermined time information included in the recording request signal is 3:00:30 p.m. and the predetermined time length information is 2 seconds, the
processor 130 may obtain the first voice data regarding the voice detected through themicrophone 110 from 3:00:30 p.m. to 3:00:32 p.m. - Here, the first voice data and the second voice data being obtained are generated based on the recording request signal, and thus correspond to voice data detected by the
first home appliance 100 and thesecond home appliance 200 at the same time, respectively. The second voice data is voice data for noise directly detected in thesecond home appliance 200, and thus may include a noise having a magnitude larger than that included in the first voice data. - Accordingly, when the second voice data is used, the noise included in the first voice data may be more accurately extracted than when only the first voice data is used.
- In addition, the
processor 130 may generate noise data which is sound data corresponding to pure noise of a noise source in thesecond home appliance 200 using the second voice data. - The
processor 130 may use a noise path estimation filter to generate noise data. Here, the noise path estimation filter refers to a filter for filtering components other than the noise of the noise source included in the second voice data. Here, the noise path estimation filter may be referred to as a filtering algorithm. - For example, when the second voice data is input to the noise path estimation filter, a user's speech component included in the second voice data may be filtered to output noise data. Meanwhile, a specific operation of generating noise data using the noise path estimation filter will be described later with reference to
FIGS. 6 through 10 . - In addition, the
processor 130 may use a pre-stored noise path estimation filter or receive a noise path estimation filter from an external device, and generate noise data using the received noise path estimation filter. Meanwhile, a specific operation of receiving the noise path estimation filter from outside and generating noise data will be described later with reference toFIGS. 11 through 13 . - The
processor 130 may perform preprocessing on the first voice data using the noise data. Specifically, theprocessor 130 may perform preprocessing on the first voice data by removing a component corresponding to noise of the noise source from the first voice data using the noise data generated through the noise path estimation filter. - The operations of the
processor 130 may be repeatedly performed according to a predetermined period. Specifically, theprocessor 130 may perform a sequential operation of requesting voice data from thesecond home appliance 200, receiving second voice data obtained based on the request from thesecond home appliance 200, obtaining first voice data corresponding to the request, generating noise data using the received second voice data, and performing preprocessing on the first voice data using the generated noise data, according to a predetermined period. - Meanwhile, the
processor 130 may change the preprocessing method for the first voice data according to whether noise is produced in thesecond home appliance 200 at present. - For example, it is assumed that a sound generated in the
second home appliance 200 is produced by a motor included in thesecond home appliance 200. In this case, theprocessor 130 may change the preprocessing method for the first voice data according to whether the motor of thesecond home appliance 200 is driven. - Specifically, the
processor 130 may control thecommunication device 120 to determine whether the motor of thesecond home appliance 200 is driven. If the motor of thesecond home appliance 200 is driven, the noise produced in thesecond home appliance 200 is large in magnitude, and thus, preprocessing may be performed on the first voice data using the second voice data according to the method described above. - Meanwhile, when the motor of the
second home appliance 200 is not driven, there is no noise produced in thesecond home appliance 200 itself, and thus, voice recognition may be immediately performed using the first voice data without using the second voice data or preprocessing may be performed on the first voice data according to the method of the related art. - Meanwhile, even when the noise source of the
second home appliance 200 is a fan or the like, instead of the motor, theprocessor 130 may change the preprocessing method for the first voice data according to whether the noise source is driven. - The
processor 130 may perform voice recognition using the preprocessed first voice data. - Meanwhile, in
FIG. 2 , it is illustrated and described that the first home appliance performs the operation of transmitting the recording request signal to thesecond home appliance 200, receiving the second voice data from thesecond home appliance 200, and performing preprocessing on the first voice data, but it may be implemented such that theserver 300 instead of thefirst home appliance 100 performs the operation described above and thefirst home appliance 100 simply receives preprocessed first voice data from theserver 300 and performs voice recognition. A specific operation thereof will be described later with reference toFIG. 5 . - In addition, although only the simple components configuring the first home appliance has been illustrated and described above, various components may be further provided at the time of implementation. This will be described below with reference to
FIG. 3 . -
FIG. 3 is a block diagram illustrating a specific configuration of a first home appliance according to an embodiment of the disclosure. - Referring to
FIG. 3 , thefirst home appliance 100 according to an embodiment of the disclosure may include themicrophone 110, thecommunication device 120, theprocessor 130, aninput device 140, amemory 150, and adisplay 160. - The
microphone 110 and thecommunication device 120 perform the same functions as those ofFIG. 2 , and thus redundant descriptions thereof will be omitted. Also, theprocessor 130 has been described above with reference toFIG. 2 , and thus, the descriptions ofFIG. 2 are not repeated and only the contents related to the components added toFIG. 3 will be described below. - The
input device 140 may include a plurality of function keys for the user to set or select various functions supported by thefirst home appliance 100. Through this, the user may input various control commands for thefirst home appliance 100. For example, when thefirst home appliance 100 is a washing machine, the user may input a spin-dry command of the washing machine through theinput device 140. - The
memory 150 stores various data for an overall operation of thefirst home appliance 100, such as a program for processing or controlling theprocessor 130. Specifically, thememory 150 may store a plurality of application programs run in thefirst home appliance 100 and data and instructions for operating thefirst home appliance 100. - The
memory 150 may be accessed by theprocessor 130, and data reading/writing/modifying/deleting/updating may be performed by theprocessor 130. Thememory 150 may be implemented not only as a storage medium in thefirst home appliance 100 but also as an external storage medium, a removable disk including a USB memory, a web server through a network, and the like. - The
memory 150 may store a noise path estimation filter necessary to generate noise data. - The
display 160 may display various types of information provided by thefirst home appliance 100. Specifically, thedisplay 160 may display an operation state of thefirst home appliance 100 or may display a user interface window for selecting a function and an option selected by the user. - For example, when the
first home appliance 100 is a washing machine, thedisplay 160 may display that the washing machine is performing a spin-dry operation or display an interface window allowing the user to select how many minutes the spin-dry function is to be operated. Alternatively, thedisplay 160 may display a result of performing the voice recognition function or may display an interface window so that the user may change a setting for the voice recognition function. - In the related art, for preprocessing, a beamforming technology of obtaining a plurality of voice signals including a voice and noise through a plurality of microphones and separating the voice and the noise using features that the voice and noise are incident in different directions and frequency spectrums thereof are different is used to remove noise.
- However, if a magnitude of noise is larger than that of a voice in an actual environment, it is difficult to recognize an expected directional difference or spectral difference, leading to a problem that it is difficult to distinguish between the voice and the noise in the related art method. For example, when large magnitude noise is produced from other home appliances arranged in the vicinity, it is difficult to sufficiently remove noise for voice recognition by the related art alone.
- Meanwhile, as described above, the first home appliance according to the present embodiment obtains second voice data regarding the corresponding noise from another home appliance that produces large magnitude noise, and performs preprocessing using the obtained second voice data, whereby noise included in the voice data may be accurately removed even though a magnitude of the noise produced from outside is large.
-
FIG. 4 is a block diagram illustrating a simple configuration of a second home appliance according to an embodiment of the disclosure. - Referring to
FIG. 4 , thesecond home appliance 200 according to an embodiment of the disclosure may include amicrophone 210, acommunication device 220, aprocessor 240, amotor 230, anaccelerometer 250, aninput device 260, and amemory 270. - The
microphone 210 is a device that converts a sound into a sound signal and may output a user's spoken voice and surrounding sounds as voice data. Themicrophone 210 may delivery the output voice data to theprocessor 240. - The
microphone 210 may be disposed on a surface of the housing. Specifically, in order to collect the user's spoken voice, themicrophone 210 may be disposed on the surface of the housing to generate second voice data corresponding to a sound around thesecond home appliance 200. - Alternatively, the
microphone 210 may be disposed in the housing. Specifically, to collect a noise sound produced in thesecond home appliance 200 itself, themicrophone 210 may be disposed inside the housing (specifically, near a noise source that produces noise such as a motor) to generate second voice data corresponding to the sound generated by thesecond home appliance 200. - Meanwhile, an arrangement position of the
microphone 210 is not limited to the example described above. In addition, in the disclosure, although it is illustrated and described that thesecond home appliance 200 includes one microphone but thesecond home appliance 200 may include two or more microphones when implemented. - The
communication device 220 may be connected to an external device and receive various data from the external device. Specifically, thecommunication device 220 may be connected to an external device through a local area network (LAN) and the Internet, as well as through a universal serial bus (USB) port or a wireless communication Wi-Fi 802.11a/b/g/n, NFC, Bluetooth) port. The external device may be a PC, a notebook, a smartphone, a server, or the like. - In addition, the
communication device 220 may be connected to thefirst home appliance 100 or theserver 300 in thevoice recognition system 1000. For example, thecommunication device 220 may communicate with thefirst home appliance 100 in thevoice recognition system 1000 through Bluetooth communication and communicate with theserver 300 in thevoice recognition system 1000 through Wi-Fi communication. Meanwhile, the method for thesecond home appliance 200 to communicate with thefirst home appliance 100 or theserver 300 is not limited to the example described above. - The
motor 230 is disposed in thesecond home appliance 200 to drive a component related to performing of a function of thesecond home appliance 200. For example, when thesecond home appliance 200 is a washing machine, themotor 230 may rotate a drum containing laundry at a high speed for spin-dry of the laundry. Here, vibration and noise may occur while themotor 230 starts the drum. - As another example, when the
second home appliance 200 is a refrigerator, themotor 230 may start a refrigerant compressor that generates a refrigerant. Here, vibration and noise may be generated while themotor 230 starts the refrigerant compressor. - As described above, the
motor 230 may be a noise source when thesecond home appliance 200 produces large magnitude noise by itself. Therefore, themicrophone 210 may be disposed in the vicinity of themotor 230 and detect a sound produced by themotor 230 and generate second voice data corresponding thereto. - The
processor 240 controls thesecond home appliance 200. Specifically, theprocessor 240 may control each component of thesecond home appliance 200 according to a control command of the user. For example, when thesecond home appliance 200 is a washing machine, theprocessor 240 may control the operation of the motor to provide a rotational force to the drum containing the laundry when a spin-dry command is received. - The
processor 240 may receive a request for transmission of second voice data. Specifically, theprocessor 240 may receive a recording request signal requesting generation and transmission of the second voice data which is voice data regarding noise produced in thesecond home appliance 200 from thefirst home appliance 100 or theserver 300 connected to thesecond home appliance 200. - In addition, the
processor 240 may obtain second voice data based on the received request. Specifically, theprocessor 240 may detect noise using themicrophone 210 based on the received recording request signal and obtain second voice data. More specifically, theprocessor 240 may obtain the second voice data based on information on a predetermined time and predetermined time length information included in the received recording request signal. - In addition, the
processor 240 may control thecommunication device 220 to transmit the obtained second voice data. Specifically, theprocessor 240 may transmit the obtained second voice data to thefirst home appliance 100 or theserver 300 connected to thesecond home appliance 200. - As such, the second voice data transmitted to the
first home appliance 100 or theserver 300 is voice data regarding noise produced in thesecond home appliance 200, and thus, may be used to perform preprocessing to remove the noise included in the first voice data obtained in thefirst home appliance 100. - For example, when a noise source of the
second home appliance 200 is themotor 230, thefirst home appliance 100 or theserver 300 may generate noise data which is pure motor noise data regarding the motor of thesecond home appliance 200 and perform preprocessing to remove the motor noise from the first voice data using the generated motor noise data. - Meanwhile, as described above, in addition to the method in which the
processor 240 obtains the second voice data each time the recording request signal is received and transmits the obtained second voice data, theprocessor 240 may obtain the second voice data at each predetermined period when the recording request signal is received, and transmit the obtained second voice data as described above. - Meanwhile, in addition to the second voice data generated by detecting the noise produced in the
second home appliance 200, theprocessor 240 may also provide other types of reference data regarding the noise to thefirst home appliance 100 or theserver 300. - Here, the reference data may be data including information of the noise source. The information of the noise source may include a magnitude and phase of vibration produced from the noise source, a magnitude and phase of noise produced from the noise source, main frequency information, and the like.
- For example, the
processor 240 may generate noise data by obtaining reference data through theaccelerometer 250 or by obtaining reference data through a control command input through theinput device 260. A specific operation thereof will be described below. - The
accelerometer 250 is a device for measuring acceleration of an object. In addition, theaccelerometer 250 may be disposed in the vicinity of themotor 230 to measure acceleration of themotor 230 and generate information on the measured acceleration. - The
processor 240 may extract an operating frequency of themotor 230 from the obtained acceleration information and generate reference data using the extracted operating frequency. - For example, when 50 Hz is extracted as the operating frequency of the
motor 230 from acceleration information obtained from theaccelerometer 250, theprocessor 240 may generate reference data represented by a trigonometric function having a specific size and phase using 50 Hz. - The
input device 260 may include a plurality of function keys allowing the user to set or select various functions supported in thesecond home appliance 200. Through this, the user may input various control commands regarding thesecond home appliance 200. - Meanwhile, a control command input through the
input device 260 may be related to driving of themotor 230. In this case, an operating frequency of themotor 230 corresponding to the control command input through theinput device 260 may be checked. - For example, when the
second home appliance 200 is a washing machine and a spin-dry command is input, themotor 230 may rotate the drum of the washing machine to perform a spin-dry function. In this case, it may be checked that the operating frequency of themotor 230 corresponding to the spin-dry command is 50 Hz. - The
processor 240 may generate reference data using an operating frequency of themotor 230 corresponding to the control command. - This may be applied in the same manner to a control command generated by the
processor 240 itself according to a situation determination. - In this manner, the
processor 240 may transmit at least one of second voice data obtained through themicrophone 210 and reference data generated using a driving frequency identified from the control command or the acceleration information of theaccelerometer 250 to thefirst home appliance 100 or theserver 300. In addition, thefirst home appliance 100 or theserver 300 may generate noise data which is sound data corresponding to pure noise of the noise source in thesecond home appliance 200 using the received reference data. - Meanwhile, the
processor 240 may not transmit the above-described second voice data and reference data to thefirst home appliance 100 or theserver 300, and instead, theprocessor 240 may generate noise data using the second voice data and the reference data and transmit the generated noise data to thefirst home appliance 100 or theserver 300. To this end, a noise path estimation filter may be previously stored in thesecond home appliance 200. - In this case, the
first home appliance 100 or theserver 300 may receive noise data and perform preprocessing to remove noise included in the first voice data using the received noise data. - The
memory 270 stores various data for an overall operation of thesecond home appliance 200, such as a program for processing or controlling of theprocessor 240. Specifically, thememory 270 may store a plurality of application programs driven in thesecond home appliance 200 and data and instructions for the operation of thesecond home appliance 200. - For example, the
memory 270 may store driving frequency information of themotor 230 corresponding to the control command input through theinput device 260. Theprocessor 240 may identify a driving frequency corresponding to the input control command and generate reference data using the identified driving frequency. - Meanwhile, although the noise source is illustrated and described as being the
motor 230 inFIG. 4 , but the noise source may be a fan or another component, not themotor 230 when implemented. - In addition, in
FIG. 4 , a single motor is illustrated and described, but a plurality of motors may be provided at the time of implementation, a plurality of motors may be provided, and noise may be estimated using reference data for each of the plurality of motors. - In addition, in
FIG. 4 , it is illustrated and described that the second home appliance includes both a microphone and an accelerometer, but only the accelerometer may be provided at the time of implementation and reference data may be obtained through the accelerometer. Alternatively, only the microphone may be provided and second voice data may be obtained through the microphone. Alternatively, the second microphone and the accelerometer may not be provided, and reference data may be obtained through an input device. - In addition, in
FIG. 4 , it is illustrated and described that the operation of generating the reference data using the operating frequency of the motor is performed by the processor, but a sine wave signal generator generating a sine wave signal upon receiving operating frequency information may be provided at the time of implementation. In this case, the signal generated by the sine wave signal generator may be used as reference data. - In addition, in
FIG. 4 , it is illustrated and described that thesecond home appliance 200 is a home appliance having a different configuration from that of thefirst home appliance 100, but thesecond home appliance 200 may be a home appliance having the same configuration as thefirst home appliance 100 at the time of implementation. - As described above, the second home appliance generates reference data regarding the noise of the internal noise source and provides the generated reference data to the first home appliance or the server so that the first home appliance or the server may perform preprocessing to remove noise included in voice data including user's speech by using the reference data.
-
FIG. 5 is a block diagram illustrating a simple configuration of a server according to an embodiment of the disclosure. - Referring to
FIG. 5 , theserver 300 may include acommunication device 310, amemory 320, and aprocessor 330. - The
server 300 refers to a computer or a program that provides information or services to a client through a network. - The
communication device 310 may be connected to an external device and may receive various data from the external device. Specifically, thecommunication device 310 may be connected to an external device through a local area network (LAN) and the Internet, as well as through a universal serial bus (USB) port or wireless communication (Wi-Fi 802.11a/b/g/n, NFC, Bluetooth) port. The external device may be a PC, a notebook, a smartphone, a server, or the like. - In addition, the
communication device 310 may be connected to thefirst home appliance 100 or thesecond home appliance 200 in thevoice recognition system 1000. For example, thecommunication device 310 may perform communication with thefirst home appliance 100 or thesecond home appliance 200 in thevoice recognition system 1000 through Wi-Fi communication. Meanwhile, the method for theserver 300 to communicate with thefirst home appliance 100 or thesecond home appliance 200 is not limited to the example described above. - The
memory 320 stores various data for an overall operation of theserver 300 such as a program for processing or controlling of theprocessor 330. Specifically, thememory 320 may store a plurality of application programs run in theserver 300 and data and instructions for the operation of theserver 300. - The
memory 320 may be accessed by theprocessor 330, and data reading/writing/modifying/deleting/updating may be performed by theprocessor 330. Thememory 320 may be implemented not only as a storage medium in theserver 300, but also as an external storage medium, a removable disk including a USB memory, a web server through a network, and the like. - The
memory 320 may store device information of a plurality of home appliances. Specifically, thememory 320 may store device information of a plurality of home appliances connected to theserver 300 or a plurality of home appliances having a history of connection to the server. The device information of the home appliance may include serial number or universally unique identifier (UUID) information, and the information included is not limited to the example described above. - The
memory 320 may store a noise path estimation filter required to generate noise data. Alternatively, thememory 320 may store an artificial intelligence model that generates a noise path estimation filter. Meanwhile, a specific operation of generating a noise path estimation filter using an artificial intelligence model will be described later with reference toFIG. 11 . - The
processor 330 controls theserver 300. Specifically, theprocessor 330 may control each component of theserver 300 according to a user's control command. - In addition, the
processor 330 may perform preprocessing on the voice data obtained by thefirst home appliance 100 in place of thefirst home appliance 100. Specifically, theprocessor 330 may receive first voice data corresponding to sound in the vicinity thefirst home appliance 100 from thefirst home appliance 100. In addition, preprocessing may be performed on the received first voice data. - Meanwhile, when the
second home appliance 200 is disposed in the vicinity of thefirst home appliance 100 and produces large magnitude noise, theprocessor 330 may perform preprocessing on the received first voice data using information on the noise produced in thesecond home appliance 200. - Hereinafter, a specific operation of performing preprocessing on the first voice data using information on noise produced in the
second home appliance 200 will be described. - First, the
processor 330 may receive information on thesecond home appliance 200 connected to thefirst home appliance 100 from thefirst home appliance 100. Specifically, theprocessor 330 may receive information on thesecond home appliance 200 which may be connected to thefirst home appliance 100 through short-range wireless communication or thesecond home appliance 200 connected to an access point (AP) to which thefirst home appliance 100 is connected, from thefirst home appliance 100. - The
processor 330 may be connected to thesecond home appliance 200. Specifically, theprocessor 330 may be connected to thesecond home appliance 200 using the information of thesecond home appliance 200 received from thefirst home appliance 100. - The
processor 330 may control thecommunication device 310 to request voice data from thefirst home appliance 100 and thesecond home appliance 200. Specifically, theprocessor 330 may control thecommunication device 310 to transmit a recording request signal for requesting generation and transmission of voice data to thefirst home appliance 100 and thesecond home appliance 200. - The recording request signal may include information on a predetermined time and information on a predetermined time length. The information included in the recording request signal is not limited to the example described above.
- The
processor 330 may receive voice data generated based on the request. Specifically, theprocessor 330 may receive first voice data obtained based on the recording request signal from thefirst home appliance 100. Theprocessor 330 may receive second voice data obtained based on the recording request signal from thesecond home appliance 200. - The
processor 330 may generate noise data which is sound data corresponding to pure noise of a noise source in thesecond home appliance 200 using the received second voice data. - In addition, the
processor 330 may receive other types of reference data regarding the noise in addition to the second voice data generated by detecting the noise produced in thesecond home appliance 200. - For example, the
processor 330 may receive reference data obtained through the accelerometer of thesecond home appliance 200 or reference data obtained through a control command input to thesecond home appliance 200. Here, the reference data may be generated based on noise source information including a magnitude and a phase of vibration produced from the noise source, a magnitude and a phase of noise produced from the noise source, and main frequency information. - The
processor 330 may generate noise data which is sound data corresponding to pure noise of the noise source in thesecond home appliance 200 using the received reference data. - The
processor 330 may use a noise path estimation filter to generate noise data. Meanwhile, a specific operation of generating noise data using the noise path estimation filter will be described later with reference toFIGS. 6 through 10 . - The
processor 330 may also be implemented to receive the noise data itself generated in thesecond home appliance 200, without receiving the second voice data or the reference data from thesecond home appliance 200. - The
processor 330 may perform preprocessing on the first voice data using the noise data. Specifically, theprocessor 330 may perform preprocessing on the first voice data by removing a component corresponding to the noise of the noise source from the first voice data using the noise data generated through the noise path estimation filter. - In addition, the
processor 330 may transmit the preprocessed first voice data to thefirst home appliance 100. In addition, thefirst home appliance 100 may perform voice recognition using the preprocessed first voice data received from theserver 300. - In this manner, the
server 300 receives the first voice data from the first home appliance which is to perform the voice recognition function, obtains the second voice data or reference data regarding the noise from the second home appliance that produces large magnitude noise, and performs preprocessing on the first voice data using the obtained second voice data or reference data, whereby the noise included in the voice data may be accurately removed even if the noise is large in magnitude. -
FIGS. 6 and 7 are diagrams illustrating a noise removing method according to a first embodiment. -
FIG. 6 is a simple block diagram illustrating a noise removing method according to the first embodiment. - Referring to
FIG. 6 , noise data y is generated using reference data r, voice data d, and a noise estimation algorithm, and noise of voice data d is removed using the generated noise data y. - Here, the reference data r may be at least one of the second voice data obtained through the
microphone 210 of thesecond home appliance 200, the acceleration information of theaccelerometer 250 of thesecond home appliance 200, or reference data generated using a driving frequency checked from the control command. - The voice data d may correspond to the first voice data obtained by the
microphone 110 of thefirst home appliance 100. - The
processor 130 of thefirst home appliance 100 or theprocessor 330 of theserver 300 may generate the noise data y using the noise estimation algorithm and the reference data r. Specifically, theprocessors - The noise path estimation filter may be implemented as a finite impulse response (FIR) filter or an infinite impulse response (IIR) filter in a time domain. Alternatively, the noise path estimation filter may be implemented in the form of a transfer function predetermined for each frequency band in a frequency domain.
- The noise path estimation filter may have a linear structure as in the example described above, but is not limited thereto and may have a non-linear structure.
- The noise path estimation filter may be fixed and used as one noise path estimation filter, and in case that a plurality of noise path estimation filter information are stored in advance, one of the noise path estimation filters may be selected and used to generate the noise data y according to situations.
- The
processors - The
processor 330 of theserver 300 may perform preprocessing and transmit noise-removed voice data e to thefirst home appliance 100. In addition, theprocessor 130 of thefirst home appliance 100 may perform voice recognition using the voice data e from which noise was removed by performing preprocessing. - Meanwhile, the
processor - Specifically, after performing preprocessing, the
processors FIG. 7 . -
FIG. 7 is a block diagram illustrating a method of removing noise in a frequency domain according to the first embodiment. - Referring to
FIG. 7 , a noise estimation algorithm using a noise path estimation filter H in the frequency domain and performing updating using voice data d including noise may be identified. - The
processors processors processors processors - Meanwhile, after performing the preprocessing, the
processors processor - For example, it may be assumed that a k−1th noise path estimation filter is H(k−1), voice data including subsequent kth noise is d(k), kth reference data is r(k), voice data converted into kth frequency domain is D(k), and reference data converted into the kth frequency domain is R(k). In this case, a new noise path estimation filter H(k)=(H(k)=GRR(k)·GDR(k) (−1)) may be generated by multiplying an inverse matrix of GRR(k) which is an auto correlation value of the kth converted reference data R(k) and GDR(k) which is a cross correlation value of the kth reference data R(k) and the voice data D(k).
- Here, GRR(k) is GRR(k)=(1−λ)·GRR(k−1)+λ·R(k)·R(k) H(R(k) H is a Hermitian matrix of reference data (R(k))) and GDR(k)=(1−λ)·GDR(k−1)+λ·D(k)·R(k) H.
- In addition, as the λ value, a constant determined according to systems may be used, or the λ value may be variably used for stability of the algorithm. For example, when used in a variable manner, the kth λ value may be λ(k)=α·(ef(k)/(1+ef(k)). Also, f(k)=β·sqrt(V(k)−M(k)2)/|M(k)|, M(k)=(1−γ)·M(k−1)+γ·D(k), V(k)=(1−γ)·V(k−1)+γ·D(k)·
D(k) (α, β, and γ are constants). - When the k+1th reference data (r(k+1) and voice data (d(k+1)) are obtained, the
processors - Meanwhile, the method of updating the noise path estimation filter H is not limited to the example described above. For example, in the example described above, updating is performed each time the voice data including noise and the reference data are obtained, but it may be implemented such that updating is performed when voice data including a predetermined number of noise and reference data are obtained. In this case, updating may be performed using the voice data including the predetermined number of noise and the reference data together.
- In
FIGS. 6 and 7 , it is illustrated and described that that theprocessors -
FIGS. 8 through 10 are diagrams illustrating a noise removing method according to a second embodiment. -
FIG. 8 is a simple block diagram illustrating a noise removing method according to a second embodiment. - Referring to
FIG. 8 , it can be seen that noise data y is generated using the reference data r, the noise-removed voice data e, and the noise estimation algorithm, and noise of the voice data d is removed using the generated noise data y. - Here, the reference data r may be at least one of the second voice data obtained through the
microphone 210 of thesecond home appliance 200 and the reference data generated using a driving frequency identified from the control command or the acceleration information of theaccelerometer 250 of thesecond home appliance 200. - The voice data d may correspond to the first voice data obtained from the
microphone 110 of thefirst home appliance 100. - The
processors FIG. 7 , and thus, redundant description thereof will be omitted. - The
processors - The
processor 330 of theserver 300 may perform preprocessing to transmit the noise-removed voice data e to thefirst home appliance 100. In addition, theprocessor 130 of thefirst home appliance 100 may perform voice recognition using the noise-removed voice data e by performing preprocessing. - A difference between the second embodiment of
FIG. 8 and the first embodiment ofFIG. 6 lies in the method of updating the noise path estimation filter. Specifically, in the second embodiment, theprocessors FIGS. 9 and 10 . -
FIG. 9 is a block diagram illustrating a method of removing noise in a time domain according to the second embodiment. - Referring to
FIG. 9 , a noise estimation algorithm using a noise path estimation filter, which is an FIR filter, in the time domain and performing updating using noise-removed voice data may be identified. - The
processors processors - Meanwhile, the
processors processors - For example, it may be assumed that a k−1th noise path estimation filter is h(k−1), kth noise-removed voice data is e(k), and kth reference data is r(k). In this case, the kth reference data r(k) may be stored to the buffer r(k) as much as a length of the noise path estimation filter h(k−1) and a new noise path estimation filter h(k)(h(k)=h(k−1)+μ·e(k)·r(k)) may be generated using the buffer r(k) and the kth noise-removed voice data e(k).
- Here, as μ, a constant determined according to systems may be used or μ may be variably used for stability of the algorithm. For example, when used in a variable manner, μ may be μ(k)=α/P(k). Also, P(k)=(1−γ)·P(k−1)+γ·r(k)T·r(k)) (α and γ are constants).
- When the
processors processors - Meanwhile, the method of updating the noise path estimation filter h in the time domain is not limited to the example described above. For example, in the example described above, updating is performed each time the noise-removed voice data and the reference data are obtained, but it may also be implemented such that updating is performed when the voice data from which a predetermined number of noise was removed and reference data are obtained. In this case, updating may be performed using the voice data from which the predetermined number of noise was removed and the reference data together.
- Meanwhile, the noise path estimation filter h may be a filter implemented in the form of a predetermined transfer function for each frequency band in the frequency domain, rather than an FIR filter in the time domain, and the noise path estimation filter h may be updated using the noise-removed voice signal e. Details thereof will be described later with reference to
FIG. 10 . -
FIG. 10 is a block diagram illustrating a method of removing noise in a frequency domain according to the second embodiment. - Referring to
FIG. 10 , a noise estimation algorithm using a noise path estimation filter in a frequency domain and performing updating using noise-removed voice data may be identified. - The
processors processors processors processor 130 may use the converted noise data y to remove noise of the voice data d. - Meanwhile, the
processors processors - For example, it may be assumed that k−1th noise path estimation filter is H(k−1), kth converted noise-removed voice data is E(k), and kth reference data converted into the frequency domain is R(k). Here, a new noise path estimation filter H(k)(H(k)=H(k+1)+μ·R(k) H·E(k)) may be generated by adding a component obtained by multiplying a Hermitian matrix of the kth reference data R(k) converted into the frequency domain and the kth noise-removed voice data E(k) to the k−1th noise path estimation filter H(k−1).
- Here, as μ, a constant determined according to systems may be used or μ may be variably used for stability of the algorithm. For example, in the case of using μ in a variable manner, the kth μ may be μ(k)=α·(ef(k)/(1+ef(k)). Also, it may be implemented that f(k)=β·sqrt(V(k)−|M(k)|2)/|M(k)|, M(k)=(1−γ)·M(k−1)+γ·D(k), V(k)=(1−γ)·V(k−1)+γ·D(k)·
D(k) (α, β, and γ are constants). - When next reference data r(k+1) and the voice data d(k+1) are obtained, the
processors - Meanwhile, the method of updating the noise path estimation filter H is not limited to the example described above. For example, in the example described above, updating is performed each time the noise-removed voice data and the reference data are obtained, but updating may also be performed when a predetermined number of noise-removed voice data and reference data are obtained. In this case, updating may be performed using the predetermined number of noise-removed voice data and the reference data together.
- Also, in
FIGS. 8 through 10 , it is illustrated and described that theprocessors -
FIGS. 11 through 13 are diagrams illustrating a noise removing method according to a third embodiment. - The information on the noise path estimation filter may be previously stored in the
first home appliance 100 or theserver 300 and used to generate noise data as described above. However, the noise path estimation filter may not be pre-stored in the manufacturing stage of thefirst home appliance 100 but may be generated through an artificial intelligence model after thefirst home appliance 100 is installed in a home. Alternatively, theserver 300 may generate a noise path estimation filter suitable for a home appliance connected through an artificial intelligence model. - In addition, the
first home appliance 100 or theserver 300 may perform noise removal using the generated noise path estimation filter. - In the following description, it is assumed that the
first home appliance 100 obtains a noise path estimation filter from anexternal device 400 including an artificial intelligence model. -
FIG. 11 is a sequence diagram illustrating a method of obtaining a noise path estimation filter by thefirst home appliance 100. - Referring to
FIG. 11 , first, thefirst home appliance 100 may obtain voice data and reference data at a time when there is no user's speech (S1110). Specifically, in order to generate a noise path estimation filter, voice data and reference data in which there is no user's spoken voice and only noise of a noise source are detected are required. Therefore, when voice data determined not to have a user's speech is determined as a result of performing voice recognition, reference data obtained at the same time point as that of corresponding voice data may be checked. - In operation S1120, the
first home appliance 100 may transmit the obtained voice data and reference data to theexternal device 400. Specifically, thefirst home appliance 100 may transmit the obtained voice data and reference data as a signal in a time domain or convert it into a frequency domain and transmit the same to theexternal device 400. - Meanwhile, when the
first home appliance 100 has only voice data at the time when there is no user's speech, theexternal device 400 or thefirst home appliance 100 may request reference data at the corresponding time from thesecond home appliance 200 and obtain the reference data. - In addition, the noise of the noise source of the
second home appliance 200 may be different according to a change in an operation mode of thesecond home appliance 200 or an environment. Therefore, it is needed to generate a noise path estimation filter to be applied to each case. To this end, thefirst home appliance 100 may transmit information on an operation mode of thesecond home appliance 200 or information on a surrounding environment together, when transmitting information to theexternal device 400. - For example, when the
second home appliance 200 is a washing machine, a rotation speed of the motor included in the washing machine may vary according to an operation mode. Therefore, a magnitude or characteristics of noise of the noise source (motor) may vary according to the operation mode. - Therefore, the information on each operation mode of the
second home appliance 200 and the voice data and the reference data obtained when thesecond home appliance 200 operates in each operation mode may be transmitted together, so that theexternal device 400 may generate a noise path estimation filter applicable to each operation mode. - The
external device 400 may calculate a noise path estimation filter using the received voice data and reference data (S1130). Specifically, theexternal device 400 may obtain a noise path estimation filter using an artificial intelligence model that receives voice data and reference data and outputs a noise path estimation filter corresponding thereto. The artificial intelligence model may be a linear regression model. - The
external device 400 may transmit the calculated noise path estimation filter information to the first home appliance 100 (S1140). Meanwhile, when thesecond home appliance 200 is implemented in such a manner as to obtain second voice data and generate noise data using the obtained second voice data, the noise path estimation filter may be transmitted to thesecond home appliance 200. - The
first home appliance 100 may store information of the received noise path estimation filter (S1150). - In addition, the
first home appliance 100 may generate noise data using the obtained reference data and noise path estimation filter information, and remove noise of the voice data including user's spoken voice using the generated noise data. Thefirst home appliance 100 may perform the voice recognition function using the noise-removed voice data. - Meanwhile, in
FIG. 11 , it is illustrated and described that the external device is a device other than the voice recognition system but it may be implemented as a server in the voice recognition system. - In addition, in
FIG. 11 , it is illustrated and described that the voice data and the reference data are transmitted to the external device and the noise path estimation filter generated by an artificial intelligence mode is received from the external device, but the first home appliance may generate a noise path estimation filter using the previously stored artificial intelligence model when implemented. - Alternatively, a separate device in the home appliance distinguished from the processor may generate a noise path estimation filter using the pre-stored artificial intelligence model, and the processor may use the generated noise path estimation filter. Meanwhile, the method of generating the noise path estimation filter is not limited to the example described above.
- Hereinafter, a specific operation of removing noise using a noise path estimation filter generated using an artificial intelligence model will be described.
-
FIG. 12 is a block diagram illustrating a method of removing noise in a time domain according to the third embodiment. - Referring to
FIG. 12 , a noise removing method using a noise path estimation filter h in a time domain may be identified. - The
processor 130 of thefirst home appliance 100 may remove noise using the noise path estimation filter h received from theexternal device 400. Alternatively, when theserver 300 generates the noise path estimation filter using the artificial intelligence model, theprocessor 330 of theserver 300 may perform noise removal using the generated noise path estimation filter h. - Specifically, the
processors processors -
FIG. 13 is a block diagram illustrating a method of removing noise in a frequency domain according to the third embodiment. - Referring to
FIG. 13 , a noise removing method using a noise path estimation filter H in a frequency domain may be identified. - The
processors processors - In addition, the
processors processors - In
FIGS. 12 and 13 , it is illustrated and described that updating is not performed on the noise path estimation filter, but updating may be performed on the noise path estimation filter according to the updating method described above when implemented. -
FIGS. 14 through 16 are sequence diagrams illustrating a voice recognition method performed without intervention of a server. -
FIG. 14 is a sequence diagram illustrating a voice recognition method performed without intervention of a server according to the first embodiment of the disclosure. - Referring to
FIG. 14 , first, thefirst home appliance 100 may transmit a recording request signal to the second home appliance 200 (S1410). In this case, the recording request signal may include information on a predetermined time and information on a predetermined time length. - The
first home appliance 100 may obtain first voice data corresponding to the recording request signal (S1420). Specifically, thefirst home appliance 100 may obtain first voice data based on predetermined time information and predetermined time length information included in the recording request signal. - The
second home appliance 200 may obtain second voice data corresponding to the recording request signal (S1425). Specifically, thesecond home appliance 200 may obtain second voice data based on the predetermined time information and the predetermined time length information included in the recording request signal. - The
second home appliance 200 may transmit the obtained second voice data to the first home appliance 100 (S1430). - The
first home appliance 100 may generate noise data using the received second voice data (S1440). Specifically, thefirst home appliance 100 may generate noise data which is sound data corresponding to pure noise of a noise source in thesecond home appliance 200 by applying the received second voice data to the noise path estimation filter. - The
first home appliance 100 may perform preprocessing on the first voice data using the noise data (S1450). - The
first home appliance 100 may update the noise path estimation filter (S1460). Specifically, thefirst home appliance 100 may update the noise path estimation filter using the second voice data and the first voice data or may update the noise path estimation filter using the second voice data and the preprocessed first voice data. Meanwhile, the method of updating the noise path estimation filter has been described above, and thus, a redundant description thereof will be omitted. - The
first home appliance 100 may perform voice recognition using the preprocessed first voice data (S1470). -
FIG. 15 is a sequence diagram illustrating a voice recognition method performed without intervention of a server according to the second embodiment of the disclosure. - Referring to
FIG. 15 , first, thefirst home appliance 100 may transmit a recording request signal to the second home appliance 200 (S1510). - The
first home appliance 100 may obtain first voice data corresponding to the recording request signal (S1520). - The
second home appliance 200 may obtain second voice data corresponding to the recording request signal (S1525). - The
second home appliance 200 may transmit the obtained second voice data to the first home appliance 100 (S1530). - The
first home appliance 100 may update the noise path estimation filter before performing the preprocessing on the first voice data (S1540). - For example, as illustrated in
FIG. 7 , when the noise path estimation filter is determined by GRR, which is an auto correlation value of reference data, and GDR, which is a cross correlation value of reference data and voice data (H(k)=GRR(k)·GDR(k) (−1)), the noise path estimation filter may be updated by updating GRR and GDR. - As another example, as shown in
FIG. 9 , when the noise path estimation filter is determined by the buffer of the reference data r and the noise-removed voice data (h(k)=h(k−1)+μ(k)·e(k)·r(k), e(k)=h(k−1)*d(k), μ(k)=α/P(k), P(k)=(1−γ)·P(k−1)+γ·r(k)T·r(k)), the noise path estimation filter may be updated by updating the reference data r of the buffer and P(k). - In another example, when the noise path estimation filter is determined by the Hermitian matrix of reference data and the noise-removed voice data (H(k)=H(k−1)+μ·R(k) H·E(k), E(k)=D(k)−R(k)·H(k−1)) as shown in
FIG. 10 , the noise path estimation filter may be updated by updating the Hermitian matrix of the reference data. - The
first home appliance 100 may generate noise data using the received second voice data (S1550). Specifically, thefirst home appliance 100 may generate noise data which is sound data corresponding to pure noise of a noise source in thesecond home appliance 200 by applying the received second voice data to the updated noise path estimation filter. - The
first home appliance 100 may perform preprocessing on the first voice data using the noise data (S1560). - The
first home appliance 100 may perform voice recognition using the preprocessed first voice data (S1570). -
FIG. 16 is a sequence diagram illustrating a voice recognition method performed without intervention of a server according to a third embodiment of the disclosure. - Referring to
FIG. 16 , first, thefirst home appliance 100 may transmit a recording request signal to the second home appliance 200 (S1610). - In operation S1620, the
first home appliance 100 may obtain first voice data corresponding to the recording request signal. Specifically, thefirst home appliance 100 may obtain first voice data based on predetermined time information and predetermined time length information included in the recording request signal. - The
second home appliance 200 may obtain second voice data corresponding to the recording request signal (S1625). - The
second home appliance 200 may generate noise data using the second voice data (S1630). Specifically, thesecond home appliance 200 may generate noise data which is sound data corresponding to pure noise of the noise source in thesecond home appliance 200 by applying the second voice data to the noise path estimation filter. The noise path estimation filter may be a filter previously stored in thesecond home appliance 200 or generated by an artificial intelligence model of theserver 300 or theexternal device 400 and transmitted to thesecond home appliance 200. - The
second home appliance 200 may transmit the generated noise data to the first home appliance 100 (S1640). - The
first home appliance 100 may perform preprocessing on the first voice data using the received noise data (S1650). - The
first home appliance 100 may perform voice recognition using the preprocessed first voice data (S1660). - As described above, the
first home appliance 100 may estimate noise using information on the noise produced in the other home appliance without intervention of the server in various manners, and reduce the estimated noise. -
FIGS. 17 and 18 are sequence diagrams illustrating a voice recognition method performed according to intervention of a server. -
FIG. 17 is a sequence diagram illustrating a voice recognition method performed according to intervention of a server according to the first embodiment of the disclosure. - Referring to
FIG. 17 , first, theserver 300 may transmit a recording request signal to the first home appliance 100 (S1710). Theserver 300 may transmit a recording request signal to the second home appliance 200 (S1715). In this case, the recording request signal may include information on a predetermined time and information on a predetermined time length. - The
first home appliance 100 may obtain first voice data corresponding to the recording request signal (S1720). Specifically, thefirst home appliance 100 may obtain first voice data based on the predetermined time information and the predetermined time length information included in the recording request signal. - The
second home appliance 200 may obtain second voice data corresponding to the recording request signal (S1725). Specifically, thesecond home appliance 200 may obtain second voice data based on the predetermined time information and the predetermined time length information included in the recording request signal. - The
first home appliance 100 may transmit the obtained first voice data to the server 300 (S1730). Thesecond home appliance 200 may transmit the obtained second voice data to the server 300 (S1735). - The
server 300 may generate noise data using the received second voice data (S1740). Specifically, theserver 300 may generate noise data which is sound data corresponding to pure noise of the noise source in thesecond home appliance 200 by applying the received second voice data to the noise path estimation filter. - The
server 300 may perform preprocessing on the first voice data using the noise data (S1750). Theserver 300 may transmit the preprocessed first voice data to the first home appliance 100 (S1760). - The
server 300 may update the noise path estimation filter (S1770). Specifically, theserver 300 may update the noise path estimation filter using the second voice data and the first voice data or update the noise path estimation filter using the second voice data and the preprocessed first voice data. Meanwhile, the method of updating the noise path estimation filter has been described above, and thus, a redundant description thereof will be omitted. - The
first home appliance 100 may perform voice recognition using the preprocessed first voice data (S1780). -
FIG. 18 is a sequence diagram illustrating a voice recognition method performed under intervention of a server according to the second embodiment of the disclosure. - Referring to
FIG. 18 , first, theserver 300 may transmit a recording request signal to the first home appliance 100 (S1810). Theserver 300 may transmit a recording request signal to the second home appliance 200 (S1815). - The
first home appliance 100 may obtain first voice data corresponding to the recording request signal (S1820). Thesecond home appliance 200 may obtain second voice data based on the recording request signal (S1825). - The
first home appliance 100 may transmit the obtained first voice data to the server 300 (S1830). Thesecond home appliance 200 may transmit the obtained second voice data to the server 300 (S1835). - The
server 300 may update the noise path estimation filter before performing the preprocessing on the first voice data (S1840). The operation of updating the noise path estimation filter before performing the preprocessing has been described above with reference toFIG. 15 , and thus, a redundant description thereof will be omitted. - The
server 300 may generate noise data using the received second voice data (S1850). Specifically, theserver 300 may generate noise data corresponding to pure noise of the noise source in thesecond home appliance 200 by applying the received second voice data to the updated noise path estimation filter. - The
server 300 may perform preprocessing on the first voice data using the noise data (S1860). Theserver 300 may transmit the preprocessed first voice data to the first home appliance 100 (S1870). - The
first home appliance 100 may perform voice recognition using the preprocessed first voice data (S1880). - As described above, the server may estimate the noise using the information on the noise produced in the second home appliance in various manners and reduce the estimated noise from the voice data of the first home appliance.
- In
FIGS. 17 and 18 , it is illustrated and described that theserver 300 is connected to two home appliances but the server may be connected to two or more home appliances and estimate noise using information on the noise produced in the two or more home appliances when implemented. -
FIG. 19 is a flowchart illustrating a voice recognition method of a first home appliance according to an embodiment of the disclosure. - Referring to
FIG. 19 , first, first voice data is obtained through a microphone of a first home appliance (S1910). - In addition, other home appliances that may be connected to the first home appliance may be detected. Specifically, a second home appliance which may be connected to the first home appliance through short-range wireless communication or a second home appliance connected to an access point AP to which the first home appliance is connected may be detected.
- The first home appliance may request transmission of voice data from the second home appliance. Specifically, a recording request signal for requesting generation and transmission of second voice data, which is voice data regarding noise produced in the second home appliance, may be transmitted to the second home appliance.
- Here, the recording request signal may include information on a predetermined time or information on a predetermined time length. Meanwhile, the information that may be included in the recording request signal is not limited to the example described above.
- Second voice data is received from the second home appliance (S1920). Specifically, the second voice data generated based on the predetermined time information and the predetermined time length information included in the recording request signal may be received from the second home appliance.
- First voice data may be obtained based on the request transmitted to the second home appliance. Specifically, first voice data may be obtained through a microphone of the first home appliance based on the predetermined time information and the predetermined time length information included in the recording request signal transmitted to the second home appliance.
- Noise data is generated using the received second voice data (S1930). Here, the noise data refers to sound data corresponding to noise of the noise source.
- The noise data may be obtained by extracting only a component corresponding to the noise of the noise source from the second voice data or filtering remaining components other than the component corresponding to the noise of the noise source.
- The noise data may be generated by filtering the remaining components other than the component corresponding to the noise source included in the second voice data using at least one of a finite impulse response (FIR) filter or an infinite impulse response (IIR) filter in the time domain.
- Alternatively, the noise data may extract a component corresponding to the noise of the noise source from the second voice data using a transfer function previously determined for each frequency band on the frequency domain.
- The information on the above-described filter or transfer function may be pre-stored in a home appliance at a manufacturing stage and used to generate noise data, but is not limited thereto.
- For example, when communication with an external device is possible, information on a filter or a transfer function may be received through communication with the external device, and noise data may be generated using the information on the received filter or transfer function.
- In this case, the information on the filter or the transfer function received from the external device may be information obtained using an artificial intelligence model included in the external device.
- Signal processing is performed on the first voice data using the generated noise data (S1940). Specifically, signal processing for removing a component corresponding to noise of a noise source included in the first voice data may be performed using the noise data.
- Meanwhile, the noise source of the second home appliance may be a motor disposed in a housing of the second home appliance to perform a predetermined function of the second home appliance. In this case, the second voice data may be voice data obtained from a microphone of the second home appliance. The noise data may be motor noise data of the motor of the second home appliance, and preprocessing on the first voice data may be performed by removing the motor noise from the first voice data.
- A signal processing method for the first voice data may be changed according to whether noise is produced in the second home appliance.
- For example, it is assumed that a sound produced in the second home appliance is produced by the motor included in the second home appliance. In this case, the preprocessing method regarding the first voice data may be changed depending on whether the motor of the second home appliance is driven.
- Specifically, it may be determined whether the motor of the second home appliance is driven. If the motor of the second home appliance is driven, noise produced in the second home appliance is large in magnitude, and thus preprocessing on the first voice data may be performed using the second voice data according to the method described above.
- Meanwhile, when the motor of the second home appliance is not driven, there is no noise produced in the second home appliance itself, and thus, voice recognition may be performed directly using the first voice data without using the second voice data or preprocessing may be performed on the first voice data according to the related art method.
- Meanwhile, even when the noise source of the second home appliance is a fan or the like, rather than the motor, the method of preprocessing on the first voice data may be changed according to whether the noise source is driven.
- Voice recognition is performed using the signal-processed first voice data (S1950).
- In addition, the method of generating noise data may be updated using at least one of the first voice data including noise, the signal-processed first voice data, or the second voice data to generate accurate noise data even when noise of the noise source is changed or a surrounding environment is changed.
- Specifically, the method of generating noise data may be updated using a correlation between at least one of the first voice data and the signal-processed first voice data and the second voice data.
- Meanwhile, the operation of updating the method of generating noise data may be performed after signal processing is performed on the first voice data as described above, but alternatively, the operation of updating the method of generating noise data may be first performed before signal processing is performed on the first voice data.
- Accordingly, in the voice recognition method of the home appliance of the disclosure, the voice data of the noise is obtained from another home appliance which produces large magnitude noise and the preprocessing is performed on the obtained noise using the voice data, whereby the noise included in the voice data obtained through the microphone may be accurately removed even though a magnitude of the noise produced from outside is large. The voice recognition method as shown in
FIG. 19 may also be executed on a home appliance having the configuration ofFIG. 2 or 3 or may be executed on a home appliance having another configuration. - In addition, the voice recognition method as described above may be implemented by at least one executable program for executing the voice recognition method as described above, and such an executable program may be stored in a non-transitory readable medium.
- The non-transitory readable medium refers to a medium which stores data semi-permanently and is readable by a device, not a medium storing data for a short time such as a register, a cache, a memory, and the like. Specifically, various applications or programs may be stored and provided in a non-transitory readable medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.
-
FIG. 20 is a flowchart illustrating a voice recognition method of a second home appliance according to an embodiment of the disclosure. - Referring to
FIG. 20 , first, a voice data request signal may be received (S2010). Specifically, a recording request signal requesting generation and transmission of second voice data which is voice data of the noise produced in the second home appliance may be received from a server or a first home appliance connected to the second home appliance. - Second voice data may be obtained through a microphone of the second home appliance (S2020). Specifically, noise may be detected using the microphone of the second home appliance and the second voice data may be obtained based on the received recording request signal. More specifically, the second voice data may be obtained based on information on a predetermined time and predetermined time length information included in the received recording request signal.
- The obtained second voice data may be transmitted (S2030). Specifically, the obtained second voice data may be transmitted to the server or the first home appliance connected to the second home appliance.
- Meanwhile, the second voice data may not be transmitted to the server or the first home appliance connected to the second home appliance, and noise data may be generated using the second voice data and the generated noise data may be transmitted to the first home appliance or the server.
- Thus, in the voice recognition method of the home appliance according to the disclosure, because the reference data regarding the noise of the internal noise source is generated and provided to the home appliance or server which is to perform voice recognition function, the home appliance or the server may accurately remove the noise included in the voice data including user's speech. The voice recognition method as shown in
FIG. 20 may be executed on a home appliance having the configuration ofFIG. 4 or may be executed on a home appliance having another configuration. - In addition, the voice recognition method as described above may be implemented by at least one executable program for executing the voice recognition method as described above, and such an executable program may be stored in a non-transitory readable medium.
-
FIG. 21 is a flowchart illustrating a voice recognition method of a server according to an embodiment of the disclosure. - Referring to
FIG. 21 , first, voice data may be requested from a first home appliance and a second home appliance (S2110). Specifically, a recording request signal for requesting generation and transmission of voice data may be transmitted to the first home appliance and the second home appliance. - In addition, the recording request signal may include information on a predetermined time and information on a predetermined time length. The information included in the recording request signal is not limited to the examples described above.
- First voice data and second voice data may be received (S2120). Specifically, the first voice data obtained based on the recording request signal may be received from the first home appliance. The second voice data obtained based on the recording request signal may be received from the second home appliance.
- Noise data may be generated using the second voice data (S2130). Here, the noise data refers to sound data corresponding to pure noise of a noise source in the second home appliance.
- Meanwhile, a method of generating noise data has been described above, and thus, a redundant description thereof will be omitted.
- Signal processing may be performed on the first voice data using the noise data (S2140). Specifically, signal processing for removing a component corresponding to the noise of the noise source included in the first voice data may be performed using the noise data.
- The signal-processed first voice data may be transmitted to the first home appliance (S2150).
- Therefore, in the voice recognition method of the server of the disclosure, the first voice data is received from the first home appliance which is to perform the voice recognition function, the second voice data regarding the noise or reference data is obtained from the second home appliance that produces large magnitude noise, and preprocessing is performed on the first voice data using the obtained second voice data or reference data, so that the noise included in the voice data may be accurately removed even if the noise is large in magnitude. The voice recognition method as shown in
FIG. 21 may be executed on a server having the configuration ofFIG. 5 or may be executed on a server having another configuration. - In addition, the voice recognition method as described above may be implemented with at least one executable program for executing the voice recognition method as described above, and such an executable program may be stored in a non-transitory readable medium.
- While the disclosure has been described with reference to the accompanying drawings, it is to be understood that the scope of the disclosure is defined by the claims described hereinafter and should not be construed as being limited to the above-described embodiments and/or drawings. It is to be clearly understood that improvements, changes, and modifications that are obvious to those skilled in the art are also within the scope of the disclosure as defined in the claims.
- Although the present disclosure has been described with various embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims.
Claims (20)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR10-2018-0170469 | 2018-12-27 | ||
KR1020180170469A KR102570384B1 (en) | 2018-12-27 | 2018-12-27 | Home appliance and method for voice recognition thereof |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200211539A1 true US20200211539A1 (en) | 2020-07-02 |
US11404054B2 US11404054B2 (en) | 2022-08-02 |
Family
ID=71121828
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/726,031 Active 2040-05-25 US11404054B2 (en) | 2018-12-27 | 2019-12-23 | Home appliance and method for voice recognition thereof |
Country Status (6)
Country | Link |
---|---|
US (1) | US11404054B2 (en) |
EP (1) | EP3844750A4 (en) |
KR (1) | KR102570384B1 (en) |
CN (1) | CN113287168A (en) |
AU (1) | AU2019416316A1 (en) |
WO (1) | WO2020138844A1 (en) |
Cited By (56)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112164397A (en) * | 2020-09-30 | 2021-01-01 | 广东美的厨房电器制造有限公司 | Noise reduction control method, cooking appliance, and readable storage medium |
CN112712683A (en) * | 2020-12-14 | 2021-04-27 | 珠海格力电器股份有限公司 | Control method and system of household appliance, remote controller and server |
US11200894B2 (en) | 2019-06-12 | 2021-12-14 | Sonos, Inc. | Network microphone device with command keyword eventing |
US11212612B2 (en) | 2016-02-22 | 2021-12-28 | Sonos, Inc. | Voice control of a media playback system |
US11288039B2 (en) | 2017-09-29 | 2022-03-29 | Sonos, Inc. | Media playback system with concurrent voice assistance |
US11308962B2 (en) * | 2020-05-20 | 2022-04-19 | Sonos, Inc. | Input detection windowing |
US11308959B2 (en) | 2020-02-11 | 2022-04-19 | Spotify Ab | Dynamic adjustment of wake word acceptance tolerance thresholds in voice-controlled devices |
US11308958B2 (en) | 2020-02-07 | 2022-04-19 | Sonos, Inc. | Localized wakeword verification |
US11315556B2 (en) | 2019-02-08 | 2022-04-26 | Sonos, Inc. | Devices, systems, and methods for distributed voice processing by transmitting sound data associated with a wake word to an appropriate device for identification |
US11315553B2 (en) * | 2018-09-20 | 2022-04-26 | Samsung Electronics Co., Ltd. | Electronic device and method for providing or obtaining data for training thereof |
US11330335B1 (en) * | 2017-09-21 | 2022-05-10 | Amazon Technologies, Inc. | Presentation and management of audio and visual content across devices |
US11328722B2 (en) * | 2020-02-11 | 2022-05-10 | Spotify Ab | Systems and methods for generating a singular voice audio stream |
US11343614B2 (en) | 2018-01-31 | 2022-05-24 | Sonos, Inc. | Device designation of playback and network microphone device arrangements |
US11361756B2 (en) | 2019-06-12 | 2022-06-14 | Sonos, Inc. | Conditional wake word eventing based on environment |
US11405430B2 (en) | 2016-02-22 | 2022-08-02 | Sonos, Inc. | Networked microphone device control |
US11432030B2 (en) | 2018-09-14 | 2022-08-30 | Sonos, Inc. | Networked devices, systems, and methods for associating playback devices based on sound codes |
US11482978B2 (en) | 2018-08-28 | 2022-10-25 | Sonos, Inc. | Audio notifications |
US11482224B2 (en) | 2020-05-20 | 2022-10-25 | Sonos, Inc. | Command keywords with input detection windowing |
US11501773B2 (en) | 2019-06-12 | 2022-11-15 | Sonos, Inc. | Network microphone device with command keyword conditioning |
US11500611B2 (en) | 2017-09-08 | 2022-11-15 | Sonos, Inc. | Dynamic computation of system response volume |
US11513763B2 (en) | 2016-02-22 | 2022-11-29 | Sonos, Inc. | Audio response playback |
US11514898B2 (en) | 2016-02-22 | 2022-11-29 | Sonos, Inc. | Voice control of a media playback system |
US11531520B2 (en) | 2016-08-05 | 2022-12-20 | Sonos, Inc. | Playback device supporting concurrent voice assistants |
US11538451B2 (en) | 2017-09-28 | 2022-12-27 | Sonos, Inc. | Multi-channel acoustic echo cancellation |
US11538460B2 (en) | 2018-12-13 | 2022-12-27 | Sonos, Inc. | Networked microphone devices, systems, and methods of localized arbitration |
US11540047B2 (en) | 2018-12-20 | 2022-12-27 | Sonos, Inc. | Optimization of network microphone devices using noise classification |
US11545169B2 (en) | 2016-06-09 | 2023-01-03 | Sonos, Inc. | Dynamic player selection for audio signal processing |
US11551669B2 (en) | 2019-07-31 | 2023-01-10 | Sonos, Inc. | Locally distributed keyword detection |
US11551678B2 (en) | 2019-08-30 | 2023-01-10 | Spotify Ab | Systems and methods for generating a cleaned version of ambient sound |
US11556306B2 (en) | 2016-02-22 | 2023-01-17 | Sonos, Inc. | Voice controlled media playback system |
US11557294B2 (en) | 2018-12-07 | 2023-01-17 | Sonos, Inc. | Systems and methods of operating media playback systems having multiple voice assistant services |
US11563842B2 (en) | 2018-08-28 | 2023-01-24 | Sonos, Inc. | Do not disturb feature for audio notifications |
US11562740B2 (en) | 2020-01-07 | 2023-01-24 | Sonos, Inc. | Voice verification for media playback |
US11641559B2 (en) | 2016-09-27 | 2023-05-02 | Sonos, Inc. | Audio playback settings for voice interaction |
US11646023B2 (en) | 2019-02-08 | 2023-05-09 | Sonos, Inc. | Devices, systems, and methods for distributed voice processing |
US11646045B2 (en) | 2017-09-27 | 2023-05-09 | Sonos, Inc. | Robust short-time fourier transform acoustic echo cancellation during audio playback |
US11696074B2 (en) | 2018-06-28 | 2023-07-04 | Sonos, Inc. | Systems and methods for associating playback devices with voice assistant services |
US11698771B2 (en) | 2020-08-25 | 2023-07-11 | Sonos, Inc. | Vocal guidance engines for playback devices |
US11710487B2 (en) | 2019-07-31 | 2023-07-25 | Sonos, Inc. | Locally distributed keyword detection |
US11714600B2 (en) | 2019-07-31 | 2023-08-01 | Sonos, Inc. | Noise classification for event detection |
US11727919B2 (en) | 2020-05-20 | 2023-08-15 | Sonos, Inc. | Memory allocation for keyword spotting engines |
US11726742B2 (en) | 2016-02-22 | 2023-08-15 | Sonos, Inc. | Handling of loss of pairing between networked devices |
US11727933B2 (en) | 2016-10-19 | 2023-08-15 | Sonos, Inc. | Arbitration-based voice recognition |
US11741948B2 (en) | 2018-11-15 | 2023-08-29 | Sonos Vox France Sas | Dilated convolutions and gating for efficient keyword spotting |
US20230290346A1 (en) * | 2018-03-23 | 2023-09-14 | Amazon Technologies, Inc. | Content output management based on speech quality |
US11769505B2 (en) | 2017-09-28 | 2023-09-26 | Sonos, Inc. | Echo of tone interferance cancellation using two acoustic echo cancellers |
US11790937B2 (en) | 2018-09-21 | 2023-10-17 | Sonos, Inc. | Voice detection optimization using sound metadata |
US11792590B2 (en) | 2018-05-25 | 2023-10-17 | Sonos, Inc. | Determining and adapting to changes in microphone performance of playback devices |
US11790911B2 (en) | 2018-09-28 | 2023-10-17 | Sonos, Inc. | Systems and methods for selective wake word detection using neural network models |
US11798553B2 (en) | 2019-05-03 | 2023-10-24 | Sonos, Inc. | Voice assistant persistence across multiple network microphone devices |
US11797263B2 (en) | 2018-05-10 | 2023-10-24 | Sonos, Inc. | Systems and methods for voice-assisted media content selection |
US11822601B2 (en) | 2019-03-15 | 2023-11-21 | Spotify Ab | Ensemble-based data comparison |
US11862161B2 (en) | 2019-10-22 | 2024-01-02 | Sonos, Inc. | VAS toggle based on device orientation |
US11869503B2 (en) | 2019-12-20 | 2024-01-09 | Sonos, Inc. | Offline voice control |
US11899519B2 (en) | 2018-10-23 | 2024-02-13 | Sonos, Inc. | Multiple stage network microphone device with reduced power consumption and processing load |
US11900937B2 (en) | 2017-08-07 | 2024-02-13 | Sonos, Inc. | Wake-word detection suppression |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20220069611A (en) * | 2020-11-20 | 2022-05-27 | 삼성전자주식회사 | Electronic apparatus and control method thereof |
Family Cites Families (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2839815B2 (en) * | 1993-02-26 | 1998-12-16 | 松下電器産業株式会社 | Sound pickup device for video camera |
US7617099B2 (en) * | 2001-02-12 | 2009-11-10 | FortMedia Inc. | Noise suppression by two-channel tandem spectrum modification for speech signal in an automobile |
KR100413797B1 (en) | 2001-08-23 | 2003-12-31 | 삼성전자주식회사 | Speech signal compensation method and the apparatus thereof |
US6742208B2 (en) * | 2001-08-24 | 2004-06-01 | Maytag Corporation | Clothes washing machine incorporating noise reduction system |
KR100446626B1 (en) | 2002-03-28 | 2004-09-04 | 삼성전자주식회사 | Noise suppression method and apparatus |
KR100474340B1 (en) | 2002-08-02 | 2005-03-10 | 엘지전자 주식회사 | Voice controlled vacuum cleaner and method thereof |
US8345890B2 (en) * | 2006-01-05 | 2013-01-01 | Audience, Inc. | System and method for utilizing inter-microphone level differences for speech enhancement |
KR100913197B1 (en) | 2008-08-06 | 2009-08-24 | 주식회사 현대오토넷 | Apparatus and method of noise removal |
JP2010217453A (en) | 2009-03-16 | 2010-09-30 | Fujitsu Ltd | Microphone system for voice recognition |
EP2652737B1 (en) * | 2010-12-15 | 2014-06-04 | Koninklijke Philips N.V. | Noise reduction system with remote noise detector |
KR101832952B1 (en) | 2011-04-07 | 2018-02-28 | 엘지전자 주식회사 | Robot cleaner and controlling method of the same |
CN102348151B (en) | 2011-09-10 | 2015-07-29 | 歌尔声学股份有限公司 | Noise canceling system and method, intelligent control method and device, communication equipment |
JP6466844B2 (en) * | 2013-08-29 | 2019-02-06 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Device control method and device control system |
CN103632666B (en) * | 2013-11-14 | 2016-09-28 | 华为技术有限公司 | Audio recognition method, speech recognition apparatus and electronic equipment |
KR102188090B1 (en) * | 2013-12-11 | 2020-12-04 | 엘지전자 주식회사 | A smart home appliance, a method for operating the same and a system for voice recognition using the same |
GB2542058B (en) * | 2014-06-04 | 2021-09-08 | Cirrus Logic Int Semiconductor Ltd | Reducing instantaneous wind noise |
JP2016042132A (en) | 2014-08-18 | 2016-03-31 | ソニー株式会社 | Voice processing device, voice processing method, and program |
DE112015003945T5 (en) * | 2014-08-28 | 2017-05-11 | Knowles Electronics, Llc | Multi-source noise reduction |
US10755726B2 (en) * | 2015-01-07 | 2020-08-25 | Google Llc | Detection and suppression of keyboard transient noise in audio streams with auxiliary keybed microphone |
US10178474B2 (en) * | 2015-04-21 | 2019-01-08 | Google Llc | Sound signature database for initialization of noise reduction in recordings |
US10013996B2 (en) * | 2015-09-18 | 2018-07-03 | Qualcomm Incorporated | Collaborative audio processing |
JP6927661B2 (en) | 2015-09-30 | 2021-09-01 | 東芝ライフスタイル株式会社 | Vacuum cleaner |
US9922637B2 (en) * | 2016-07-11 | 2018-03-20 | Microsoft Technology Licensing, Llc | Microphone noise suppression for computing device |
US10283138B2 (en) * | 2016-10-03 | 2019-05-07 | Google Llc | Noise mitigation for a voice interface device |
WO2018147687A1 (en) * | 2017-02-10 | 2018-08-16 | Samsung Electronics Co., Ltd. | Method and apparatus for managing voice-based interaction in internet of things network system |
KR101898911B1 (en) | 2017-02-13 | 2018-10-31 | 주식회사 오르페오사운드웍스 | Noise cancelling method based on sound reception characteristic of in-mic and out-mic of earset, and noise cancelling earset thereof |
KR20180127100A (en) * | 2017-05-19 | 2018-11-28 | 엘지전자 주식회사 | Method for operating home appliance and voice recognition server system |
US10403299B2 (en) * | 2017-06-02 | 2019-09-03 | Apple Inc. | Multi-channel speech signal enhancement for robust voice trigger detection and automatic speech recognition |
US10468020B2 (en) * | 2017-06-06 | 2019-11-05 | Cypress Semiconductor Corporation | Systems and methods for removing interference for audio pattern recognition |
CN207367598U (en) * | 2017-09-04 | 2018-05-15 | 湖南海翼电子商务股份有限公司 | Vehicle electronic device |
US10602268B1 (en) * | 2018-12-20 | 2020-03-24 | Sonos, Inc. | Optimization of network microphone devices using noise classification |
-
2018
- 2018-12-27 KR KR1020180170469A patent/KR102570384B1/en active IP Right Grant
-
2019
- 2019-12-19 WO PCT/KR2019/018113 patent/WO2020138844A1/en unknown
- 2019-12-19 AU AU2019416316A patent/AU2019416316A1/en active Pending
- 2019-12-19 EP EP19904108.8A patent/EP3844750A4/en active Pending
- 2019-12-19 CN CN201980086395.2A patent/CN113287168A/en active Pending
- 2019-12-23 US US16/726,031 patent/US11404054B2/en active Active
Cited By (71)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11832068B2 (en) | 2016-02-22 | 2023-11-28 | Sonos, Inc. | Music service selection |
US11556306B2 (en) | 2016-02-22 | 2023-01-17 | Sonos, Inc. | Voice controlled media playback system |
US11405430B2 (en) | 2016-02-22 | 2022-08-02 | Sonos, Inc. | Networked microphone device control |
US11212612B2 (en) | 2016-02-22 | 2021-12-28 | Sonos, Inc. | Voice control of a media playback system |
US11736860B2 (en) | 2016-02-22 | 2023-08-22 | Sonos, Inc. | Voice control of a media playback system |
US11863593B2 (en) | 2016-02-22 | 2024-01-02 | Sonos, Inc. | Networked microphone device control |
US11726742B2 (en) | 2016-02-22 | 2023-08-15 | Sonos, Inc. | Handling of loss of pairing between networked devices |
US11514898B2 (en) | 2016-02-22 | 2022-11-29 | Sonos, Inc. | Voice control of a media playback system |
US11750969B2 (en) | 2016-02-22 | 2023-09-05 | Sonos, Inc. | Default playback device designation |
US11513763B2 (en) | 2016-02-22 | 2022-11-29 | Sonos, Inc. | Audio response playback |
US11545169B2 (en) | 2016-06-09 | 2023-01-03 | Sonos, Inc. | Dynamic player selection for audio signal processing |
US11531520B2 (en) | 2016-08-05 | 2022-12-20 | Sonos, Inc. | Playback device supporting concurrent voice assistants |
US11641559B2 (en) | 2016-09-27 | 2023-05-02 | Sonos, Inc. | Audio playback settings for voice interaction |
US11727933B2 (en) | 2016-10-19 | 2023-08-15 | Sonos, Inc. | Arbitration-based voice recognition |
US11900937B2 (en) | 2017-08-07 | 2024-02-13 | Sonos, Inc. | Wake-word detection suppression |
US11500611B2 (en) | 2017-09-08 | 2022-11-15 | Sonos, Inc. | Dynamic computation of system response volume |
US11330335B1 (en) * | 2017-09-21 | 2022-05-10 | Amazon Technologies, Inc. | Presentation and management of audio and visual content across devices |
US11758232B2 (en) * | 2017-09-21 | 2023-09-12 | Amazon Technologies, Inc. | Presentation and management of audio and visual content across devices |
US20220303630A1 (en) * | 2017-09-21 | 2022-09-22 | Amazon Technologies, Inc. | Presentation and management of audio and visual content across devices |
US11646045B2 (en) | 2017-09-27 | 2023-05-09 | Sonos, Inc. | Robust short-time fourier transform acoustic echo cancellation during audio playback |
US11538451B2 (en) | 2017-09-28 | 2022-12-27 | Sonos, Inc. | Multi-channel acoustic echo cancellation |
US11769505B2 (en) | 2017-09-28 | 2023-09-26 | Sonos, Inc. | Echo of tone interferance cancellation using two acoustic echo cancellers |
US11288039B2 (en) | 2017-09-29 | 2022-03-29 | Sonos, Inc. | Media playback system with concurrent voice assistance |
US11893308B2 (en) | 2017-09-29 | 2024-02-06 | Sonos, Inc. | Media playback system with concurrent voice assistance |
US11689858B2 (en) | 2018-01-31 | 2023-06-27 | Sonos, Inc. | Device designation of playback and network microphone device arrangements |
US11343614B2 (en) | 2018-01-31 | 2022-05-24 | Sonos, Inc. | Device designation of playback and network microphone device arrangements |
US20230290346A1 (en) * | 2018-03-23 | 2023-09-14 | Amazon Technologies, Inc. | Content output management based on speech quality |
US11797263B2 (en) | 2018-05-10 | 2023-10-24 | Sonos, Inc. | Systems and methods for voice-assisted media content selection |
US11792590B2 (en) | 2018-05-25 | 2023-10-17 | Sonos, Inc. | Determining and adapting to changes in microphone performance of playback devices |
US11696074B2 (en) | 2018-06-28 | 2023-07-04 | Sonos, Inc. | Systems and methods for associating playback devices with voice assistant services |
US11482978B2 (en) | 2018-08-28 | 2022-10-25 | Sonos, Inc. | Audio notifications |
US11563842B2 (en) | 2018-08-28 | 2023-01-24 | Sonos, Inc. | Do not disturb feature for audio notifications |
US11778259B2 (en) | 2018-09-14 | 2023-10-03 | Sonos, Inc. | Networked devices, systems and methods for associating playback devices based on sound codes |
US11432030B2 (en) | 2018-09-14 | 2022-08-30 | Sonos, Inc. | Networked devices, systems, and methods for associating playback devices based on sound codes |
US11315553B2 (en) * | 2018-09-20 | 2022-04-26 | Samsung Electronics Co., Ltd. | Electronic device and method for providing or obtaining data for training thereof |
US11790937B2 (en) | 2018-09-21 | 2023-10-17 | Sonos, Inc. | Voice detection optimization using sound metadata |
US11790911B2 (en) | 2018-09-28 | 2023-10-17 | Sonos, Inc. | Systems and methods for selective wake word detection using neural network models |
US11899519B2 (en) | 2018-10-23 | 2024-02-13 | Sonos, Inc. | Multiple stage network microphone device with reduced power consumption and processing load |
US11741948B2 (en) | 2018-11-15 | 2023-08-29 | Sonos Vox France Sas | Dilated convolutions and gating for efficient keyword spotting |
US11557294B2 (en) | 2018-12-07 | 2023-01-17 | Sonos, Inc. | Systems and methods of operating media playback systems having multiple voice assistant services |
US11538460B2 (en) | 2018-12-13 | 2022-12-27 | Sonos, Inc. | Networked microphone devices, systems, and methods of localized arbitration |
US11540047B2 (en) | 2018-12-20 | 2022-12-27 | Sonos, Inc. | Optimization of network microphone devices using noise classification |
US11315556B2 (en) | 2019-02-08 | 2022-04-26 | Sonos, Inc. | Devices, systems, and methods for distributed voice processing by transmitting sound data associated with a wake word to an appropriate device for identification |
US11646023B2 (en) | 2019-02-08 | 2023-05-09 | Sonos, Inc. | Devices, systems, and methods for distributed voice processing |
US11822601B2 (en) | 2019-03-15 | 2023-11-21 | Spotify Ab | Ensemble-based data comparison |
US11798553B2 (en) | 2019-05-03 | 2023-10-24 | Sonos, Inc. | Voice assistant persistence across multiple network microphone devices |
US11361756B2 (en) | 2019-06-12 | 2022-06-14 | Sonos, Inc. | Conditional wake word eventing based on environment |
US11854547B2 (en) | 2019-06-12 | 2023-12-26 | Sonos, Inc. | Network microphone device with command keyword eventing |
US11501773B2 (en) | 2019-06-12 | 2022-11-15 | Sonos, Inc. | Network microphone device with command keyword conditioning |
US11200894B2 (en) | 2019-06-12 | 2021-12-14 | Sonos, Inc. | Network microphone device with command keyword eventing |
US11551669B2 (en) | 2019-07-31 | 2023-01-10 | Sonos, Inc. | Locally distributed keyword detection |
US11714600B2 (en) | 2019-07-31 | 2023-08-01 | Sonos, Inc. | Noise classification for event detection |
US11710487B2 (en) | 2019-07-31 | 2023-07-25 | Sonos, Inc. | Locally distributed keyword detection |
US11551678B2 (en) | 2019-08-30 | 2023-01-10 | Spotify Ab | Systems and methods for generating a cleaned version of ambient sound |
US11862161B2 (en) | 2019-10-22 | 2024-01-02 | Sonos, Inc. | VAS toggle based on device orientation |
US11869503B2 (en) | 2019-12-20 | 2024-01-09 | Sonos, Inc. | Offline voice control |
US11562740B2 (en) | 2020-01-07 | 2023-01-24 | Sonos, Inc. | Voice verification for media playback |
US11961519B2 (en) | 2020-02-07 | 2024-04-16 | Sonos, Inc. | Localized wakeword verification |
US11308958B2 (en) | 2020-02-07 | 2022-04-19 | Sonos, Inc. | Localized wakeword verification |
US11328722B2 (en) * | 2020-02-11 | 2022-05-10 | Spotify Ab | Systems and methods for generating a singular voice audio stream |
US11810564B2 (en) | 2020-02-11 | 2023-11-07 | Spotify Ab | Dynamic adjustment of wake word acceptance tolerance thresholds in voice-controlled devices |
US11308959B2 (en) | 2020-02-11 | 2022-04-19 | Spotify Ab | Dynamic adjustment of wake word acceptance tolerance thresholds in voice-controlled devices |
US20230352024A1 (en) * | 2020-05-20 | 2023-11-02 | Sonos, Inc. | Input detection windowing |
US20220319513A1 (en) * | 2020-05-20 | 2022-10-06 | Sonos, Inc. | Input detection windowing |
US11308962B2 (en) * | 2020-05-20 | 2022-04-19 | Sonos, Inc. | Input detection windowing |
US11482224B2 (en) | 2020-05-20 | 2022-10-25 | Sonos, Inc. | Command keywords with input detection windowing |
US11727919B2 (en) | 2020-05-20 | 2023-08-15 | Sonos, Inc. | Memory allocation for keyword spotting engines |
US11694689B2 (en) * | 2020-05-20 | 2023-07-04 | Sonos, Inc. | Input detection windowing |
US11698771B2 (en) | 2020-08-25 | 2023-07-11 | Sonos, Inc. | Vocal guidance engines for playback devices |
CN112164397A (en) * | 2020-09-30 | 2021-01-01 | 广东美的厨房电器制造有限公司 | Noise reduction control method, cooking appliance, and readable storage medium |
CN112712683A (en) * | 2020-12-14 | 2021-04-27 | 珠海格力电器股份有限公司 | Control method and system of household appliance, remote controller and server |
Also Published As
Publication number | Publication date |
---|---|
EP3844750A4 (en) | 2021-10-27 |
KR20200084458A (en) | 2020-07-13 |
US11404054B2 (en) | 2022-08-02 |
WO2020138844A1 (en) | 2020-07-02 |
CN113287168A (en) | 2021-08-20 |
EP3844750A1 (en) | 2021-07-07 |
AU2019416316A1 (en) | 2021-05-06 |
KR102570384B1 (en) | 2023-08-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11404054B2 (en) | Home appliance and method for voice recognition thereof | |
US20220301555A1 (en) | Home appliance and method for voice recognition thereof | |
KR102513297B1 (en) | Electronic device and method for executing function of electronic device | |
US10713007B2 (en) | Architecture for a hub configured to control a second device while a connection to a remote system is unavailable | |
US7065487B2 (en) | Speech recognition method, program and apparatus using multiple acoustic models | |
KR20150007422A (en) | Electric equipment and method for controlling the same | |
JP2019204074A (en) | Speech dialogue method, apparatus and system | |
US20200286635A1 (en) | Home appliance and method for controlling thereof | |
EP2731360A2 (en) | Automatic audio enhancement system | |
KR20190097391A (en) | Apparatus and method for generating audio signal in which noise is attenuated based on phase change in accordance with a frequency change of audio signal | |
JP2010217453A (en) | Microphone system for voice recognition | |
US20180309833A1 (en) | Voice user interface for pairing between devices | |
US11531360B2 (en) | Electronic device and method for controlling thereof | |
KR20200076441A (en) | Electronic apparatus and control method thereof | |
KR20210054246A (en) | Electorinc apparatus and control method thereof | |
US11275347B2 (en) | Home appliance and method for controlling the same | |
US20240005934A1 (en) | Information exchange on mobile devices using audio | |
US20220148575A1 (en) | Electronic apparatus and controlling method thereof | |
WO2023097768A1 (en) | Electronic device control method and apparatus, and electronic device and readable storage medium | |
JP2010178295A (en) | Noise suppressing device, voice acquisition device, and conference system | |
WO2017119901A1 (en) | System and method for speech detection adaptation | |
WO2023042019A1 (en) | Dynamic operation of a voice controlled device | |
CN117198280A (en) | Parameter adjusting method and device, voice system, air conditioner and storage medium | |
KR20200144318A (en) | Method and apparatus for noise control of a washing machine |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LEE, NOKHAENG;REEL/FRAME:051360/0271 Effective date: 20191212 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |