WO2021175336A1 - System and method for using sound to monitor the operation of a washing machine appliance - Google Patents
System and method for using sound to monitor the operation of a washing machine appliance Download PDFInfo
- Publication number
- WO2021175336A1 WO2021175336A1 PCT/CN2021/082469 CN2021082469W WO2021175336A1 WO 2021175336 A1 WO2021175336 A1 WO 2021175336A1 CN 2021082469 W CN2021082469 W CN 2021082469W WO 2021175336 A1 WO2021175336 A1 WO 2021175336A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- washing machine
- sound
- machine appliance
- spectrogram
- appliance
- Prior art date
Links
- 238000005406 washing Methods 0.000 title claims abstract description 102
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000008569 process Effects 0.000 claims abstract description 23
- 230000005236 sound signal Effects 0.000 claims abstract description 20
- 238000012544 monitoring process Methods 0.000 claims abstract description 12
- 238000013473 artificial intelligence Methods 0.000 claims abstract description 6
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 16
- 238000013019 agitation Methods 0.000 claims description 9
- 238000013528 artificial neural network Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 6
- 239000000725 suspension Substances 0.000 claims description 2
- 239000012530 fluid Substances 0.000 description 29
- 238000004891 communication Methods 0.000 description 20
- 239000003599 detergent Substances 0.000 description 11
- 239000000654 additive Substances 0.000 description 7
- 230000009471 action Effects 0.000 description 4
- 230000000996 additive effect Effects 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 239000007844 bleaching agent Substances 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000001035 drying Methods 0.000 description 2
- 230000005284 excitation Effects 0.000 description 2
- 239000002979 fabric softener Substances 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000030808 detection of mechanical stimulus involved in sensory perception of sound Effects 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- -1 e.g. Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000013505 freshwater Substances 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 235000012771 pancakes Nutrition 0.000 description 1
- 239000000843 powder Substances 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 239000008400 supply water Substances 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F34/00—Details of control systems for washing machines, washer-dryers or laundry dryers
- D06F34/14—Arrangements for detecting or measuring specific parameters
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F33/00—Control of operations performed in washing machines or washer-dryers
- D06F33/30—Control of washing machines characterised by the purpose or target of the control
- D06F33/32—Control of operational steps, e.g. optimisation or improvement of operational steps depending on the condition of the laundry
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F33/00—Control of operations performed in washing machines or washer-dryers
- D06F33/30—Control of washing machines characterised by the purpose or target of the control
- D06F33/32—Control of operational steps, e.g. optimisation or improvement of operational steps depending on the condition of the laundry
- D06F33/34—Control of operational steps, e.g. optimisation or improvement of operational steps depending on the condition of the laundry of water filling
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F33/00—Control of operations performed in washing machines or washer-dryers
- D06F33/30—Control of washing machines characterised by the purpose or target of the control
- D06F33/32—Control of operational steps, e.g. optimisation or improvement of operational steps depending on the condition of the laundry
- D06F33/40—Control of operational steps, e.g. optimisation or improvement of operational steps depending on the condition of the laundry of centrifugal separation of water from the laundry
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F33/00—Control of operations performed in washing machines or washer-dryers
- D06F33/30—Control of washing machines characterised by the purpose or target of the control
- D06F33/44—Control of the operating time, e.g. reduction of overall operating time
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F2103/00—Parameters monitored or detected for the control of domestic laundry washing machines, washer-dryers or laundry dryers
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F2103/00—Parameters monitored or detected for the control of domestic laundry washing machines, washer-dryers or laundry dryers
- D06F2103/18—Washing liquid level
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F2103/00—Parameters monitored or detected for the control of domestic laundry washing machines, washer-dryers or laundry dryers
- D06F2103/24—Spin speed; Drum movements
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F2103/00—Parameters monitored or detected for the control of domestic laundry washing machines, washer-dryers or laundry dryers
- D06F2103/26—Imbalance; Noise level
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F2105/00—Systems or parameters controlled or affected by the control systems of washing machines, washer-dryers or laundry dryers
- D06F2105/52—Changing sequence of operational steps; Carrying out additional operational steps; Modifying operational steps, e.g. by extending duration of steps
-
- D—TEXTILES; PAPER
- D06—TREATMENT OF TEXTILES OR THE LIKE; LAUNDERING; FLEXIBLE MATERIALS NOT OTHERWISE PROVIDED FOR
- D06F—LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES
- D06F34/00—Details of control systems for washing machines, washer-dryers or laundry dryers
- D06F34/04—Signal transfer or data transmission arrangements
Definitions
- the present subject matter relates generally to washing machine appliances, or more specifically, to systems and methods for monitoring sounds within a washing machine appliance and analyzing those sounds to identify sound signatures associated with particular events.
- Washing machine appliances generally include a tub for containing water or wash fluid, e.g., water and detergent, bleach, and/or other wash additives.
- a basket is rotatably mounted within the tub and defines a wash chamber for receipt of articles for washing.
- the wash fluid is directed into the tub and onto articles within the wash chamber of the basket.
- the basket or an agitation element can rotate at various speeds to agitate articles within the wash chamber, to wring wash fluid from articles within the wash chamber, etc.
- a drain pump assembly may operate to discharge water from within sump.
- washing machine appliance it is frequently desirable to monitor sounds generated by a washing machine appliance during operation, e.g., to identify unintended objects in a wash load, to diagnose mechanical failures, or to detect other operating conditions.
- conventional washing machines lack any sound feedback systems. Certain washing machine may monitor sounds and provide a notification when a sound exceeds a certain threshold, but such systems have limited usefulness and effectiveness.
- a washing machine appliance with features for improved operation would be desirable. More specifically, a system and method for monitoring sounds generated by a washing machine appliance and identifying sound signatures associated with particular operating conditions would be particularly beneficial.
- a washing machine appliance including a wash tub positioned within a cabinet and defining a wash chamber, a wash basket rotatably mounted within the wash tub and being configured for receiving of a load of articles for washing, and a motor operably coupled to the wash basket for selectively rotating the wash basket.
- a microphone is provided for monitoring sound generated during operation of the washing machine appliance and a controller is operably coupled to the microphone.
- the controller is configured for obtaining a sound signal generated during operation of the washing machine appliance using the microphone, generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time, identifying a sound signature by analyzing the spectrogram using an image recognition process, and adjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
- a method of operating a washing machine appliance includes a wash basket rotatably mounted within a wash tub, a motor operably coupled to the wash basket for selectively rotating the wash basket, and a microphone for monitoring sound generated by the washing machine appliance.
- the method includes obtaining a sound signal generated during operation of the washing machine appliance using the microphone, generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time, identifying a sound signature by analyzing the spectrogram using an image recognition process, and adjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
- an appliance including a microphone for monitoring sound generated during operation of the appliance and a controller operably coupled to the microphone.
- the controller is configured for obtaining a sound signal generated during operation of the appliance using the microphone, generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time, identifying a sound signature by analyzing the spectrogram using an image processing technique, and adjusting at least one operating parameter of the appliance based at least in part on the identification of the sound signature.
- FIG. 1 provides a perspective view of an exemplary washing machine appliance according to an exemplary embodiment of the present subject matter.
- FIG. 2 provides a side cross-sectional view of the exemplary washing machine appliance of FIG 1.
- FIG. 3 illustrates a method for using sounds generated by a washing machine appliance to identify operating conditions in accordance with one embodiment of the present disclosure.
- FIG. 4 provides an exemplary spectrogram according to an exemplary embodiment of the present subject matter.
- FIG. 1 is a perspective view of an exemplary horizontal axis washing machine appliance 100 and FIG. 2 is a side cross-sectional view of washing machine appliance 100.
- washing machine appliance 100 generally defines a vertical direction V, a lateral direction L, and a transverse direction T, each of which is mutually perpendicular, such that an orthogonal coordinate system is generally defined.
- Washing machine appliance 100 includes a cabinet 102 that extends between a top 104 and a bottom 106 along the vertical direction V, between a left side 108 and a right side 110 along the lateral direction, and between a front 112 and a rear 114 along the transverse direction T.
- wash basket 120 is rotatably mounted within cabinet 102 such that it is rotatable about an axis of rotation A.
- a motor 122 e.g., such as a pancake motor, is in mechanical communication with wash basket 120 to selectively rotate wash basket 120 (e.g., during an agitation or a rinse cycle of washing machine appliance 100) .
- Wash basket 120 is received within a wash tub 124 and defines a wash chamber 126 that is configured for receipt of articles for washing.
- the wash tub 124 holds wash and rinse fluids for agitation in wash basket 120 within wash tub 124.
- wash fluid may refer to water, detergent, fabric softener, bleach, or any other suitable wash additive or combination thereof. Indeed, for simplicity of discussion, these terms may all be used interchangeably herein without limiting the present subject matter to any particular “wash fluid. ”
- Wash basket 120 may define one or more agitator features that extend into wash chamber 126 to assist in agitation and cleaning articles disposed within wash chamber 126 during operation of washing machine appliance 100.
- a plurality of ribs 128 extends from basket 120 into wash chamber 126. In this manner, for example, ribs 128 may lift articles disposed in wash basket 120 during rotation of wash basket 120.
- cabinet 102 also includes a front panel 130 which defines an opening 132 that permits user access to wash basket 120 of wash tub 124. More specifically, washing machine appliance 100 includes a door 134 that is positioned over opening 132 and is rotatably mounted to front panel 130. In this manner, door 134 permits selective access to opening 132 by being movable between an open position (not shown) facilitating access to a wash tub 124 and a closed position (FIG. 1) prohibiting access to wash tub 124.
- a window 136 in door 134 permits viewing of wash basket 120 when door 134 is in the closed position, e.g., during operation of washing machine appliance 100.
- Door 134 also includes a handle (not shown) that, e.g., a user may pull when opening and closing door 134.
- door 134 is illustrated as mounted to front panel 130, it should be appreciated that door 134 may be mounted to another side of cabinet 102 or any other suitable support according to alternative embodiments.
- wash basket 120 also defines a plurality of perforations 140 in order to facilitate fluid communication between an interior of basket 120 and wash tub 124.
- a sump 142 is defined by wash tub 124 at a bottom of wash tub 124 along the vertical direction V.
- sump 142 is configured for receipt of and generally collects wash fluid during operation of washing machine appliance 100.
- wash fluid may be urged by gravity from basket 120 to sump 142 through plurality of perforations 140.
- a drain pump assembly 144 is located beneath wash tub 124 and is in fluid communication with sump 142 for periodically discharging soiled wash fluid from washing machine appliance 100.
- Drain pump assembly 144 may generally include a drain pump 146 which is in fluid communication with sump 142 and with an external drain 148 through a drain hose 150.
- drain pump 146 urges a flow of wash fluid from sump 142, through drain hose 150, and to external drain 148.
- drain pump 146 includes a motor (not shown) which is energized during a drain cycle such that drain pump 146 draws wash fluid from sump 142 and urges it through drain hose 150 to external drain 148.
- a spout 154 is configured for directing a flow of fluid into wash tub 124.
- spout 154 may be in fluid communication with a water supply 155 (FIG. 2) in order to direct fluid (e.g., clean water or wash fluid) into wash tub 124.
- Spout 154 may also be in fluid communication with the sump 142.
- pump assembly 144 may direct wash fluid disposed in sump 142 to spout 154 in order to circulate wash fluid in wash tub 124.
- a detergent drawer 156 is slidably mounted within front panel 130.
- Detergent drawer 156 receives a wash additive (e.g., detergent, fabric softener, bleach, or any other suitable liquid or powder) and directs the fluid additive to wash tub 124 during operation of washing machine appliance 100.
- a wash additive e.g., detergent, fabric softener, bleach, or any other suitable liquid or powder
- detergent drawer 156 may also be fluidly coupled to spout 154 to facilitate the complete and accurate dispensing of wash additive.
- a water supply valve or control valve 158 may provide a flow of water from a water supply source (such as a municipal water supply 155) into detergent dispenser 156 and into wash tub 124.
- control valve 158 may generally be operable to supply water into detergent dispenser 156 to generate a wash fluid, e.g., for use in a wash cycle, or a flow of fresh water, e.g., for a rinse cycle.
- control valve 158 may be positioned at any other suitable location within cabinet 102.
- control valve 158 is described herein as regulating the flow of “wash fluid, ” it should be appreciated that this term includes, water, detergent, other additives, or some mixture thereof.
- a control panel 160 including a plurality of input selectors 162 is coupled to front panel 130.
- Control panel 160 and input selectors 162 collectively form a user interface input for operator selection of machine cycles and features.
- a display 164 indicates selected features, a countdown timer, and/or other items of interest to machine users.
- washing machine appliance 100 Operation of washing machine appliance 100 is controlled by a controller or processing device 166 (FIG. 1) that is operatively coupled to control panel 160 for user manipulation to select washing machine cycles and features.
- controller 166 operates the various components of washing machine appliance 100 to execute selected machine cycles and features.
- Controller 166 may include a memory and microprocessor, such as a general or special purpose microprocessor operable to execute programming instructions or micro-control code associated with a cleaning cycle.
- the memory may represent random access memory such as DRAM, or read only memory such as ROM or FLASH.
- the processor executes programming instructions stored in memory.
- the memory may be a separate component from the processor or may be included onboard within the processor.
- controller 166 may be constructed without using a microprocessor, e.g., using a combination of discrete analog and/or digital logic circuitry (such as switches, amplifiers, integrators, comparators, flip-flops, AND gates, and the like) to perform control functionality instead of relying upon software.
- Control panel 160 and other components of washing machine appliance 100 may be in communication with controller 166 via one or more signal lines or shared communication busses.
- washing machine appliance 100 During operation of washing machine appliance 100, laundry items are loaded into wash basket 120 through opening 132, and washing operation is initiated through operator manipulation of input selectors 162. Wash tub 124 is filled with water, detergent, and/or other fluid additives, e.g., via spout 154 and or detergent drawer 156.
- One or more valves e.g., control valve 158, can be controlled by washing machine appliance 100 to provide for filling wash basket 120 to the appropriate level for the amount of articles being washed and/or rinsed.
- the contents of wash basket 120 can be agitated (e.g., with ribs 128) for washing of laundry items in wash basket 120.
- wash tub 124 can be drained. Laundry articles can then be rinsed by again adding fluid to wash tub 124, depending on the particulars of the cleaning cycle selected by a user. Ribs 128 may again provide agitation within wash basket 120.
- One or more spin cycles may also be used. In particular, a spin cycle may be applied after the wash cycle and/or after the rinse cycle in order to wring wash fluid from the articles being washed.
- basket 120 is rotated at relatively high speeds and drain pump assembly 144 may discharge wash fluid from sump 142. After articles disposed in wash basket 120 are cleaned, washed, and/or rinsed, the user can remove the articles from wash basket 120, e.g., by opening door 134 and reaching into wash basket 120 through opening 132.
- Washing machine appliance 100 may further include a microphone 180 that is used for monitoring the sound waves, noises, or other vibrations generated during the operation of washing machine appliance 100.
- microphone 180 may be one or more microphones, acoustic detection devices, vibration sensors, or any other suitable acoustic transducers that are positioned at one or more locations in or around washing machine appliance 100.
- microphone 180 may be mounted within cabinet 102.
- microphone 180 may be positioned elsewhere within the room or residence where washing machine appliance 100 is located.
- any suitable microphone 180 that is acoustically coupled with washing machine appliance 100 may be used to monitor sounds generated by washing machine appliance 100.
- the sounds generated during operation of washing machine appliance may be associated with one or more operating conditions, failure modes, event occurrences, the presence of one or more distinct items within a wash load, etc.
- the noise of these items striking wash basket 120 may create a unique sound signature, identifiable for example by natural resonant frequencies, amplitudes, the time-based excitations, the excitation rate (e.g., the speed at which a particular sound is triggered) , the time decay of the generated sound waves, or any other acoustic signature or characteristic.
- aspects of the present subject matter are directed to systems and methods for monitoring sounds generated by an appliance, converting those sounds into a three-dimensional spectrogram, and using artificial intelligence image recognition processes to identify sounds signatures in the spectrogram.
- washing machine appliance 100 may generally include an external communication system 190 which is configured for enabling the user to interact with washing machine appliance 100 using a remote device 192.
- external communication system 190 is configured for enabling communication between a user, an appliance, and a remote server or network 194.
- washing machine appliance 100 may communicate with a remote device 192 either directly (e.g., through a local area network (LAN) , Wi-Fi, Bluetooth, etc. ) or indirectly (e.g., via a network 194) , as well as with a remote server, e.g., to receive notifications, provide confirmations, input operational data, transmit sound signals and sound signatures, etc.
- LAN local area network
- Wi-Fi Wireless Fidelity
- remote device 192 may be any suitable device for providing and/or receiving communications or commands from a user.
- remote device 192 may include, for example, a personal phone, a tablet, a laptop computer, or another mobile device.
- communication between the appliance and the user may be achieved directly through an appliance control panel (e.g., control panel 160) .
- network 194 can be any type of communication network.
- network 194 can include one or more of a wireless network, a wired network, a personal area network, a local area network, a wide area network, the internet, a cellular network, etc.
- communication with network may use any of a variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP) , encodings or formats (e.g. HTML, XML) , and/or protection schemes (e.g., VPN, secure HTTP, SSL) .
- External communication system 190 is described herein according to an exemplary embodiment of the present subject matter. However, it should be appreciated that the exemplary functions and configurations of external communication system 190 provided herein are used only as examples to facilitate description of aspects of the present subject matter. System configurations may vary, other communication devices may be used to communicate directly or indirectly with one or more appliances, other communication protocols and steps may be implemented, etc. These variations and modifications are contemplated as within the scope of the present subject matter.
- horizontal axis washing machine appliance 100 While described in the context of a specific embodiment of horizontal axis washing machine appliance 100, using the teachings disclosed herein it will be understood that horizontal axis washing machine appliance 100 is provided by way of example only. Other washing machine appliances having different configurations, different appearances, and/or different features may also be utilized with the present subject matter as well, e.g., vertical axis washing machine appliances. Moreover, the systems and methods described herein may be used to monitor sounds generated by any other suitable appliance or appliances.
- washing machine appliance 100 and the configuration of controller 166 according to exemplary embodiments have been presented, an exemplary method 200 of operating a washing machine appliance will be described.
- the discussion below refers to the exemplary method 200 of operating washing machine appliance 100, one skilled in the art will appreciate that the exemplary method 200 is applicable to the operation of a variety of other washing machine appliances, such as vertical axis washing machine appliances.
- the various method steps as disclosed herein may be performed by controller 166 or a separate, dedicated controller.
- method 200 includes, at step 210, obtaining a sound signal generated during operation of a washing machine appliance using a microphone.
- microphone 180 may be used to detect noises, sounds, vibrations, or other acoustic waves generated during the operation of washing machine appliance 100.
- step 210 may include monitoring the sounds generated by washing machine appliance 100 while it is not in operation, sounds generated during a diagnostic procedure, or any other suitable beeps, indicators, or sound waves that emanate from washing machine appliance 100.
- Step 220 includes generating a spectrogram from the sound signal.
- controller 166 may be configured for converting a sound clip or sound recording into a spectrogram for subsequent analysis.
- the original recording of sound from step 210 may be in the form of noise amplitude versus time, noise frequency versus time, noise amplitude versus noise frequency (e.g., a full Fourier transform or FFT) , or any other suitable two-dimensional representation of the measured sound.
- any suitable duration of sound may be measured at step 210 and converted at step 220.
- the sound signal is between about 0.1 seconds and 10 seconds, between about 1 in 5 seconds, or about 3 seconds.
- the spectrogram generated at step 220 may be a three-dimensional representation of sound pressure or amplitude at a given frequency and time.
- spectrograms may be a two-dimensional graphs, with a third dimension represented by colors.
- the spectrogram represents both a sound frequency and a sound amplitude of over time.
- such a spectrogram may be a visual representation of the spectrum of frequencies of a signal as it varies with time, sometimes referred to as waterfall diagrams.
- FIG. 4 provides an exemplary spectrogram that may be generated and analyzed according to aspects of the present subject matter.
- controller 166 may use various image recognition processes or processing tools to identify noise sources and operating conditions, and may use such information for improving machine performance, e.g., by scheduling maintenance visits, adjusting operating parameters, providing user notifications, etc.
- spectrogram images may add the element of time and may use color temperature to signal intensity or noise amplitude for improved knowledge of the appliance state or operation.
- Step 230 includes identifying a sound signature by analyzing the spectrogram using an image recognition process.
- image recognition processes that rely on artificial intelligence, neural networks, or any other suitable known image processing techniques may be used while remaining within the scope of the present subject matter.
- using such a spectrogram image provides several advantages over existing sound recognition processes.
- a spectrogram provides the potential to use a variety of sophisticated image recognitions models.
- portions of the image recognition processes may use single-label image convolution neural networks (CNNs) as the main algorithm to compare/classify spectrograms.
- CNNs single-label image convolution neural networks
- image recognition and similar terms may be used generally to refer to any suitable method of observation, analysis, image decomposition, feature extraction, image classification, etc. of the spectrogram generated from sound signals measured from washing machine appliance 100. It should be appreciated that any suitable image recognition software or process may be used to analyze the spectrograms and controller 166 may be programmed to perform such processes and take corrective action.
- controller may implement a form of image recognition called convolutional neural network ( “CNN” ) image recognition.
- CNN may include taking an input image (e.g., a spectrogram) and using a convolutional neural network to identify unique signatures in the image, referred to herein generally as “sound signatures. ”
- the image recognition process may use any other suitable neural network process.
- an Adam optimizer may be used, binary cross-entropy may be used as a loss function, and softmax as a last layer activation may be used.
- Any other suitable image classification technique may be used according to alternative embodiments. For example, various transfer techniques may be used, but use of such techniques is not required. If using transfer techniques learning, a neural network architecture may be pretrained such as VGG16 /VGG19 /ResNet50 with a public dataset then the last layer may be retrained with an appliance specific dataset.
- the image recognition process may detect dryness or other events that depend on comparison of initial conditions. For example, a dry-initial spectrogram image may be subtracted from a spectrogram image while clothes are drying. The subtracted image may be used to train a neural network with two classes: dry, not dry. If not using any transfer learning VGG16 may be the neural net architecture of choice.
- two spectrogram images may be stacked, e.g., the dry initial spectrogram image from the spectrogram image on top and the spectrogram image while drying on the bottom of the image. In other words, according to exemplary embodiments, two images could be concatenated in any suitable manner and order.
- two or more images could be combined by subtracting two spectrogram images or modifying such images in any other suitable manner.
- This combined image may be used in a similar way to train a neural network with two classes: dry, not dry. If detection of sound events does not require a comparison from the initial conditions, image combination may be avoided. To detect, for example, the washer being ON, a wide variety of spectrograms recording of this event may be collected, label, and trained.
- spectrograms include privacy. For example, sound data collected as an image in inherently more private.
- the spectrogram contains no information about the exact, or even approximate, phase of the signal that it represents, the sound may be protected and may not be derivable from the spectrogram. For this reason, it may not be possible to reverse the process and generate a copy of the original signal from a spectrogram.
- a spectrogram image may allow for more effective memory use since it can be compressed. Notably, compressing the spectrogram may make it easier or less data intensive to transmit.
- controller 166 may further be configured for transmitting the spectrogram (e.g., or the compressed spectrogram) to a remote server (e.g., such as remote server 194) for analysis.
- Controller 166 may further be configured for receiving analytic feedback from remote server 194. In this manner, data processing may be offloaded from controller 166.
- controller 166 may further be configured for learning sound signatures associated with a washing machine appliance 100. For example, common conditions or operating noises may be intentionally generated to train a neural network model. That model may then be used to detect particular sound signatures associated with particular events. Such sound signatures may be stored locally on controller 166 or a remote server 194. In addition, sound signatures may be appliance specific, may be stored according to a particular model or appliance configuration, or may be associated with a washing machine appliance or another appliance in any other suitable manner.
- Step 240 includes adjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
- controller 166 may take corrective action, e.g., by adjusting one or more operating parameters or implementing some other action in response to detecting that sound signature.
- an “operating parameter” of washing machine appliance 100 is any cycle setting, operating time, component setting, spin speed, part configuration, or other operating characteristic that may affect the performance of washing machine appliance 100.
- references to operating parameter adjustments or “adjusting at least one operating parameter” are intended to refer to control actions intended to improve system performance based on the sound signature or other system parameters.
- adjusting an operating parameter may include adjusting an agitation time or an agitation profile, adjusting a water level, limiting a spin speed of wash basket 120, identifying service needs, providing a user with operating guidance, etc.
- Other operating parameter adjustments are possible and within the scope of the present subject matter.
- adjusting an operating parameter may include providing a user notification when the sound signature indicates that a predetermined operating condition exists.
- the sound signature may be associated with sounds generated from one or more of a bearing, a belt, the motor 122, a water valve (e.g., dripping or stuck in the ON position) , a pump, a suspension system, harmonics of structural components, undesirable contact between components or subsystems, etc.
- a user notification may be provided via display 164 or directly to a user’s remote device 192 (e.g., a cell phone, via wireless connection) .
- FIG. 3 depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the steps of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, or modified in various ways without deviating from the scope of the present disclosure. Moreover, although aspects of method 200 are explained using washing machine appliance 100 as an example, it should be appreciated that these methods may be applied to the operation of any suitable washing machine appliance.
Landscapes
- Engineering & Computer Science (AREA)
- Textile Engineering (AREA)
- Control Of Washing Machine And Dryer (AREA)
Abstract
A washing machine appliance includes a microphone for monitoring sound generated during operation of the washing machine appliance and a controller is operably coupled to the microphone. The controller is configured for obtaining a sound signal generated during operation of the washing machine appliance and converting the sound signal into a spectrogram that represents a sound frequency and a sound amplitude over time. An artificial intelligence image recognition process is used to analyze the spectrogram to identify one or more sound signatures that are associated with particular operating conditions, and operation of the washing machine appliance is adjusted based at least in part on the identification of the sound signature.
Description
The present subject matter relates generally to washing machine appliances, or more specifically, to systems and methods for monitoring sounds within a washing machine appliance and analyzing those sounds to identify sound signatures associated with particular events.
Washing machine appliances generally include a tub for containing water or wash fluid, e.g., water and detergent, bleach, and/or other wash additives. A basket is rotatably mounted within the tub and defines a wash chamber for receipt of articles for washing. During normal operation of such washing machine appliances, the wash fluid is directed into the tub and onto articles within the wash chamber of the basket. The basket or an agitation element can rotate at various speeds to agitate articles within the wash chamber, to wring wash fluid from articles within the wash chamber, etc. During a spin or drain cycle, a drain pump assembly may operate to discharge water from within sump.
Notably, it is frequently desirable to monitor sounds generated by a washing machine appliance during operation, e.g., to identify unintended objects in a wash load, to diagnose mechanical failures, or to detect other operating conditions. However, conventional washing machines lack any sound feedback systems. Certain washing machine may monitor sounds and provide a notification when a sound exceeds a certain threshold, but such systems have limited usefulness and effectiveness.
Accordingly, a washing machine appliance with features for improved operation would be desirable. More specifically, a system and method for monitoring sounds generated by a washing machine appliance and identifying sound signatures associated with particular operating conditions would be particularly beneficial.
BRIEF DESCRIPTION OF THE INVENTION
Advantages of the invention will be set forth in part in the following description, or may be apparent from the description, or may be learned through practice of the invention.
In accordance with one exemplary embodiment of the present disclosure, a washing machine appliance is provided including a wash tub positioned within a cabinet and defining a wash chamber, a wash basket rotatably mounted within the wash tub and being configured for receiving of a load of articles for washing, and a motor operably coupled to the wash basket for selectively rotating the wash basket. A microphone is provided for monitoring sound generated during operation of the washing machine appliance and a controller is operably coupled to the microphone. The controller is configured for obtaining a sound signal generated during operation of the washing machine appliance using the microphone, generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time, identifying a sound signature by analyzing the spectrogram using an image recognition process, and adjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
In accordance with another exemplary embodiment of the present disclosure, a method of operating a washing machine appliance is provided. The washing machine appliance includes a wash basket rotatably mounted within a wash tub, a motor operably coupled to the wash basket for selectively rotating the wash basket, and a microphone for monitoring sound generated by the washing machine appliance. The method includes obtaining a sound signal generated during operation of the washing machine appliance using the microphone, generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time, identifying a sound signature by analyzing the spectrogram using an image recognition process, and adjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
In accordance with another exemplary embodiment of the present disclosure, an appliance is provided including a microphone for monitoring sound generated during operation of the appliance and a controller operably coupled to the microphone. The controller is configured for obtaining a sound signal generated during operation of the appliance using the microphone, generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time, identifying a sound signature by analyzing the spectrogram using an image processing technique, and adjusting at least one operating parameter of the appliance based at least in part on the identification of the sound signature.
These and other features, aspects and advantages of the present invention will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
A full and enabling disclosure of the present invention, including the best mode thereof, directed to one of ordinary skill in the art, is set forth in the specification, which makes reference to the appended figures.
FIG. 1 provides a perspective view of an exemplary washing machine appliance according to an exemplary embodiment of the present subject matter.
FIG. 2 provides a side cross-sectional view of the exemplary washing machine appliance of FIG 1.
FIG. 3 illustrates a method for using sounds generated by a washing machine appliance to identify operating conditions in accordance with one embodiment of the present disclosure.
FIG. 4 provides an exemplary spectrogram according to an exemplary embodiment of the present subject matter.
Repeat use of reference characters in the present specification and drawings is intended to represent the same or analogous features or elements of the present invention.
Reference now will be made in detail to embodiments of the invention, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the invention, not limitation of the invention. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the scope or spirit of the invention. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present invention covers such modifications and variations as come within the scope of the appended claims and their equivalents.
Referring now to the figures, FIG. 1 is a perspective view of an exemplary horizontal axis washing machine appliance 100 and FIG. 2 is a side cross-sectional view of washing machine appliance 100. As illustrated, washing machine appliance 100 generally defines a vertical direction V, a lateral direction L, and a transverse direction T, each of which is mutually perpendicular, such that an orthogonal coordinate system is generally defined. Washing machine appliance 100 includes a cabinet 102 that extends between a top 104 and a bottom 106 along the vertical direction V, between a left side 108 and a right side 110 along the lateral direction, and between a front 112 and a rear 114 along the transverse direction T.
Referring to FIG. 2, a wash basket 120 is rotatably mounted within cabinet 102 such that it is rotatable about an axis of rotation A. A motor 122, e.g., such as a pancake motor, is in mechanical communication with wash basket 120 to selectively rotate wash basket 120 (e.g., during an agitation or a rinse cycle of washing machine appliance 100) . Wash basket 120 is received within a wash tub 124 and defines a wash chamber 126 that is configured for receipt of articles for washing. The wash tub 124 holds wash and rinse fluids for agitation in wash basket 120 within wash tub 124. As used herein, “wash fluid” may refer to water, detergent, fabric softener, bleach, or any other suitable wash additive or combination thereof. Indeed, for simplicity of discussion, these terms may all be used interchangeably herein without limiting the present subject matter to any particular “wash fluid. ”
Referring generally to FIGS. 1 and 2, cabinet 102 also includes a front panel 130 which defines an opening 132 that permits user access to wash basket 120 of wash tub 124. More specifically, washing machine appliance 100 includes a door 134 that is positioned over opening 132 and is rotatably mounted to front panel 130. In this manner, door 134 permits selective access to opening 132 by being movable between an open position (not shown) facilitating access to a wash tub 124 and a closed position (FIG. 1) prohibiting access to wash tub 124.
A window 136 in door 134 permits viewing of wash basket 120 when door 134 is in the closed position, e.g., during operation of washing machine appliance 100. Door 134 also includes a handle (not shown) that, e.g., a user may pull when opening and closing door 134. Further, although door 134 is illustrated as mounted to front panel 130, it should be appreciated that door 134 may be mounted to another side of cabinet 102 or any other suitable support according to alternative embodiments.
Referring again to FIG. 2, wash basket 120 also defines a plurality of perforations 140 in order to facilitate fluid communication between an interior of basket 120 and wash tub 124. A sump 142 is defined by wash tub 124 at a bottom of wash tub 124 along the vertical direction V. Thus, sump 142 is configured for receipt of and generally collects wash fluid during operation of washing machine appliance 100. For example, during operation of washing machine appliance 100, wash fluid may be urged by gravity from basket 120 to sump 142 through plurality of perforations 140.
A drain pump assembly 144 is located beneath wash tub 124 and is in fluid communication with sump 142 for periodically discharging soiled wash fluid from washing machine appliance 100. Drain pump assembly 144 may generally include a drain pump 146 which is in fluid communication with sump 142 and with an external drain 148 through a drain hose 150. During a drain cycle, drain pump 146 urges a flow of wash fluid from sump 142, through drain hose 150, and to external drain 148. More specifically, drain pump 146 includes a motor (not shown) which is energized during a drain cycle such that drain pump 146 draws wash fluid from sump 142 and urges it through drain hose 150 to external drain 148.
A spout 154 is configured for directing a flow of fluid into wash tub 124. For example, spout 154 may be in fluid communication with a water supply 155 (FIG. 2) in order to direct fluid (e.g., clean water or wash fluid) into wash tub 124. Spout 154 may also be in fluid communication with the sump 142. For example, pump assembly 144 may direct wash fluid disposed in sump 142 to spout 154 in order to circulate wash fluid in wash tub 124.
As illustrated in FIG. 2, a detergent drawer 156 is slidably mounted within front panel 130. Detergent drawer 156 receives a wash additive (e.g., detergent, fabric softener, bleach, or any other suitable liquid or powder) and directs the fluid additive to wash tub 124 during operation of washing machine appliance 100. According to the illustrated embodiment, detergent drawer 156 may also be fluidly coupled to spout 154 to facilitate the complete and accurate dispensing of wash additive.
In addition, a water supply valve or control valve 158 may provide a flow of water from a water supply source (such as a municipal water supply 155) into detergent dispenser 156 and into wash tub 124. In this manner, control valve 158 may generally be operable to supply water into detergent dispenser 156 to generate a wash fluid, e.g., for use in a wash cycle, or a flow of fresh water, e.g., for a rinse cycle. It should be appreciated that control valve 158 may be positioned at any other suitable location within cabinet 102. In addition, although control valve 158 is described herein as regulating the flow of “wash fluid, ” it should be appreciated that this term includes, water, detergent, other additives, or some mixture thereof.
A control panel 160 including a plurality of input selectors 162 is coupled to front panel 130. Control panel 160 and input selectors 162 collectively form a user interface input for operator selection of machine cycles and features. For example, in one embodiment, a display 164 indicates selected features, a countdown timer, and/or other items of interest to machine users.
Operation of washing machine appliance 100 is controlled by a controller or processing device 166 (FIG. 1) that is operatively coupled to control panel 160 for user manipulation to select washing machine cycles and features. In response to user manipulation of control panel 160, controller 166 operates the various components of washing machine appliance 100 to execute selected machine cycles and features.
During operation of washing machine appliance 100, laundry items are loaded into wash basket 120 through opening 132, and washing operation is initiated through operator manipulation of input selectors 162. Wash tub 124 is filled with water, detergent, and/or other fluid additives, e.g., via spout 154 and or detergent drawer 156. One or more valves (e.g., control valve 158) can be controlled by washing machine appliance 100 to provide for filling wash basket 120 to the appropriate level for the amount of articles being washed and/or rinsed. By way of example for a wash mode, once wash basket 120 is properly filled with fluid, the contents of wash basket 120 can be agitated (e.g., with ribs 128) for washing of laundry items in wash basket 120.
After the agitation phase of the wash cycle is completed, wash tub 124 can be drained. Laundry articles can then be rinsed by again adding fluid to wash tub 124, depending on the particulars of the cleaning cycle selected by a user. Ribs 128 may again provide agitation within wash basket 120. One or more spin cycles may also be used. In particular, a spin cycle may be applied after the wash cycle and/or after the rinse cycle in order to wring wash fluid from the articles being washed. During a final spin cycle, basket 120 is rotated at relatively high speeds and drain pump assembly 144 may discharge wash fluid from sump 142. After articles disposed in wash basket 120 are cleaned, washed, and/or rinsed, the user can remove the articles from wash basket 120, e.g., by opening door 134 and reaching into wash basket 120 through opening 132.
Notably, the sounds generated during operation of washing machine appliance may be associated with one or more operating conditions, failure modes, event occurrences, the presence of one or more distinct items within a wash load, etc. For example, if a user accidently leaves loose coins or a belt in a wash load, the noise of these items striking wash basket 120 may create a unique sound signature, identifiable for example by natural resonant frequencies, amplitudes, the time-based excitations, the excitation rate (e.g., the speed at which a particular sound is triggered) , the time decay of the generated sound waves, or any other acoustic signature or characteristic. As explained in more detail below, aspects of the present subject matter are directed to systems and methods for monitoring sounds generated by an appliance, converting those sounds into a three-dimensional spectrogram, and using artificial intelligence image recognition processes to identify sounds signatures in the spectrogram.
In addition, referring again to FIG. 1, washing machine appliance 100 may generally include an external communication system 190 which is configured for enabling the user to interact with washing machine appliance 100 using a remote device 192. Specifically, according to an exemplary embodiment, external communication system 190 is configured for enabling communication between a user, an appliance, and a remote server or network 194. According to exemplary embodiments, washing machine appliance 100 may communicate with a remote device 192 either directly (e.g., through a local area network (LAN) , Wi-Fi, Bluetooth, etc. ) or indirectly (e.g., via a network 194) , as well as with a remote server, e.g., to receive notifications, provide confirmations, input operational data, transmit sound signals and sound signatures, etc.
In general, remote device 192 may be any suitable device for providing and/or receiving communications or commands from a user. In this regard, remote device 192 may include, for example, a personal phone, a tablet, a laptop computer, or another mobile device. In addition, or alternatively, communication between the appliance and the user may be achieved directly through an appliance control panel (e.g., control panel 160) .
In general, network 194 can be any type of communication network. For example, network 194 can include one or more of a wireless network, a wired network, a personal area network, a local area network, a wide area network, the internet, a cellular network, etc. In general, communication with network may use any of a variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP) , encodings or formats (e.g. HTML, XML) , and/or protection schemes (e.g., VPN, secure HTTP, SSL) .
While described in the context of a specific embodiment of horizontal axis washing machine appliance 100, using the teachings disclosed herein it will be understood that horizontal axis washing machine appliance 100 is provided by way of example only. Other washing machine appliances having different configurations, different appearances, and/or different features may also be utilized with the present subject matter as well, e.g., vertical axis washing machine appliances. Moreover, the systems and methods described herein may be used to monitor sounds generated by any other suitable appliance or appliances.
Now that the construction of washing machine appliance 100 and the configuration of controller 166 according to exemplary embodiments have been presented, an exemplary method 200 of operating a washing machine appliance will be described. Although the discussion below refers to the exemplary method 200 of operating washing machine appliance 100, one skilled in the art will appreciate that the exemplary method 200 is applicable to the operation of a variety of other washing machine appliances, such as vertical axis washing machine appliances. In exemplary embodiments, the various method steps as disclosed herein may be performed by controller 166 or a separate, dedicated controller.
Referring generally to FIG. 3, a method of operating a washing machine appliance is provided. According to exemplary embodiments, method 200 includes, at step 210, obtaining a sound signal generated during operation of a washing machine appliance using a microphone. For example, continuing the example from above, microphone 180 may be used to detect noises, sounds, vibrations, or other acoustic waves generated during the operation of washing machine appliance 100. In addition, or alternatively, step 210 may include monitoring the sounds generated by washing machine appliance 100 while it is not in operation, sounds generated during a diagnostic procedure, or any other suitable beeps, indicators, or sound waves that emanate from washing machine appliance 100.
Step 220 includes generating a spectrogram from the sound signal. In this regard, for example, controller 166 may be configured for converting a sound clip or sound recording into a spectrogram for subsequent analysis. Thus, the original recording of sound from step 210 may be in the form of noise amplitude versus time, noise frequency versus time, noise amplitude versus noise frequency (e.g., a full Fourier transform or FFT) , or any other suitable two-dimensional representation of the measured sound. In addition, any suitable duration of sound may be measured at step 210 and converted at step 220. For example, according to exemplary embodiments, the sound signal is between about 0.1 seconds and 10 seconds, between about 1 in 5 seconds, or about 3 seconds.
Notably, the spectrogram generated at step 220 may be a three-dimensional representation of sound pressure or amplitude at a given frequency and time. Specifically, spectrograms may be a two-dimensional graphs, with a third dimension represented by colors. According to exemplary embodiments, the spectrogram represents both a sound frequency and a sound amplitude of over time. For example, such a spectrogram may be a visual representation of the spectrum of frequencies of a signal as it varies with time, sometimes referred to as waterfall diagrams. FIG. 4 provides an exemplary spectrogram that may be generated and analyzed according to aspects of the present subject matter. Notably, once the sound signal is converted to a spectrogram, controller 166 may use various image recognition processes or processing tools to identify noise sources and operating conditions, and may use such information for improving machine performance, e.g., by scheduling maintenance visits, adjusting operating parameters, providing user notifications, etc. In this regard, spectrogram images may add the element of time and may use color temperature to signal intensity or noise amplitude for improved knowledge of the appliance state or operation.
Step 230 includes identifying a sound signature by analyzing the spectrogram using an image recognition process. For example, image recognition processes that rely on artificial intelligence, neural networks, or any other suitable known image processing techniques may be used while remaining within the scope of the present subject matter. Specifically, using such a spectrogram image provides several advantages over existing sound recognition processes.
For example, the use of a spectrogram provides the potential to use a variety of sophisticated image recognitions models. According to an exemplary embodiment, portions of the image recognition processes may use single-label image convolution neural networks (CNNs) as the main algorithm to compare/classify spectrograms. As used herein, the terms image recognition and similar terms may be used generally to refer to any suitable method of observation, analysis, image decomposition, feature extraction, image classification, etc. of the spectrogram generated from sound signals measured from washing machine appliance 100. It should be appreciated that any suitable image recognition software or process may be used to analyze the spectrograms and controller 166 may be programmed to perform such processes and take corrective action.
According to an exemplary embodiment, controller may implement a form of image recognition called convolutional neural network ( “CNN” ) image recognition. Generally speaking, CNN may include taking an input image (e.g., a spectrogram) and using a convolutional neural network to identify unique signatures in the image, referred to herein generally as “sound signatures. ” According to still other embodiments, the image recognition process may use any other suitable neural network process.
In addition, or alternatively, an Adam optimizer may be used, binary cross-entropy may be used as a loss function, and softmax as a last layer activation may be used. Any other suitable image classification technique may be used according to alternative embodiments. For example, various transfer techniques may be used, but use of such techniques is not required. If using transfer techniques learning, a neural network architecture may be pretrained such as VGG16 /VGG19 /ResNet50 with a public dataset then the last layer may be retrained with an appliance specific dataset.
In addition, or alternatively, the image recognition process may detect dryness or other events that depend on comparison of initial conditions. For example, a dry-initial spectrogram image may be subtracted from a spectrogram image while clothes are drying. The subtracted image may be used to train a neural network with two classes: dry, not dry. If not using any transfer learning VGG16 may be the neural net architecture of choice. In addition, or alternatively, two spectrogram images may be stacked, e.g., the dry initial spectrogram image from the spectrogram image on top and the spectrogram image while drying on the bottom of the image. In other words, according to exemplary embodiments, two images could be concatenated in any suitable manner and order. Moreover, according to alternative embodiments, two or more images could be combined by subtracting two spectrogram images or modifying such images in any other suitable manner. This combined image may be used in a similar way to train a neural network with two classes: dry, not dry. If detection of sound events does not require a comparison from the initial conditions, image combination may be avoided. To detect, for example, the washer being ON, a wide variety of spectrograms recording of this event may be collected, label, and trained.
Notably, additional advantages of the use of spectrograms include privacy. For example, sound data collected as an image in inherently more private. In this regard, since the spectrogram contains no information about the exact, or even approximate, phase of the signal that it represents, the sound may be protected and may not be derivable from the spectrogram. For this reason, it may not be possible to reverse the process and generate a copy of the original signal from a spectrogram. In addition, a spectrogram image may allow for more effective memory use since it can be compressed. Notably, compressing the spectrogram may make it easier or less data intensive to transmit. Thus, for example, controller 166 may further be configured for transmitting the spectrogram (e.g., or the compressed spectrogram) to a remote server (e.g., such as remote server 194) for analysis. Controller 166 may further be configured for receiving analytic feedback from remote server 194. In this manner, data processing may be offloaded from controller 166.
Notably, controller 166 may further be configured for learning sound signatures associated with a washing machine appliance 100. For example, common conditions or operating noises may be intentionally generated to train a neural network model. That model may then be used to detect particular sound signatures associated with particular events. Such sound signatures may be stored locally on controller 166 or a remote server 194. In addition, sound signatures may be appliance specific, may be stored according to a particular model or appliance configuration, or may be associated with a washing machine appliance or another appliance in any other suitable manner.
Step 240 includes adjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature. In this regard, if a sound signature associated with a specific condition is identified at step 230, controller 166 may take corrective action, e.g., by adjusting one or more operating parameters or implementing some other action in response to detecting that sound signature.
As used herein, an “operating parameter” of washing machine appliance 100 is any cycle setting, operating time, component setting, spin speed, part configuration, or other operating characteristic that may affect the performance of washing machine appliance 100. Thus, references to operating parameter adjustments or “adjusting at least one operating parameter” are intended to refer to control actions intended to improve system performance based on the sound signature or other system parameters. For example, adjusting an operating parameter may include adjusting an agitation time or an agitation profile, adjusting a water level, limiting a spin speed of wash basket 120, identifying service needs, providing a user with operating guidance, etc. Other operating parameter adjustments are possible and within the scope of the present subject matter.
In addition, according to exemplary embodiments, adjusting an operating parameter may include providing a user notification when the sound signature indicates that a predetermined operating condition exists. For example, according to one exemplary embodiment, the sound signature may be associated with sounds generated from one or more of a bearing, a belt, the motor 122, a water valve (e.g., dripping or stuck in the ON position) , a pump, a suspension system, harmonics of structural components, undesirable contact between components or subsystems, etc. When a sound signature is generated that indicates a particular operating condition, e.g., such as a potential failure of one of these components, a user notification may be provided via display 164 or directly to a user’s remote device 192 (e.g., a cell phone, via wireless connection) .
FIG. 3 depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the steps of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, or modified in various ways without deviating from the scope of the present disclosure. Moreover, although aspects of method 200 are explained using washing machine appliance 100 as an example, it should be appreciated that these methods may be applied to the operation of any suitable washing machine appliance.
This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they include structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.
Claims (20)
- A washing machine appliance comprising:a wash tub positioned within a cabinet and defining a wash chamber;a wash basket rotatably mounted within the wash tub and being configured for receiving of a load of articles for washing;a motor operably coupled to the wash basket for selectively rotating the wash basket;a microphone for monitoring sound generated during operation of the washing machine appliance; anda controller operably coupled to the microphone, the controller being configured for:obtaining a sound signal generated during operation of the washing machine appliance using the microphone;generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time;identifying a sound signature by analyzing the spectrogram using an image recognition process; andadjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
- The washing machine appliance of claim 1, wherein the image recognition process uses artificial intelligence (AI) to analyze the spectrogram.
- The washing machine appliance of claim 1, wherein the image recognition process comprises a convolution neural network (CNN) .
- The washing machine appliance of claim 1, wherein the controller is further configured for:learning a plurality of sound signatures associated with various operating conditions.
- The washing machine appliance of claim 1, wherein the sound signature is associated with sounds generated from at least one of a bearing, a belt, the motor, a water valve, a pump, a suspension system, harmonics of structural components, or undesirable contact between components or subsystems.
- The washing machine appliance of claim 1, wherein adjusting the at least one operating parameter comprises:adjusting an agitation time or profile, adjusting a water level, limiting a spin speed, identifying service needs, or providing a user with operating guidance.
- The washing machine appliance of claim 1, wherein adjusting the at least one operating parameter comprises:selecting an operating cycle based on the sound signature.
- The washing machine appliance of claim 1, wherein the controller is further configured for:providing a user notification when the sound signature indicates that a predetermined operating characteristic exists.
- The washing machine appliance of claim 1, wherein the sound signature is associated with the presence of an unwashable item, and wherein adjusting the at least one operating parameter comprises stopping the wash cycle.
- The washing machine appliance of claim 1, wherein the controller is further configured for:transmitting the spectrogram to a remote server for analysis; andreceiving analytic feedback from the remote server.
- The washing machine appliance of claim 1, wherein the microphone is positioned outside the cabinet and remote from the washing machine appliance.
- A method of operating a washing machine appliance, the washing machine appliance comprising a wash basket rotatably mounted within a wash tub, a motor operably coupled to the wash basket for selectively rotating the wash basket, and a microphone for monitoring sound generated by the washing machine appliance, the method comprising:obtaining a sound signal generated during operation of the washing machine appliance using the microphone;generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time;identifying a sound signature by analyzing the spectrogram using an image recognition process; andadjusting at least one operating parameter of the washing machine appliance based at least in part on the identification of the sound signature.
- The method of claim 12, wherein the image recognition process uses artificial intelligence (AI) to analyze the spectrogram.
- The method of claim 12, wherein the image recognition process comprises a convolution neural network (CNN) .
- The method of claim 12, further comprising:learning a plurality of sound signatures associated with various operating conditions.
- The method of claim 12, wherein the sound signature is associated with the presence of an unwashable item, and wherein adjusting the at least one operating parameter comprises stopping the wash cycle.
- The method of claim 12, further comprising:transmitting the spectrogram to a remote server for analysis; andreceiving analytic feedback from the remote server.
- An appliance comprising:a microphone for monitoring sound generated during operation of the appliance; anda controller operably coupled to the microphone, the controller being configured for:obtaining a sound signal generated during operation of the appliance using the microphone;generating a spectrogram from the sound signal, the spectrogram representing a sound frequency and a sound amplitude over time;identifying a sound signature by analyzing the spectrogram using an image processing technique; andadjusting at least one operating parameter of the appliance based at least in part on the identification of the sound signature.
- The appliance of claim 18, wherein the image recognition process uses artificial intelligence (AI) to analyze the spectrogram.
- The appliance of claim 18, wherein the image recognition process comprises a convolution neural network (CNN) .
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202180019122.3A CN115398057B (en) | 2020-03-03 | 2021-03-23 | System and method for monitoring operation of washing machine device using sound |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/807,269 | 2020-03-03 | ||
US16/807,269 US20210277564A1 (en) | 2020-03-03 | 2020-03-03 | System and method for using sound to monitor the operation of a washing machine appliance |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021175336A1 true WO2021175336A1 (en) | 2021-09-10 |
Family
ID=77555569
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/082469 WO2021175336A1 (en) | 2020-03-03 | 2021-03-23 | System and method for using sound to monitor the operation of a washing machine appliance |
Country Status (3)
Country | Link |
---|---|
US (1) | US20210277564A1 (en) |
CN (1) | CN115398057B (en) |
WO (1) | WO2021175336A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220120026A1 (en) * | 2020-10-16 | 2022-04-21 | Haier Us Appliance Solutions, Inc. | System and method for using sound to monitor the operation of a dryer appliance |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CA3115423A1 (en) * | 2020-05-01 | 2021-11-01 | Systemes De Controle Actif Soft Db Inc. | A system and a method for sound recognition |
US11802894B2 (en) * | 2020-09-17 | 2023-10-31 | Silicon Laboratories Inc. | Compressing information in an end node using an autoencoder neural network |
CA3202105A1 (en) * | 2020-11-17 | 2022-05-27 | Alliance Laundry Systems Llc | Monitoring laundry machine operation using machine learning analysis of acoustic transducer signal information |
CN114488861B (en) * | 2022-01-24 | 2024-08-20 | 杭州老板电器股份有限公司 | Cooking parameter adjustment method and device and electronic equipment |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5842288A (en) * | 1996-12-10 | 1998-12-01 | U.S. Controls Corporation | Clothes dryer with chiming alarm |
JP4375407B2 (en) * | 1995-01-17 | 2009-12-02 | 株式会社日立製作所 | Home appliances |
CN101970744A (en) * | 2008-03-11 | 2011-02-09 | Bsh博世和西门子家用器具有限公司 | Laundry treatment appliance comprising a drum and a sensor, and method for the operation thereof |
CN106868796A (en) * | 2017-03-09 | 2017-06-20 | 青岛海尔智能技术研发有限公司 | A kind of washing machine noise reduction method and washing machine |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102547543B (en) * | 2010-12-10 | 2015-10-21 | 塞舌尔商元鼎音讯股份有限公司 | Increase listens to barrier, and person hears method and the hearing aids of sound correctness |
CN108642802A (en) * | 2018-04-02 | 2018-10-12 | 青岛海尔洗衣机有限公司 | A kind of control method and washing machine of washing machine |
-
2020
- 2020-03-03 US US16/807,269 patent/US20210277564A1/en not_active Abandoned
-
2021
- 2021-03-23 CN CN202180019122.3A patent/CN115398057B/en active Active
- 2021-03-23 WO PCT/CN2021/082469 patent/WO2021175336A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4375407B2 (en) * | 1995-01-17 | 2009-12-02 | 株式会社日立製作所 | Home appliances |
US5842288A (en) * | 1996-12-10 | 1998-12-01 | U.S. Controls Corporation | Clothes dryer with chiming alarm |
CN101970744A (en) * | 2008-03-11 | 2011-02-09 | Bsh博世和西门子家用器具有限公司 | Laundry treatment appliance comprising a drum and a sensor, and method for the operation thereof |
CN106868796A (en) * | 2017-03-09 | 2017-06-20 | 青岛海尔智能技术研发有限公司 | A kind of washing machine noise reduction method and washing machine |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220120026A1 (en) * | 2020-10-16 | 2022-04-21 | Haier Us Appliance Solutions, Inc. | System and method for using sound to monitor the operation of a dryer appliance |
US11692301B2 (en) * | 2020-10-16 | 2023-07-04 | Haier Us Appliance Solutions, Inc. | System and method for using sound to monitor the operation of a dryer appliance |
Also Published As
Publication number | Publication date |
---|---|
US20210277564A1 (en) | 2021-09-09 |
CN115398057B (en) | 2024-01-30 |
CN115398057A (en) | 2022-11-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021175336A1 (en) | System and method for using sound to monitor the operation of a washing machine appliance | |
US20200248354A1 (en) | Washing machine and method for controlling the same | |
CN102888735B (en) | Laundering device vibration control | |
US11692301B2 (en) | System and method for using sound to monitor the operation of a dryer appliance | |
US20110016639A1 (en) | Control method of a laundry treatment machine | |
US9518351B2 (en) | Washing machine appliance | |
US11692299B2 (en) | Dryer appliance and methods for improved operation | |
US20120090098A1 (en) | Method and apparatus to detect an over-suds condition | |
US20230124027A1 (en) | Systems and methods for capturing images for use in artificial intelligence processes in a laundry appliance | |
US10465324B1 (en) | Method for detecting a low water level in a washing machine appliance | |
US20220243377A1 (en) | Image quality detection for a camera assembly in a laundry appliance | |
US20220316124A1 (en) | Leftover load detection method using image recognition in a laundry appliance | |
US11459686B2 (en) | Washing machine, control method of washing machine and server for supporting washing | |
CN103088601A (en) | Roller washing machine and control method thereof | |
US11725324B2 (en) | Washing machine and control method thereof | |
US11053619B2 (en) | Smart auto wash cycle for a washing machine appliance | |
US11021825B2 (en) | Washing machine appliance with location detection of imbalanced loads | |
US10822735B2 (en) | Spin inefficiency detection for a washing machine appliance | |
JP2007167175A (en) | Washing machine | |
US20220178064A1 (en) | Image recognition processes for detecting tangling in a washing machine appliance | |
US20240065515A1 (en) | Dishwasher appliance with camera for determination of spray assembly operation | |
US11739463B2 (en) | Method of using image recognition processes for improved operation of a laundry appliance | |
US20240065520A1 (en) | Clean dish verification using a camera assembly in a dishwasher appliance | |
US11851799B2 (en) | Systems and methods of operating a washing machine appliance to detect and mitigate out-of-balance conditions | |
US12091806B2 (en) | Method of using image recognition processes for improved operation of a laundry appliance |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21764627 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21764627 Country of ref document: EP Kind code of ref document: A1 |