WO2022197005A1

WO2022197005A1 - Audio signal-based device and method for controlling same

Info

Publication number: WO2022197005A1
Application number: PCT/KR2022/003292
Authority: WO
Inventors: 한윤창; 이수빈; 박정수; 정일영; 이돈문; 임현기
Original assignee: 주식회사 코클
Priority date: 2021-03-16
Filing date: 2022-03-08
Publication date: 2022-09-22

Abstract

The present invention relates to a method for controlling an electronic device comprising a microphone, memory, a control unit, and a drive unit, the method being characterized by comprising the steps of: receiving an input of an audio signal from a user; determining whether the input audio signal is a whistle sound; if it is determined that the audio signal is a whistle sound, analyzing the features of the whistle sound; and controlling the operation of the electronic device using the analysis result.

Description

Audio signal-based device and method for controlling the same

The present invention relates to a device using an audio signal such as a whistle and a method for controlling the same.

Recently, artificial intelligence technologies such as deep learning are being applied to process audio. Audio identification technology, which is one of audio-related processing technologies, is developed for the purpose of detecting whether an audio input is generated from a subject and under what circumstances of the subject.

As audio identification technology gradually develops, research on applied technology using the same is being actively conducted. Specifically, various methods for controlling an electronic device based on a user's utterance have emerged. Examples of commercialized methods of speech recognition technology, such as Siri and S Voice, exist.

On the other hand, a method for controlling an electronic device using non-verbal audio that can be generated by a human is limitedly applied. For example, simple methods such as turning a light on and off in response to a sound of applause are being used.

Unlike verbal audio, non-verbal audio such as applause is somewhat limited in containing meaning, so it is difficult to control electronic devices for various purposes.

On the other hand, a whistle sound that can be easily generated by a plurality of users may have various features such as pitch, sound change pattern, and melody. Accordingly, if the audio identification technology is applied to the whistle sound generated by the user, it is possible to control the electronic device only with the simple whistle sound without the user speaking.

The present invention intends to propose a method of extracting an operation command of an electronic device from a whistle sound generated by a user using an audio identification technology and controlling the electronic device based on the extracted operation command.

It is an object of the present invention to provide an electronic device that operates based on non-verbal audio instead of voice.

It is an object of the present invention to provide a control method capable of operating an electronic device based on a whistle sound.

It is an object of the present invention to analyze the pitch, melody, and change of a whistle, and to provide a control method for operating an electronic device based on the analysis result.

The present invention relates to a control method of an electronic device including a microphone, a memory, a control unit and a driving unit, comprising the steps of: receiving an audio signal from a user; determining whether the inputted audio signal is a whistle sound; When it is determined that the signal is a whistle sound, the method may include analyzing the characteristics of the whistle sound and controlling the operation of the electronic device using the analysis result.

According to the present invention, since the user can operate the electronic device only by generating a whistling sound, there is an effect that the user's convenience is improved.

According to the present invention, there is an advantage in that a limitation in that the trigger of the audio-based electronic device control technology is limited to the user's utterance can be solved by using a non-verbal signal.

1 is a conceptual diagram illustrating components of an electronic device that is a target of a control method proposed in the present invention.

2 is a flowchart illustrating a method for controlling an electronic device according to an embodiment of the present invention.

3A and 3B are block diagrams illustrating components of a refrigerator to which the electronic device control method according to the present invention is applied.

4 is a conceptual diagram illustrating a method of controlling an operation of a refrigerator using a whistle sound.

5 is a block diagram showing the components of an AI speaker to which the electronic device control method according to the present invention is applied.

6 is a conceptual diagram illustrating a method of controlling an operation of an AI speaker using a whistle sound.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the drawings and examples disclosed below. In addition, in order to clearly explain the present invention in the drawings, parts irrelevant to the present invention are omitted, and the same or similar symbols in the drawings indicate the same or similar components.

Objects and effects of the present invention can be naturally understood or made clearer by the following description, and if it is determined that the gist of the present invention may be unnecessarily obscured, the detailed description will be omitted. and effects are not limited.

In FIG. 1 , the components of the electronic device 100 are described.

The electronic device 100 described herein is defined as various devices that perform data communication, perform data processing, and perform predetermined operations, such as a refrigerator, artificial intelligence speaker, cleaner, mobile terminal, washing machine, and dryer. can be

For example, when the electronic device 100 is a mobile terminal, the electronic device 100 includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, personal digital assistants (PDA), and a portable terminal (PMP). multimedia player), navigation, slate PC, tablet PC, ultrabook, wearable device, for example, watch-type terminal (smartwatch), glass-type terminal (smart glass) , a head mounted display (HMD), etc. may be included.

However, it will be readily apparent to those skilled in the art that the configuration according to the embodiments described in this specification may be applied to fixed terminals such as digital TV, desktop computer, digital signage, etc., except when applicable only to mobile terminals. will be.

The electronic device 100 includes a wireless communication unit 110 , an input unit 120 , a sensing unit 140 , an output unit 150 , an interface unit 160 , a memory 170 , a control unit 180 , and a power supply unit 190 . ) and the like.

More specifically, the wireless communication unit 110 of the components, between the electronic device 100 and the wireless communication system, between the electronic device 100 and another electronic device 100, or the electronic device 100 and an external server It may include one or more modules that enable wireless communication between them. Also, the wireless communication unit 110 may include one or more modules for connecting the electronic device 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast reception module 111 , a mobile communication module 112 , a wireless Internet module 113 , a short-range communication module 114 , and a location information module 115 . .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 or an audio input unit for inputting an audio signal, and a user input unit 123 for receiving information from a user, for example, , a touch key, a push key, etc.). The voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the collection device, surrounding environment information surrounding the collection device, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity. Sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor: infrared sensor), fingerprint sensor (finger scan sensor), ultrasonic sensor , optical sensors (eg, cameras (see 121)), microphones (see 122), battery gauges, environmental sensors (eg, barometers, hygrometers, thermometers, radiation sensors, It may include at least one of a thermal sensor, a gas sensor, etc.) and a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.).

The output unit 150 is for generating an output related to visual, auditory or tactile sense, and includes at least one of a display unit 151 , a sound output unit 152 , a haptip module 153 , and an optical output unit 154 . can do. The display unit 151 may implement a touch screen by forming a layer structure with the touch sensor or being formed integrally with the touch sensor. Such a touch screen may function as the user input unit 123 providing an input interface between the electronic device 100 and the user, and may provide an output interface between the electronic device 100 and the user.

The interface unit 160 serves as a passage with various types of external devices connected to the electronic device 100 . Such an interface unit 160, a wired / wireless headset port (port), an external charger port (port), a wired / wireless data port (port), a memory card (memory card) port, for connecting a device equipped with an identification module It may include at least one of a port, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, and an earphone port. In response to the connection of the external device to the interface unit 160 , the electronic device 100 may perform appropriate control related to the connected external device.

In addition, the memory 170 stores data supporting various functions of the electronic device 100 . The memory 170 may store a plurality of application programs (or applications) driven in the electronic device 100 , data for operation of the electronic device 100 , and commands. At least some of these application programs may be downloaded from an external server through wireless communication. In addition, at least some of these application programs may exist on the electronic device 100 from the time of shipment for basic functions (eg, incoming calls, outgoing functions, message reception, and outgoing functions) of the electronic device 100 . Meanwhile, the application program may be stored in the memory 170 , installed on the electronic device 100 , and driven to perform an operation (or function) of the audio analysis information collecting device by the controller 180 .

In addition to the operation related to the application program, the controller 180 generally controls the overall operation of the electronic device 100 . The controller 180 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170 .

Also, the controller 180 may control at least some of the components discussed with reference to FIG. 1 in order to drive an application program stored in the memory 170 . Furthermore, in order to drive the application program, the controller 180 may operate at least two or more of the components included in the electronic device 100 in combination with each other.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power to each component included in the electronic device 100 . The power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the respective components may operate in cooperation with each other to implement the operation, control, or control method of the apparatus for collecting audio analysis information according to various embodiments to be described below. In addition, the operation, control, or control method of the audio analysis information collecting apparatus may be implemented on the audio analysis information collecting apparatus by driving at least one application program stored in the memory 170 .

Hereinafter, before looking at various embodiments implemented through the electronic device 100 as described above, the above-listed components will be described in more detail with reference to FIG. 1 .

First, referring to the wireless communication unit 110 , the broadcast reception module 111 of the wireless communication unit 110 receives a broadcast signal and/or broadcast related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more of the broadcast reception modules may be provided in the electronic device 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

Mobile communication module 112, the technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV -DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced, etc.) transmits and receives radio signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to (Long Term Evolution-Advanced).

The wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be built-in or external to the electronic device 100 . The wireless Internet module 113 is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

As wireless Internet technology, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), etc., and the wireless Internet module ( 113) transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above.

From the point of view that wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is made through a mobile communication network, the wireless Internet module 113 performs wireless Internet access through the mobile communication network. ) may be understood as a type of the mobile communication module 112 .

Short-range communication module 114 is for short-range communication, Bluetooth (Bluetooth), RFID (Radio Frequency Identification), infrared communication (Infrared Data Association; IrDA), UWB (Ultra Wideband), ZigBee, NFC Short-distance communication may be supported by using at least one of (Near Field Communication), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, and Wireless Universal Serial Bus (USB) technologies. The short-distance communication module 114, between the electronic device 100 and the wireless communication system, between the electronic device 100 and the other electronic device 100, or the electronic device 100 through a wireless local area network (Wireless Area Networks) ) and a network in which another terminal (or external server) is located can support wireless communication. The local area network may be local area networks (Wireless Personal Area Networks).

Here, another terminal is a wearable device capable of exchanging (or interworking) data with the electronic device 100 according to the present invention, for example, a smart watch, a smart glass. , can be a head mounted display (HMD). The short-range communication module 114 may detect (or recognize) a wearable device capable of communicating with the electronic device 100 in the vicinity of the electronic device 100 . Furthermore, when the sensed wearable device is a device authenticated to communicate with the electronic device 100 according to the present invention, the controller 180 transmits at least a portion of data processed by the electronic device 100 to the short-range communication module ( 114) to transmit to the wearable device. Accordingly, the user of the wearable device may use data processed by the electronic device 100 through the wearable device. For example, according to this, when a call is received in the electronic device 100, the user performs a phone call through the wearable device, or when a message is received in the electronic device 100, the user receives the received call through the wearable device. It is possible to check the message.

The location information module 115 is a module for acquiring a location (or current location) of a mobile terminal, and a representative example thereof includes a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, if the audio analysis information collecting device utilizes a GPS module, it may acquire the location of the audio analysis information collecting device using a signal transmitted from a GPS satellite. As another example, when the audio analysis information collecting device utilizes the Wi-Fi module, the location of the audio analysis information collecting device is based on the Wi-Fi module and the information of the wireless AP (Wireless Access Point) that transmits or receives the wireless signal. can be obtained. If necessary, the location information module 115 may perform any function of other modules of the wireless communication unit 110 in order to obtain data on the location of the audio analysis information collecting device as a substitute or additionally. The location information module 115 is a module used to obtain the location (or current location) of the audio analysis information collecting device, and is not limited to a module that directly calculates or obtains the location of the audio analysis information collecting device.

Next, the input unit 120 is for input of image information (or signal), audio information (or signal), data, or information input from a user. For input of image information, the electronic device 100 is one Alternatively, a plurality of cameras 121 may be provided. The camera 121 processes an image frame such as a still image or a moving image obtained by an image sensor in a video call mode or a shooting mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170 . On the other hand, the plurality of cameras 121 provided in the electronic device 100 may be arranged to form a matrix structure, and through the cameras 121 forming the matrix structure as described above, various angles or focal points are applied to the electronic device 100 . A plurality of image information having a plurality of images may be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes an external sound signal as electrical voice data. The processed voice data may be utilized in various ways according to a function (or a running application program) being performed by the electronic device 100 . Meanwhile, various noise removal algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone 122 .

The user input unit 123 is for receiving information from a user, and when information is input through the user input unit 123 , the controller 180 may control the operation of the electronic device 100 to correspond to the input information. . The user input unit 123 is a mechanical input means (or a mechanical key, for example, a button located on the front or back or sides of the electronic device 100 , a dome switch, a jog wheel, and a jog). switch, etc.) and a touch input means. As an example, the touch input means consists of a virtual key, a soft key, or a visual key displayed on the touch screen through software processing, or a part other than the touch screen. It may be made of a touch key (touch key) disposed on the. On the other hand, the virtual key or the visual key, it is possible to be displayed on the touch screen while having various forms, for example, graphic (graphic), text (text), icon (icon), video (video) or these can be made by a combination of

Meanwhile, the sensing unit 140 senses at least one of information in the collection device, surrounding environment information surrounding the collection device, and user information, and generates a sensing signal corresponding thereto. The controller 180 may control the driving or operation of the electronic device 100 or perform data processing, functions, or operations related to an application program installed in the electronic device 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor 141 may be disposed in an inner region of the collection device covered by the touch screen as described above or in the vicinity of the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, an infrared proximity sensor, and the like. When the touch screen is capacitive, the proximity sensor 141 may be configured to detect the proximity of an object having conductivity by a change in an electric field according to the proximity of the object. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of description, the act of approaching an object on the touch screen without contacting it so that the object is recognized that it is located on the touch screen is called “proximity touch”, and the touch The act of actually touching an object on the screen is called "contact touch". The position where the object is touched in proximity on the touch screen means a position where the object is perpendicular to the touch screen when the object is touched in proximity. The proximity sensor 141 may detect a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.) have. Meanwhile, the controller 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and furthermore, provides visual information corresponding to the processed data. It can be printed on the touch screen. Furthermore, the controller 180 may control the electronic device 100 to process different operations or data (or information) according to whether a touch to the same point on the touch screen is a proximity touch or a contact touch. .

The touch sensor receives a touch (or touch input) applied to the touch screen (or the display unit 151) using at least one of various touch methods such as a resistive film method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method. detect

As an example, the touch sensor may be configured to convert a change in pressure applied to a specific part of the touch screen or a change in capacitance occurring in a specific part of the touch screen into an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, capacitance at the time of touch, and the like where a touch object applying a touch on the touch screen is touched on the touch sensor. Here, the touch object is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen, a stylus pen, or a pointer.

As such, when there is a touch input to the touch sensor, a signal(s) corresponding thereto is sent to the touch controller. The touch controller processes the signal(s) and then transmits corresponding data to the controller 180 . Accordingly, the controller 180 can know which area of the display unit 151 has been touched, and the like. Here, the touch controller may be a component separate from the controller 180 , or may be the controller 180 itself.

Meanwhile, the controller 180 may perform different controls or may perform the same control according to the type of the touch object that touches the touch screen (or a touch key provided in addition to the touch screen). Whether to perform different control or the same control according to the type of the touch object may be determined according to the current operating state of the electronic device 100 or a running application program.

On the other hand, the above-described touch sensor and proximity sensor independently or in combination, a short (or tap) touch on the touch screen (short touch), long touch (long touch), multi touch (multi touch), drag touch (drag touch) ), flick touch, pinch-in touch, pinch-out touch, swype touch, hovering touch, etc. It can sense touch.

The ultrasonic sensor may recognize location information of a sensing target by using ultrasonic waves. Meanwhile, the controller 180 may calculate the position of the wave source based on information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source may be calculated using the property that light is much faster than ultrasonic waves, that is, the time at which light arrives at the optical sensor is much faster than the time at which ultrasonic waves reach the ultrasonic sensor. More specifically, the position of the wave source may be calculated by using a time difference from the time that the ultrasonic wave arrives using light as a reference signal.

On the other hand, the camera 121 as seen in the configuration of the input unit 120 includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to detect a touch of a sensing target for a 3D stereoscopic image. The photo sensor may be stacked on the display device, and the photo sensor is configured to scan the motion of the sensing target close to the touch screen. More specifically, the photo sensor mounts photo diodes and transistors (TRs) in rows/columns and scans the contents placed on the photo sensor using electrical signals that change according to the amount of light applied to the photo diodes. That is, the photo sensor calculates the coordinates of the sensing target according to the amount of change in light, and through this, location information of the sensing target can be obtained.

The display unit 151 displays (outputs) information processed by the electronic device 100 . For example, the display unit 151 may display execution screen information of an application program driven in the electronic device 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information. .

Also, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

A three-dimensional display method such as a stereoscopic method (glasses method), an auto stereoscopic method (glassesless method), or a projection method (holographic method) may be applied to the stereoscopic display unit.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output unit 152 also outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, etc.) performed by the electronic device 100 . The sound output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A representative example of the tactile effect generated by the haptic module 153 may be vibration. The intensity and pattern of vibration generated by the haptic module 153 may be controlled by a user's selection or setting of the controller. For example, the haptic module 153 may synthesize and output different vibrations or output them sequentially.

In addition to vibration, the haptic module 153 is configured to respond to stimuli such as a pin arrangement that moves vertically with respect to the contact skin surface, a jet or suction force of air through a nozzle or an inlet, a brush against the skin surface, contact of an electrode, an electrostatic force, etc. Various tactile effects can be generated, such as an effect caused by heat absorption and an effect by reproducing a feeling of cold and heat using an element capable of absorbing heat or generating heat.

The haptic module 153 may not only deliver a tactile effect through direct contact, but may also be implemented so that the user can feel the tactile effect through a muscle sense such as a finger or arm. Two or more haptic modules 153 may be provided according to the configuration of the electronic device 100 .

The light output unit 154 outputs a signal for notifying the occurrence of an event by using the light of the light source of the electronic device 100 . Examples of the event generated in the electronic device 100 may be message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output by the light output unit 154 is implemented as the collection device emits light of a single color or a plurality of colors toward the front or rear. The signal output may be terminated when the collection device detects the user's event confirmation.

The interface unit 160 serves as a passage with all external devices connected to the electronic device 100 . The interface unit 160 receives data from an external device, receives power and transmits it to each component inside the electronic device 100 , or allows data inside the electronic device 100 to be transmitted to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module (port), audio I/O (Input/Output) port (port), video I/O (Input/Output) port (port), earphone port (port), etc. may be included in the interface unit 160 .

On the other hand, the identification module is a chip that stores various information for authenticating the use authority of the electronic device 100, a user identification module (UIM), a subscriber identity module (subscriber identity module; SIM), universal user authentication It may include a universal subscriber identity module (USIM) and the like. A device equipped with an identification module (hereinafter, 'identification device') may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal 100 through the interface unit 160 .

In addition, the interface unit 160 is a passage through which power from the cradle is supplied to the electronic device 100 when the electronic device 100 is connected to an external cradle, or is input from the cradle by a user. It may be a path through which various command signals are transmitted to the electronic device 100 . Various command signals or the power input from the cradle may be operated as signals for recognizing that the electronic device 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 and may temporarily store input/output data (eg, a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibrations and sounds of various patterns output when a touch input is performed on the touch screen.

Memory 170 is a flash memory type (flash memory type), a hard disk type (hard disk type), an SSD type (Solid State Disk type), an SDD type (Silicon Disk Drive type), a multimedia card micro type ), card-type memory (such as SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EEPROM) -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The electronic device 100 may be operated in relation to a web storage that performs a storage function of the memory 170 on the Internet.

Meanwhile, as described above, the controller 180 controls the operation related to the application program and the general operation of the electronic device 100 . For example, if the state of the collection device satisfies a set condition, the controller 180 may execute or release a lock state that restricts input of a user's control command to applications.

In addition, the controller 180 performs control and processing related to voice calls, data communication, video calls, etc., or performs pattern recognition processing capable of recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively. can Furthermore, in order to implement various embodiments described below on the electronic device 100 according to the present invention, the controller 180 may control any one or a plurality of the components described above by combining them.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power required for operation of each component. The power supply unit 190 includes a battery, and the battery may be a built-in battery configured to be rechargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of the interface 160 to which an external charger that supplies power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 using one or more of an inductive coupling method based on a magnetic induction phenomenon or a resonance coupling method based on an electromagnetic resonance phenomenon from an external wireless power transmitter. power can be transmitted.

Meanwhile, various embodiments below may be implemented in a computer-readable recording medium using, for example, software, hardware, or a combination thereof.

The location information module 115 provided in the collecting device is for detecting, calculating, or identifying the location of the collecting device, and representative examples thereof may include a GPS (Global Position System) module and a WiFi (Wireless Fidelity) module. If necessary, the location information module 115 may perform any function of the other modules of the wireless communication unit 110 in order to obtain data on the location of the collection device as a substitute or additionally.

The GPS module 115 calculates distance information and accurate time information from three or more satellites, and then applies trigonometry to the calculated information to accurately calculate three-dimensional current location information according to latitude, longitude, and altitude. can do. Currently, a method of calculating position and time information using three satellites and correcting errors in the calculated position and time information using another one satellite is widely used. In addition, the GPS module 115 may calculate the speed information by continuously calculating the current location in real time. However, it is difficult to accurately measure the location of the collecting device using the GPS module in the shaded area of the satellite signal, such as indoors. Accordingly, in order to compensate for positioning using the GPS method, a WiFi Positioning System (WPS) may be utilized.

A Wi-Fi positioning system (WPS: WiFi Positioning System) uses a WiFi module provided in the electronic device 100 and a wireless AP (Wireless Access Point) that transmits or receives a wireless signal with the WiFi module, the electronic device 100 As a technology for tracking the location of a wireless network, it refers to a wireless local area network (WLAN)-based positioning technology using WiFi.

The Wi-Fi location tracking system may include a Wi-Fi positioning server, the electronic device 100, a wireless AP connected to the electronic device 100, and a database in which arbitrary wireless AP information is stored.

The electronic device 100 connected to the wireless AP may transmit a location information request message to the Wi-Fi location location server.

The Wi-Fi positioning server extracts information on the wireless AP connected to the electronic device 100 based on the location information request message (or signal) of the electronic device 100 . The information on the wireless AP connected to the electronic device 100 may be transmitted to the Wi-Fi positioning server through the electronic device 100 or may be transmitted from the wireless AP to the Wi-Fi positioning server.

Based on the location information request message of the electronic device 100, the extracted wireless AP information is MAC Address, Service Set IDentification (SSID), Received Signal Strength Indicator (RSSI), Reference Signal Received Power (RSRP), RSRQ ( Reference Signal Received Quality), channel information, Privacy, Network Type, signal strength, and noise strength may be at least one.

As described above, the Wi-Fi positioning server may receive information on the wireless AP connected to the electronic device 100 and extract wireless AP information corresponding to the wireless AP to which the collection device is connected from a pre-established database. At this time, the information of arbitrary wireless APs stored in the database includes MAC Address, SSID, channel information, Privacy, Network Type, latitude and longitude coordinates of the wireless AP, the name of the building where the wireless AP is located, the number of floors, and detailed indoor location information (GPS coordinates). available), the AP owner's address, phone number, and the like. In this case, in order to remove a mobile AP or a wireless AP provided using an illegal MAC address in the positioning process, the Wi-Fi positioning server may extract only a predetermined number of wireless AP information in order of increasing RSSI.

Thereafter, the Wi-Fi positioning server may extract (or analyze) the location information of the electronic device 100 using at least one wireless AP information extracted from the database. By comparing the included information and the received wireless AP information, the location information of the electronic device 100 is extracted (or analyzed).

As a method for extracting (or analyzing) the location information of the electronic device 100, a Cell-ID method, a fingerprint method, a triangulation method, a landmark method, etc. may be utilized.

The Cell-ID method is a method of determining the location of the wireless AP having the strongest signal strength among the surrounding wireless AP information collected by the collecting device as the location of the collecting device. It has the advantages of being simple to implement, not requiring additional cost, and being able to obtain location information quickly, but has a disadvantage in that positioning accuracy is lowered if the installed density of the wireless AP is low.

The fingerprint method is a method of collecting signal strength information by selecting a reference position in a service area, and estimating the position through signal strength information transmitted from a collection device based on the collected information. In order to use the fingerprint method, it is necessary to database the propagation characteristics in advance.

The triangulation method is a method of calculating the position of the collecting device based on the coordinates of at least three wireless APs and the distance between the collecting device. In order to measure the distance between the acquisition device and the wireless AP, the signal strength is converted into distance information, or the time at which the wireless signal is transmitted (Time of Arrival, ToA), the time difference of the signal is transmitted (Time Difference of Arrival, TDoA) , the angle at which the signal is transmitted (Angle of Arrival, AoA), etc. may be used.

The landmark method is a method of measuring the location of a collection device using a landmark transmitter that knows the location.

In addition to the enumerated methods, various algorithms may be utilized as a method for extracting (or analyzing) the location information of the collection device.

The location information of the electronic device 100 extracted in this way is transmitted to the electronic device 100 through the Wi-Fi positioning server, so that the electronic device 100 may obtain the location information.

The electronic device 100 may acquire location information by being connected to at least one wireless AP. In this case, the number of wireless APs required to acquire the location information of the electronic device 100 may be variously changed according to the wireless communication environment in which the electronic device 100 is located.

1, the collection device according to the present invention includes Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near). Field Communication), a short-distance communication technology such as Wireless USB (Wireless Universal Serial Bus) may be applied.

Among them, the NFC module provided in the collection device supports contactless short-range wireless communication between terminals at a distance of about 10 cm. The NFC module may operate in any one of a card mode, a reader mode, and a P2P mode. In order for the NFC module to operate in the card mode, the electronic device 100 may further include a security module for storing card information. Here, the security module may be a physical medium such as a Universal Integrated Circuit Card (UICC) (eg, a Subscriber Identification Module (SIM) or a Universal SIM (USIM)), a Secure micro SD, and a sticker, and a logical medium embedded in the collection device ( For example, it may be an embedded secure element (SE). Data exchange based on SWP (Single Wire Protocol) may be performed between the NFC module and the security module.

When the NFC module is operated in card mode, the collecting device can transmit the stored card information to the outside like a traditional IC card. Specifically, when a collection device for storing card information of a payment card such as a credit card or a bus card is brought close to a payment machine, a mobile short-distance payment can be processed, and the collection device for storing card information of the access card is approved for access Approaching the flag, the access approval process can begin. Cards such as credit cards, transportation cards, and access cards are loaded in the security module in the form of an applet, and the security module may store card information on the loaded card. Here, the card information of the payment card may be at least one of a card number, balance, and usage history, and the card information of the access card includes at least one of a user's name, number (eg, user's student number or company number), and access history. can be one

When the NFC module is operated in the reader mode, the collection device may read data from an external tag. In this case, the data that the collection device receives from the tag may be coded in a data exchange format (NFC Data Exchange Format) determined by the NFC forum. In addition, the NFC forum defines four record types. Specifically, the NFC forum defines four Record Type Definitions (RTDs) such as Smart Poster, Text, Uniform Resource Identifier (URI), and General Control. When the data received from the tag is a smart poster type, the controller executes a browser (eg, an Internet browser), and when the data received from the tag is a text type, the controller executes a text viewer. When the data received from the tag is a URI type, the controller executes a browser or makes a call, and when the data received from the tag is a general control type, an appropriate operation may be executed according to the control content.

When the NFC module is operated in a peer-to-peer (P2P) mode, the collecting device may perform P2P communication with another collecting device. In this case, a Logical Link Control Protocol (LLCP) may be applied to P2P communication. For P2P communication, a connection may be created between a collection device and another collection device. In this case, the generated connection may be divided into a connectionless mode in which one packet is exchanged and terminated, and a connection-oriented mode in which packets are continuously exchanged. Through P2P communication, data such as electronic business cards, contact information, digital photos, URLs, and setup parameters for Bluetooth and Wi-Fi connections may be exchanged. However, since the available distance of NFC communication is short, the P2P mode may be effectively utilized for exchanging small-sized data.

Hereinafter, embodiments related to a control method that can be implemented in the collecting device configured as described above will be described with reference to the accompanying drawings. It is apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit and essential characteristics of the present invention.

Meanwhile, the controller 180 of the electronic device 100 according to the present invention may be equipped with an artificial intelligence engine for performing audio recognition.

Specifically, the artificial intelligence engine mounted on the controller 180 may be configured to identify which audio event the non-verbal audio input corresponds to.

In an example, the artificial intelligence engine may perform self-learning using training data labeled with a whistle sound and training data labeled with a non-whistle sound. That is, the artificial intelligence engine builds an artificial neural network, receives training data labeled whether a whistle is heard, and uses the input training data to perform machine learning or deep learning to identify whether an unknown input audio signal is a whistle. running can be done.

In another example, when an audio signal is input, the AI engine may identify a label of the corresponding audio signal, and determine at least one of the height, intensity, and duration of at least a portion of the audio signal identified by the specific label. can be discerned.

Hereinafter, a method of controlling the above-described electronic device 100 with an audio signal is described with reference to FIG. 2 .

First, as shown in FIG. 2 , the sensing unit 140 or the microphone of the electronic device 100 may receive an audio signal from a user ( S201 ).

The controller 180 may determine whether the inputted audio signal is a whistle sound (S202).

In more detail, the controller 180 may determine whether the input audio signal is a whistle sound by using a pre-learned artificial intelligence model. In this case, the artificial intelligence model may be configured to identify information related to the subject that generated the input audio signal.

For example, the artificial intelligence model may be learned by training data labeled for each of the plurality of classes, and may calculate a probability corresponding to each of the plurality of classes with respect to the input audio. In this case, the plurality of classes may include a whistle sound.

In another example, the AI model may be trained by training data labeled as whistling and training data labeled as non-whistle to determine whether the input audio is a whistle.

In addition, if it is determined that the audio signal is a whistle sound, the controller 180 may analyze the characteristics of the whistle sound ( S203 ).

Here, the characteristic of the whistle may be related to at least one of a height, a size, and a duration of the whistle.

Specifically, the controller 180 may detect information related to a change in the whistling sound. That is, the controller 180 may detect whether the whistle changes to a high pitch in a low-pitched tone, or, conversely, detects whether the whistle changes into a low-pitched tone in a high-pitched tone.

Also, the controller 180 may detect whether the whistle sound is a sound of a specific height. Furthermore, the controller 180 may also detect whether the whistling sound corresponds to a specific melody.

In one embodiment, in performing the step of analyzing the whistle sound (S203), the controller 180 determines at least one of the height, intensity, and duration of the whistle sound, and the electronic device ( 100) may perform a process of generating an operation command corresponding to the operation command.

In more detail, in performing the step S203 of analyzing the whistle, the controller 180 detects a change in the height of the whistle and generates an operation command corresponding to the electronic device based on the detected height change. process can be performed.

Thereafter, the controller 180 may control the operation of the electronic device 100 using the analysis result ( S204 ).

Specifically, in performing the step S204 of controlling the operation of the electronic device 100 , the controller 180 controls the driving unit to perform the first operation of the electronic device when it is sensed that the height of the whistle is increased. a process of controlling the driving unit to perform a second operation of the electronic device when it is sensed that the height of the whistle is reduced 3 A process of controlling the driving unit to perform the operation may be performed.

For example, when the electronic device 100 is a refrigerator, when it is analyzed that the whistling sound is changed from a low sound to a high sound, the controller 180 may control the door driving unit to open the door of the refrigerator.

In one embodiment, the audio-based electronic device control method proposed by the present invention may be applied to a refrigerator.

In this case, the refrigerator may include a microphone for receiving an audio signal, a control unit that determines whether the input audio signal is a whistle, and a driving unit that operates according to a whistle sound. Hereinafter, for convenience of description, it is defined that the driving unit includes a door opening and closing device for opening and closing the door of the refrigerator, and a display for outputting information related to the operation of the refrigerator. In order to open or close the door of the refrigerator, the door opening and closing device may include a motor for providing a driving force to a hinge portion of the door, and a guide for limiting the direction of the driving force.

For example, when it is determined that the whistle sound changes from a low sound to a high sound, the controller may control the door opening and closing device to open the door of the refrigerator. Conversely, the controller may control the door opening and closing device to close the door when the whistling sound changes from a high tone to a low tone.

In another example, when the pitch of the whistle is maintained, the controller may change the output screen of the display of the refrigerator.

In another embodiment, the audio-based electronic device control method proposed by the present invention may be applied to IoT devices such as artificial intelligence speakers.

In general, an artificial intelligence speaker performs communication with a plurality of devices existing in an installed place, and serves to control operations of the plurality of devices. The artificial intelligence speaker may include a microphone for receiving an audio signal, a controller for processing the audio signal, a memory for storing information related to the audio signal, and a communication unit for transmitting an operation command to the plurality of devices.

For example, when the whistle sound changes from a low tone to a high tone, the controller of the artificial intelligence speaker may generate an operation command corresponding to the first device paired with the artificial intelligence speaker. Also, when the whistle sound changes from a high tone to a low tone, the controller of the artificial intelligence speaker may generate an operation command corresponding to the second device paired with the artificial intelligence speaker. In this way, the control unit of the artificial intelligence speaker may generate operation commands for different devices according to changes in the whistling sound, and may control the operation of each device based on the generated operation commands.

In another example, the controller of the artificial intelligence speaker may generate different operation commands for one device according to a change in a whistle sound.

Hereinafter, components of a refrigerator to which the electronic device control method according to the present invention is applied will be described with reference to FIG. 3 .

3A is a control block diagram of an embodiment according to the present invention.

The present invention may include a control unit 300 that manages a single photo taken by the camera 70 by dividing it into a plurality of images. In this case, the image may mean a part of the photo processed or corrected by the controller 300 .

The controller 300 may command the camera 70 to take a picture, and may receive a picture taken by the camera 70 .

In addition, the control unit 300 may provide a portion of the divided image to the display 14 to provide the user with the latest information about the food stored in the storage compartment 22 . In this case, the display 14 may be installed on the front of the refrigerator, or it may be a separate device separated from the refrigerator. That is, it is possible for the user to receive the image related to the storage room 22 through an external communication terminal such as a mobile phone and obtain information.

In this case, the control unit 300 may divide the picture taken by the camera 70 for each area, classify it into a plurality of independent images, and provide it to the display 14 . In this case, the photo selected by the control unit 100 may include information about the recent storage room state in which the user last accessed the storage room 22 and took out food from the storage room or put food in it.

An embodiment of the present invention may include a door switch 310 capable of detecting whether the door 20 opens or closes the storage compartment 22 . At this time, the door switch 310 is provided in the outer case 10 , and when the door 20 comes into contact with the outer case 10 , it is sensed that the door 20 seals the storage chamber 22 . can do. Also, when the door 20 does not contact the outer case 10 , it may be detected that the door 20 opens the storage compartment 22 .

In addition, an embodiment of the present invention may include a door sensor 320 that detects a rotation angle of the door 20 . In this case, the door sensor 320 may detect a rotation direction of the door 20 and a rotation angle of the door 20 . For example, when the door 20 is rotated by a certain angle or more, a change occurs in the door sensor 320, and the door sensor 320 can detect that the door 20 is rotated by a certain angle or more. have. In addition, when the door 20 is rotated in a specific direction, it is possible to sense the rotation direction of the door 20 by detecting a change in a pulse generated according to the direction. Of course, the door sensor 320 may be modified in various forms other than the above-described method.

In addition, the door sensor 320 includes a light emitting unit and a light receiving unit to determine whether the light irradiated from the light emitting unit is transmitted to the light receiving unit, and thus it is also possible to detect the rotation angle of the door 20 .

In particular, the door sensor 320 may be driven only when the door switch 310 determines that the door 20 has opened the storage compartment 22 .

Meanwhile, when the door sensor 320 determines that the door 20 is rotated at a specific angle, it is also possible to take a picture with the camera 70 . In this case, even if the door switch 310 detects that the door 20 is opened, the camera 70 does not immediately start taking a picture, but takes a picture when the door sensor 320 reaches a specific angle. will be filmed

When the camera 70 starts shooting, it is also possible to continuously take pictures at predetermined time intervals. Of course, if the door sensor 320 determines that the door 20 is opened by a certain angle, the camera 70 may continue to take pictures until a command to stop taking pictures is provided, instead of a single picture. do.

The control unit 300 may include a drawer detection unit 330 that detects whether the drawer 50 is drawn out or retracted.

In this case, the drawer detection unit 330 may mean a portion in which the picture taken by the camera 70 reads an image in the control unit 300 . The drawer sensing unit 330 may sense the movement of the drawer 50 as well as the movement direction of the drawer 50 .

In particular, the drawer sensing unit 330 may be implemented by software. The drawer detection unit 330 may detect the position of the drawer 50 by using information taken from a picture taken by the camera 70 .

The drawer 50 is provided with a recognition mark called a marker, and whether the drawer 50 has entered a specific section through the photograph in which the marker is taken, whether it is stopped if it has entered, or whether the movement direction is changed after stopping information can be found about The markers will be described in detail later.

Since the marker is marked on the drawer, it has the same movement as the drawer 50 . Accordingly, various information about the movement of the drawer 50 can be obtained by using the information of the picture taken by the camera 70 without a separate sensor for detecting the movement of the drawer.

The photo taken by the camera 70 may be stored in the storage unit 18 . In this case, the storage unit 18 may be provided in the refrigerator or may be provided in another device separate from the refrigerator. In this case, the storage unit 18 may be disposed in the refrigerator together with the display 14, or may be provided in a device other than the refrigerator together with the display 14, a server connected to the refrigerator and a network, or a terminal connected to the refrigerator and the network. have.

In this case, the storage unit 18 may not store all the photos taken by the camera 70 . For example, when a command to select a specific photo does not occur in the control unit 300 , the storage unit 18 may delete a previously captured photo and store the currently captured photo (First In). , First Out). The storage unit 18 stores only some of the photos taken by the camera 70, thereby reducing the storage capacity.

3B is a control block diagram of a modified form of an embodiment according to the present invention.

In FIG. 3(b), unlike FIG. 3(a), the drawer detection unit 330 does not use the information obtained from the picture taken by the camera 70, but separately moves the drawer 50. can detect

For example, the drawer sensing unit 330 may have the same shape as a plurality of Hall sensors. A plurality of Hall sensors may be installed in the movement path of the drawer 50 . When the drawer 50 is moved, it is possible to detect a change in a plurality of Hall sensors, respectively, to detect a position and a movement direction of the drawer 50 .

By detecting the movement of the drawer by the drawer detection unit 330, when the camera 70 takes a picture, it is possible to grasp the time point at which the inside of the drawer is included in the picture taken by the camera 70. . That is, the drawer detection unit 330 may detect a point in time at which information about an item stored in the drawer can be expressed in the picture taken by the camera.

The drawer sensing unit 330 may detect the movement of the drawer, and may detect a point in time when the user determines that the drawer is in the latest state after using the drawer. That is, the drawer detection unit 330 may detect a point in time including the latest information that the user has finished using the drawer in the picture taken by the camera. That is, it is possible to indirectly grasp the completion time of the arrival and departure of goods or the end time of the update of the goods.

4 is a conceptual diagram illustrating a method of controlling an operation of a refrigerator using a whistle sound. Referring to FIG. 4 , the user may control the operation of the refrigerator by blowing a whistle. In this case, the user may blow a whistle in various ways to control the operation of the refrigerator, and the controller analyzes the manner in which the user blows a whistle, and the refrigerator performs an operation of a preset pattern corresponding to the analysis result.

Various methods of blowing the whistle include a method of varying the pitch of the whistle, a method of varying the length of the whistle, a method of varying the strength of the whistle, a method of blowing a specific melody, and the like. In detail, the method of changing the pitch may include a method of increasing the height of the sound, decreasing the height of the sound, or maintaining the height of the sound constant, and the method of varying the length of the whistle is short. It may include a method of repeatedly making a long whistle, or a method of making a short or long whistle a predetermined number of times. and a method of blowing a specific melody may include a method of blowing a whistle according to the pitch of a note corresponding to a predetermined melody. Also, the user can whistle in more ways by combining the above methods.

The operation of the refrigerator corresponding to the method by which the user blows a whistle may include an operation of opening or closing a door of the refrigerator, changing an external display output screen, temperature control and humidity control in the refrigerator or freezer compartment of the refrigerator, humidity control, ice freezing, etc. . In detail, the number of doors may be different depending on the structure of the refrigerator, and may include an operation to open or close all the doors or an operation to open or close each door. It may include the operation of changing the displayed information to other information that you want to see or calling specific information, and may include the operation of raising or lowering the temperature or humidity of the freezer or refrigerating compartment of the refrigerator step by step, or changing the operation to a specific level, In the case of a refrigerator having a built-in ice maker, an operation of freezing the ice may be included. In addition, in addition to the above operation, additional functions provided according to the refrigerator may also be operated.

As described above, there may be various methods for the user to whistle, and there may be various operations of the refrigerator corresponding thereto. The various methods of blowing the whistle and the operation of the refrigerator may be matched to perform a preset operation. In addition, even if it is not a preset operation, a method of matching an operation of the refrigerator desired by the user to a specific whistling method and additionally setting an operation of the refrigerator corresponding to the corresponding whistling method may be included. A method in which a plurality of operations are sequentially performed may be included rather than corresponding.

AI speaker 500 is a TV, projector, mobile phone, smartphone, desktop computer, notebook computer, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB) ), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, a digital signage, a robot, a vehicle, etc.

Referring to FIG. 5 , the AI speaker 500 includes a communication unit 510 , an input unit 520 , a learning processor 530 , a sensing unit 540 , an output unit 550 , a memory 570 and a processor 580 , etc. may include

The communication unit 510 may transmit/receive data to and from external devices such as other AI devices 500a to 500e or an AI server (not shown) using wired/wireless communication technology. For example, the communication unit 510 may transmit and receive sensor information, a user input, a learning model, a control signal, and the like with external devices.

In this case, the communication technology used by the communication unit 510 includes Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN), and Wireless-Fidelity (Wi-Fi). ), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), and the like.

The input unit 520 may acquire various types of data.

In this case, the input unit 520 may include a camera for inputting an image signal, a microphone for receiving an audio signal, a user input unit for receiving information from a user, and the like. Here, by treating the camera or the microphone as a sensor, a signal obtained from the camera or the microphone may be referred to as sensing data or sensor information.

The input unit 520 may acquire training data for model training and input data to be used when acquiring an output using the training model. The input unit 520 may acquire raw input data, and in this case, the processor 580 or the learning processor 530 may extract an input feature by preprocessing the input data.

The learning processor 530 may train a model composed of an artificial neural network by using the training data. Here, the learned artificial neural network may be referred to as a learning model. The learning model may be used to infer a result value with respect to new input data other than the training data, and the inferred value may be used as a basis for a decision to perform a certain operation.

In this case, the learning processor 530 may perform AI processing together with a running processor (not shown) of the AI server.

In this case, the learning processor 530 may include a memory integrated or implemented in the AI speaker 500 . Alternatively, the learning processor 530 may be implemented using the memory 570 , an external memory directly coupled to the AI speaker 500 , or a memory maintained in an external device.

The sensing unit 540 may acquire at least one of internal information of the AI speaker 500 , information about the surrounding environment of the AI speaker 500 , and user information by using various sensors.

At this time, sensors included in the sensing unit 540 include a proximity sensor, an illuminance sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, and a lidar. , radar, etc.

The output unit 550 may generate an output related to sight, hearing, or touch.

In this case, the output unit 550 may include a display unit that outputs visual information, a speaker that outputs auditory information, and a haptic module that outputs tactile information.

The memory 570 may store data supporting various functions of the AI speaker 500 . For example, the memory 570 may store input data obtained from the input unit 520 , learning data, a learning model, a learning history, and the like.

The processor 580 may determine at least one executable operation of the AI speaker 500 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. In addition, the processor 580 may control the components of the AI speaker 500 to perform the determined operation.

To this end, the processor 580 may request, search, receive, or utilize the data of the learning processor 530 or the memory 570, and perform a predicted operation or an operation determined to be desirable among the at least one executable operation. It is possible to control the components of the AI speaker 500 to execute.

In this case, when the connection of the external device is required to perform the determined operation, the processor 580 may generate a control signal for controlling the corresponding external device and transmit the generated control signal to the corresponding external device.

The processor 580 may obtain intention information with respect to a user input, and determine a user's requirement based on the obtained intention information.

In this case, the processor 580 uses at least one of a speech to text (STT) engine for converting a voice input into a character string or a natural language processing (NLP) engine for obtaining intention information of a natural language. Intention information corresponding to the input may be obtained.

In this case, at least one of the STT engine and the NLP engine may be configured as an artificial neural network, at least a part of which is learned according to a machine learning algorithm. And, at least one or more of the STT engine or the NLP engine may be learned by the learning processor 530, learned by the learning processor (not shown) of the AI server, or learned by distributed processing thereof. have.

The processor 580 collects history information including the user's feedback on the operation contents or operation of the AI speaker 500 and stores it in the memory 570 or the learning processor 530, or to an external device such as an AI server. can be transmitted The collected historical information may be used to update the learning model.

The processor 580 may control at least some of the components of the AI speaker 500 in order to drive an application program stored in the memory 570 . Furthermore, the processor 580 may operate by combining two or more of the components included in the AI speaker 500 to drive the application program.

6 is a conceptual diagram illustrating a method of controlling an operation of an AI speaker using a whistle sound. Referring to FIG. 6 , the user can control the operation of the AI speaker by blowing a whistle. At this time, the user can blow a whistle in various ways to control the operation of the AI speaker, and the control unit analyzes how the user blows a whistle, and the AI speaker performs the operation of a preset pattern corresponding to the analysis result.

The operation of the AI speaker corresponding to the method of whistling by the user includes the operation of controlling the power of the AI speaker, the operation of controlling the search function, the operation of controlling the AI speaker when playing music, and the operation of controlling other devices linked to the AI speaker It may include an action to perform, an action to search for a melody, and the like. Specifically, it can include the operation of turning on or off the power of the AI speaker, turning on or off the search function of the corresponding category to search for weather, fine dust, news, etc. It may include an operation to search, and when playing music, control the AI speaker to play, pause, play the next song, play the previous song, repeat one song, random play, set and release the start and end points of section repeat, etc. In order to control other devices linked to the AI speaker, it may include an operation to turn on/off the power of another device and an operation to perform a function provided by the other device, and whistle If there is a function to search for music similar to a melody by expressing a melody with In addition, in addition to the above operation, an additional function provided according to the AI speaker may also be operated.

As above, there may be various methods for the user to whistle, and there may be various operations of the AI speaker corresponding thereto. The various methods of blowing the whistle and the operation of the AI speaker may be matched to perform a preset operation. In addition, even if it is not a preset operation, a method of matching the operation of the AI speaker desired by the user to a specific whistling method and setting the operation of the AI speaker corresponding to the corresponding whistle blowing method may be included. A method in which a plurality of operations are sequentially performed may be included rather than corresponding.

The present invention described above relates to an embodiment, which is only an embodiment, and a person skilled in the art may be able to make various modifications and equivalent other embodiments therefrom. Accordingly, the scope of the present invention is not limited by the above embodiments and the accompanying drawings.

Claims

a microphone for receiving an audio signal from a user;

a control unit for determining whether the audio signal input by the microphone is a whistle sound; and

and a driving unit operating according to the characteristics of the audio signal,

The control unit is

If it is determined that the audio signal is a whistle sound, analyzing the characteristics of the whistle sound,

An audio signal-based electronic device, characterized in that the driving unit operates in a preset pattern according to the analysis result.
According to claim 1,

The control unit is

analyzing the characteristics of the whistle by determining at least one of a height, intensity, and duration of the whistle,

Audio signal-based electronic device, characterized in that generating an operation command corresponding to the driving unit based on the analysis result.
3. The method of claim 2,

The control unit is

An audio signal-based electronic device, characterized in that detecting a change in height of the whistle, and generating an operation command corresponding to the electronic device based on the detected height change.
4. The method of claim 3,

The control unit is

when it is sensed that the height of the whistle is increased, controlling the driving unit to perform a first operation of the electronic device;

When it is sensed that the height of the whistle is reduced, controlling the driving unit to perform a second operation of the electronic device,

and when it is sensed that the height of the whistle is maintained, controlling the driving unit to perform a third operation of the electronic device.
4. The method of claim 3,

The control unit is

and determining whether the whistle sound corresponds to a preset melody, and generating an operation command corresponding to the electronic device based on the determination result.