WO2018128317A1 - Réfrigérateur et son procédé d'affichage d'informations - Google Patents

Réfrigérateur et son procédé d'affichage d'informations Download PDF

Info

Publication number
WO2018128317A1
WO2018128317A1 PCT/KR2017/015548 KR2017015548W WO2018128317A1 WO 2018128317 A1 WO2018128317 A1 WO 2018128317A1 KR 2017015548 W KR2017015548 W KR 2017015548W WO 2018128317 A1 WO2018128317 A1 WO 2018128317A1
Authority
WO
WIPO (PCT)
Prior art keywords
voice
food
display
refrigerator
user
Prior art date
Application number
PCT/KR2017/015548
Other languages
English (en)
Korean (ko)
Inventor
천은진
도영수
이형진
조남국
황도형
Original Assignee
삼성전자주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020170179174A external-priority patent/KR102412202B1/ko
Application filed by 삼성전자주식회사 filed Critical 삼성전자주식회사
Priority to CN201780082150.3A priority Critical patent/CN110168298B/zh
Priority to US16/476,008 priority patent/US11521606B2/en
Priority to EP17890733.3A priority patent/EP3546869A4/fr
Publication of WO2018128317A1 publication Critical patent/WO2018128317A1/fr

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems

Definitions

  • the disclosed invention relates to a refrigerator and a method of displaying information of a refrigerator, and more particularly, to a refrigerator and a method of displaying information of a refrigerator that can communicate with an external device.
  • the present disclosure relates to an artificial intelligence (AI) system and its application for simulating functions such as recognition and judgment of the human brain using a machine learning algorithm.
  • AI artificial intelligence
  • a refrigerator includes a display for displaying a temperature of a storage compartment and an operation mode of the refrigerator.
  • Such a display not only enables a user to easily acquire image information using a graphical user interface, but also allows a user to intuitively input a control command using a touch panel.
  • the recent display performs not only an information display function but also an information input function.
  • recent refrigerators include a communication module for connecting to an external device (eg, a server connected to the Internet).
  • an external device eg, a server connected to the Internet.
  • the refrigerator may access the Internet through a communication module, obtain various information from various servers connected to the Internet, and provide various services based on the obtained information.
  • the refrigerator may provide not only information related to food such as food information and recipe of food through the Internet, but also various services such as internet shopping.
  • the refrigerator provides more various services to the user through the display and the communication module.
  • an artificial intelligence system is a computer system that implements human-level intelligence.
  • an artificial intelligence system is a machine that learns, judges, and becomes smart. The more artificial intelligence systems are used, the better the recognition rate and the more accurate the user's taste will be. Therefore, the existing rule-based smart system is gradually being replaced by artificial intelligence system based on deep learning.
  • Machine learning is an algorithmic technique for classifying and learning features of input data.
  • Element technology is a technology using a machine learning algorithm, and may be composed of technical fields such as linguistic understanding, visual understanding, inference / prediction, knowledge representation, and motion control.
  • Linguistic understanding is a technique for recognizing and applying / processing human language / characters, including natural language processing, machine translation, dialogue system, question and answer, speech recognition / synthesis, and the like.
  • Visual understanding is a technology that recognizes and processes objects as human vision, and includes object recognition, object tracking, image retrieval, person recognition, scene understanding, spatial understanding, and image enhancement.
  • Inference / prediction is a technique for determining, logically inferring, and predicting information. It includes knowledge / probability-based inference, optimization prediction, preference-based planning, and recommendation.
  • Knowledge representation is a technology that automates human experience information into knowledge data, and includes knowledge construction (data generation / classification) and knowledge management (data utilization).
  • Motion control is a technology for controlling autonomous driving of a vehicle, movement of a robot, and the like, and includes motion control (navigation, collision, driving), operation control (action control), and the like.
  • One aspect of the disclosed invention provides a refrigerator capable of receiving a command through voice using voice recognition technology and outputting content in voice.
  • One aspect of the disclosed invention provides a refrigerator capable of providing internet shopping by command via voice.
  • One aspect of the disclosed invention provides a refrigerator capable of identifying a plurality of users and providing content suitable for the identified user.
  • a refrigerator may include a microphone; A display for displaying information according to a voice input through the microphone; And when a first voice including a food name is recognized, a list including food information having a name corresponding to the food name and a label for identifying the food, on the display, and displaying at least one of the labels. And a controller for displaying food information indicated by a label included in the second voice on the display when the second voice to be referred to is recognized.
  • the controller may display the list in the form of a card type user interface on the user interface of the application displayed on the display.
  • the controller may execute an application for providing food information indicated by a label included in the second voice and display the food information provided by the application on a display.
  • the controller may further include a communication unit, and when the first voice is recognized, the controller transmits the first voice data to the server through the communication unit, and transmits information about the list transmitted from the server in the communication unit. Upon receipt, the list can be displayed on the display.
  • the controller may further include a communication unit.
  • the controller transmits the data of the second voice to the server through the communication unit, and receives the food information transmitted from the server in the communication unit.
  • the food information may be displayed on the display.
  • the controller may further include a communication unit, and when the first voice is recognized, the controller transmits the first voice data to the server through the communication unit, and analyzes the analysis information on the first voice transmitted from the server.
  • an instruction according to the analysis information may be transmitted to an application related to performing a task according to the analysis information, so that the application may display the list.
  • the controller may further include a communication unit, and when the second voice is recognized, the controller transmits the data of the second voice to the server through the communication unit, and analyzes the analysis information on the second voice transmitted from the server.
  • an instruction according to the analysis information may be transmitted to an application related to performing work according to the analysis information, so that the application may display the food information.
  • a voice indicating that the list is displayed is output.
  • the food information is displayed. It may further include a speaker for outputting a voice indicating.
  • a refrigerator may include a microphone; A display for displaying information according to a voice input through the microphone; A speaker for outputting voice; And when a voice is input through the microphone, performs a command indicated by the input voice for the application displayed on the display, and if the command indicated by the input voice cannot be performed for the application, to the applications related to the command. And a controller for performing the command with respect to another application according to a predetermined priority.
  • the controller may output a voice indicating that the command cannot be performed to the application through the speaker.
  • the controller may output a voice requesting confirmation of whether to execute the command with respect to another application through a speaker.
  • the controller may execute a command indicated by the voice for an application indicated by the input answer.
  • the controller determines an application to execute the command according to a predetermined priority for applications related to the command, and determines the application to the determined application.
  • the command can be performed for
  • the controller may perform a command indicated by the input voice for an application displayed on the display if the object inputted by the microphone does not include a target to perform the command indicated by the voice.
  • a refrigerator may include a microphone; A display for displaying information according to a voice input through the microphone; A speaker for outputting voice; And when a voice is input through a microphone, when user identification is required in performing a command indicated by the voice, displaying a user registered in advance on the display and outputting a voice through the speaker requesting to select from the displayed users. Controller;
  • the controller may perform a command indicated by the voice in relation to the selected user, and display the result on the display.
  • the controller may display a user mark which is identified in advance on the display and each user.
  • the controller may perform a command indicated by the voice in relation to the user indicated by the mark and display the result on the display.
  • the controller when a voice is input through a microphone, when the voice includes a representation representing a user, the controller performs a command indicated by the voice in relation to a user included in the voice and displays the result on the display. can do.
  • the controller may perform a command indicated by the voice in relation to a user represented by voice data matching the input voice among prestored voice data and display the result on the display.
  • An aspect of the disclosed invention can provide a refrigerator capable of receiving a command through voice using voice recognition technology and outputting content in voice.
  • One aspect of the disclosed invention may provide a refrigerator capable of providing internet shopping with a command via voice.
  • One aspect of the disclosed invention may provide a refrigerator capable of identifying a plurality of users and providing content suitable for the identified user.
  • a refrigerator includes a storage compartment for storing food; A temperature sensor for sensing a temperature inside the storage compartment; Cooling unit for supplying cold air to the storage compartment; A display for displaying information; A microphone for receiving a voice; A display for displaying information; At least one processor electrically connected to the temperature sensing unit, the microphone, and the display; And a memory electrically connected to the at least one processor.
  • the memory may include a food list including food information corresponding to the food name and an identification mark for identifying the food information, when the processor recognizes the first voice including the food name through the microphone. If the second voice indicating the identification mark is recognized on the display and the microphone is recognized, the at least one instruction set to display purchase information of at least one food item corresponding to the identification mark is stored on the display. .
  • the memory acquires the food name by recognizing the first voice using a learning network model trained according to an artificial intelligence algorithm. And at least one instruction set to display the food list including the food information corresponding to the food name on the display.
  • the learning network model may be trained using a plurality of voices and a plurality of words corresponding to each of the plurality of voices.
  • the memory may be configured to recognize the food name and the food by recognizing the first voice using a learning network model trained according to an artificial intelligence algorithm when the first voice including the food name is input through the microphone.
  • Obtain at least one instruction configured to acquire user information uttering a first voice and display the food list corresponding to the food name on the display, the food list including the food information related to the user information.
  • the learning network model may be trained using a plurality of voices and user information corresponding to each of the plurality of voices.
  • the processor identifies whether the food associated with the food name in the storage compartment exists in the storage compartment, and if the food is present in the storage compartment, the food is the At least one instruction set to display on the display information indicating what is present in the storage compartment may be stored.
  • the memory may be further configured to identify whether or not a food associated with the food name exists in the storage compartment when the processor recognizes the first voice, and if the food does not exist in the storage compartment, the processor identifies the food name.
  • At least one instruction set to display the food list including the corresponding food information on the display may be stored.
  • the memory may store at least one instruction configured to display the food list on a user interface of an application displayed on the display in a form of a card type user interface when the first voice is recognized. Can be.
  • the memory may be configured such that, when the second voice is recognized, the processor executes an application for providing food information indicated by a label included in the second voice to display food information provided by the application on a display. At least one instruction may be stored.
  • the memory device may further include a communication unit, wherein the processor transmits data of the first voice to a server through the communication unit when the first voice is recognized, and is transmitted from the server in the communication unit.
  • the processor transmits data of the first voice to a server through the communication unit when the first voice is recognized, and is transmitted from the server in the communication unit.
  • the apparatus may further include a communication unit, wherein the memory transmits the data of the second voice to the server through the communication unit when the second voice is recognized, and is transmitted from the server in the communication unit.
  • the memory transmits the data of the second voice to the server through the communication unit when the second voice is recognized, and is transmitted from the server in the communication unit.
  • at least one instruction set to display the food information on the display may be stored.
  • the apparatus may further include a communication unit, and, when the processor recognizes the first voice, the memory transmits the first voice data to the server through the communication unit, and transmits the first voice to the server.
  • the communication information is received by the communication unit, an instruction according to the analysis information is transmitted to an application related to performing the work according to the analysis information, and the application may store at least one instruction set to display the food list. .
  • the apparatus may further include a communication unit, and, when the processor recognizes the second voice, transmits data of the second voice to the server through the communication unit, and transmits the second voice to the server.
  • the application may store at least one instruction set to display the food information. .
  • the apparatus may further include a speaker, wherein the memory outputs a voice indicating that the list is displayed through the speaker when the processor recognizes the first voice and displays the food list on the display.
  • the processor recognizes the first voice and displays the food list on the display.
  • at least one instruction set to output a voice indicating that the food information is displayed through the speaker may be stored.
  • a refrigerator includes a storage compartment for storing food; A temperature sensor for sensing a temperature inside the storage compartment; Cooling unit for supplying cold air to the storage compartment; A microphone for receiving a voice; A display for displaying information; At least one processor electrically connected to the temperature sensing unit, the microphone, and the display; And a memory electrically connected to the at least one processor.
  • the memory when the voice is input through the microphone, the memory performs a command indicated by the input voice for the application displayed on the display, and performs a command indicated by the input voice for the application. If not, store at least one instruction set to execute the command for another application according to a predetermined priority for the applications associated with the command.
  • the apparatus may further include a speaker, wherein the memory is further configured to generate a voice indicating that the processor cannot perform the command for the application if the processor cannot perform a command indicated by the input voice for the application. At least one instruction set to be output through the speaker may be stored.
  • the apparatus may further include a speaker, wherein the memory is configured to output, through the speaker, a voice requesting confirmation whether the processor performs the command for another application when the processor does not have the priority. You can store one instruction.
  • the memory may store at least one instruction set by the processor to execute a command indicated by the voice to an application indicated by the input response when the controller receives an answer to the confirmation request.
  • the memory determines the application to perform the command according to a predetermined priority for the applications associated with the command and At least one instruction configured to execute the command with respect to the determined application may be stored.
  • the memory may be configured to cause the processor to perform a command indicated by the input voice for an application displayed on the display, when the voice input through the microphone does not include a target to perform the command indicated by the voice.
  • At least one set instruction may be stored.
  • a refrigerator includes a storage compartment for storing food; A temperature sensor for sensing a temperature inside the storage compartment; Cooling unit for supplying cold air to the storage compartment; A microphone for receiving a voice; A display for displaying information; A speaker for outputting voice; And at least one processor electrically connected to the temperature sensing unit, the microphone, the speaker, and the display. And a memory electrically connected to the at least one processor.
  • the memory is configured to display a user registered in advance on the display and select from among the displayed users, when user identification is required to perform a command indicated by the voice.
  • Store at least one instruction configured to output the desired voice through the speaker.
  • the memory is further configured to perform a command indicated by the voice in relation to the selected user when the processor inputs a voice for selecting at least one user among the displayed users, and display the result on the display. At least one instruction may be stored.
  • the memory may further include at least one instruction configured to display, by the processor, a user identifying a user who is registered in advance on the display and a beacon for identifying each user when user identification is required to perform a command indicated by the voice. Can be stored.
  • the memory may further include at least one set by the processor to perform a command indicated by the voice in relation to a user indicated by the mark and to display a result on the display when a voice for selecting at least one of the marks is input. Instructions can be stored.
  • the memory when the voice is input through the microphone, the memory performs a command indicated by the voice in relation to a user included in the voice when the voice includes a representation representing a user. At least one instruction set to display a result on the display may be stored.
  • the memory is further configured to perform a command indicated by the voice in relation to a user represented by voice data matching the input voice among prestored voice data, and output the result.
  • At least one instruction set to be displayed on the display may be stored.
  • An information display method of a refrigerator may include: receiving a first voice including a food name through a microphone; Displaying a food list on a display in front of the refrigerator, the food list including food information corresponding to the food name and an identification mark for identifying the food information, based on the recognition of the first voice; Receiving a second voice indicating the identification mark through the microphone; And displaying, on the display, purchase information of at least one food item corresponding to the identification mark, based on the recognition of the second voice.
  • a refrigerator capable of receiving a command through voice using voice recognition technology and outputting a content by voice.
  • a refrigerator capable of identifying a plurality of users and providing content suitable for the identified user.
  • FIG. 1 illustrates an exterior of a refrigerator according to one embodiment.
  • FIG. 2 illustrates a front surface of a refrigerator according to one embodiment.
  • FIG 3 illustrates a configuration of a refrigerator according to one embodiment.
  • FIG. 4 illustrates a display included in a refrigerator according to one embodiment.
  • FIG. 5 illustrates a home screen displayed on a display included in a refrigerator according to one embodiment.
  • FIG. 6 and 7 illustrate a voice recognition application displayed on a display included in a refrigerator according to one embodiment.
  • FIG. 8 illustrates communication with an external device through a communication unit included in a refrigerator according to one embodiment.
  • FIG. 9 and 10 illustrate a communication between a refrigerator and a server according to an embodiment.
  • 11A to 14 illustrate a method of purchasing food by a user through a refrigerator according to one embodiment.
  • 15 and 16 illustrate a method in which a refrigerator performs a command in response to a voice command of a user.
  • 17 to 20 provide a method of identifying a user in a refrigerator according to one embodiment and providing information corresponding to the identified user.
  • 21A is a block diagram of a controller for data learning and recognition according to an embodiment.
  • 21B is a detailed block diagram of a data learner and a data recognizer, according to an exemplary embodiment.
  • FIG. 22 is a flowchart of a refrigerator displaying information according to an embodiment.
  • FIG. 23 is a flowchart of a network system according to an embodiment.
  • 24 is a view illustrating a method of purchasing food by a user through a refrigerator according to another embodiment.
  • ⁇ part may refer to a unit for processing at least one function or operation.
  • the terms may refer to at least one hardware processed by at least one piece of hardware such as a field-programmable gate array (FPGA) / application specific integrated circuit (ASIC), at least one software stored in a memory, or a processor.
  • FPGA field-programmable gate array
  • ASIC application specific integrated circuit
  • the term "food” in the present specification may include industrial products, products produced or hunted by the user manufactured and traded by a person or a machine.
  • FIG. 1 illustrates an exterior of a refrigerator according to one embodiment
  • FIG. 2 illustrates a front surface of the refrigerator according to one embodiment
  • 3 illustrates a configuration of a refrigerator according to one embodiment
  • FIG. 4 illustrates a display included in the refrigerator according to one embodiment.
  • the refrigerator 1 may include a main body 10 having a front face open and a door 30 opening and closing the open front face of the body 10. have.
  • the main body 10 is provided with a storage compartment 20 whose front surface is opened so that food can be refrigerated or frozen.
  • the main body 10 may form an appearance of the refrigerator 1.
  • the main body 10 may include an inner wound 11 forming the storage compartment 20 and an outer wound 12 coupled to an outer side of the inner wound 11 to form an outer appearance.
  • a heat insulating material (not shown) may be filled between the inner wound 11 and the outer wound 12 of the main body 10 to prevent leakage of cold air from the storage compartment 20.
  • the storage compartment 20 may be divided into a plurality of compartments by the horizontal barrier rib 21 and the vertical barrier rib 22.
  • the storage compartment 20 may be divided into an upper storage compartment 20a, a lower first storage compartment 20b, and a lower second storage compartment 20c.
  • the storage compartment 20 may be provided with a shelf 23 on which food can be placed and a sealed container 24 for hermetically storing food.
  • the storage compartment 20 may be opened and closed by the door 30.
  • the upper storage compartment 20a may be opened and closed by the upper first door 30aa and the upper second door 30ab
  • the lower first storage compartment 20b may be the lower first. It may be opened and closed by the door 30b
  • the lower second storage compartment 20c may be opened and closed by the lower second door 30c.
  • a handle 31 may be provided at the door 30 to easily open and close the door 30.
  • the handle 31 may be formed long in the vertical direction along the upper first door 30aa and the upper second door 30ab and between the lower first door 30b and the lower second door 30c. . Thereby, when the door 30 is closed, the handle 31 can be seen as being integrally provided.
  • the refrigerator 1 may include a display 120, a storage 130, a communication unit 140, a dispenser 150, a cooling unit 150, a temperature sensing unit 170, an acoustic unit 180, and a main controller ( 110) may be included.
  • the display 120 can interact with the user.
  • the user may receive a user input from the user and display an image according to the received user input.
  • the display 120 includes a display panel 121 for displaying an image, a touch panel 122 for receiving a user input, and a touch screen controller 123 for controlling / driving the display panel 121 and the touch panel 122. can do.
  • the display panel 121 may convert image data received from the main controller 110 through the touch screen controller 123 into an optical image that can be viewed by the user.
  • the display panel 101 includes a cathode ray tube (CRT) display panel, a liquid crystal display (LCD) panel, a light emitting diode (LED) panel, and an organic light emitting diode (OLED).
  • CTR cathode ray tube
  • LCD liquid crystal display
  • LED light emitting diode
  • OLED organic light emitting diode
  • Diode (PDP) panels, plasma display panels (PDPs), field emission display (FED) panels and the like can be employed.
  • the display panel 101 is not limited thereto, and the display panel 101 may employ various display means capable of visually displaying an optical image corresponding to the image data.
  • the touch panel 122 may receive a user's touch input and transmit an electrical signal corresponding to the received touch input to the touch screen controller 123.
  • the touch panel 122 senses a user's touch on the touch panel 122 and transmits an electrical signal corresponding to the coordinates of the user's touch point to the touch screen controller 123.
  • the touch screen controller 123 may acquire the coordinates of the touch point of the user based on the electrical signal received from the touch panel 122.
  • the touch panel 122 may be positioned in front of the display panel 121.
  • the touch panel 122 may be provided on a surface on which an image is displayed. Therefore, the touch panel 122 may be made of a transparent material so that the image displayed on the display panel 121 is not distorted.
  • the touch panel 122 may employ a resistive touch panel or a capacitive touch panel. However, the touch panel 122 is not limited thereto, and the touch panel 122 may detect a user's touch or approach, and output various electrical signals corresponding to the detected coordinates of the touch point or the coordinates of the access point. An input means can be employed.
  • the touch screen controller 123 may drive / control the operations of the display panel 121 and the touch panel 122.
  • the touch screen controller 123 drives the display panel 121 to display an optical image corresponding to the image data received from the main controller 110, and detects the coordinates of the touch point of the user. ) Can be controlled.
  • the touch screen controller 123 may determine the coordinates of the user's touch point based on the electrical signal output from the touch panel 122, and transmit the coordinates of the user's touch point to the main controller 110. .
  • the touch screen controller 123 may transmit an electrical signal output from the touch panel 122 to the main controller 110 so that the main controller 110 determines the coordinates of the user's touch point.
  • the touch screen controller 123 may include a memory (not shown) that stores programs and data for controlling operations of the display panel 121 and the touch panel 122, and the display panel 121 according to the programs and data stored in the memory. And a microprocessor (not shown) that performs an operation for controlling the operation of the touch panel 122.
  • the memory and the processor of the touch screen controller 123 may be provided as separate chips or may be provided as a single chip.
  • the display 120 may be installed in the door 30 for the convenience of the user.
  • the display 120 may be installed in the upper second door 30ab.
  • the display 120 installed in the upper second door 30ab will be described, but the installation position of the display 120 is not limited to the upper second door 30ab.
  • the dispenser 150 is a position that can be viewed by the user, such as the upper first door 30aa, the lower first door 30b, the lower second door 30c, and the trauma 12 of the main body 10. Can be installed anywhere.
  • the display 120 may have a wake up function that is automatically activated when the user approaches within a certain range. For example, when the user approaches within a certain range, the display 120 may be activated. In other words, the display 120 may be in an on state. On the other hand, if the user deviates within a certain range, the display 120 may be deactivated. In other words, the display 120 may be in an off state.
  • the display 120 may display various screens or images. The screen or image displayed on the display 120 is described in detail below.
  • the storage unit 130 may store a control program and control data for controlling the operation of the refrigerator 1 and various application programs and application data for performing various functions according to a user input.
  • the storage unit 130 may include an operating system (OS) program that manages the configuration and resources (software and hardware) included in the refrigerator 1, an image display application that displays a pre-stored image, and a pre-stored video.
  • OS operating system
  • the storage unit 130 may include a nonvolatile memory in which a program or data is not lost even when power is cut off.
  • the storage 130 may include a large capacity flash memory or a solid state drive (SSD) 131.
  • the communication unit 140 may transmit data to or receive data from an external device under the control of the main controller 110.
  • the communication unit 140 may include at least one of communication modules 141, 142, and 143 that transmit and receive data according to a predetermined communication protocol.
  • the communication unit 140 may connect to an external device such as a WiFi Fidelity (WiFi®) module 141 that connects to a local area network (LAN) through a wireless access device.
  • WiFi® WiFi Fidelity
  • LAN local area network
  • Bluetooth® module 142 for one-to-one communication or one-to-many communication with a few external devices, and a Zigbee module for forming a short-range communication network between a plurality of electronic devices (mainly home appliances) 143) and the like.
  • each of the plurality of communication modules 141, 142, and 143 transmits a radio signal to or receives radio signals from free space, an antenna for modulating data to be transmitted, and a demodulator / demodulator for demodulating the received radio signals. It may include.
  • the dispenser 150 may discharge water or ice according to a user input. In other words, the user may directly take out water or ice to the outside without opening the door 30 through the dispenser 150.
  • the dispenser 150 includes a dispenser lever 151 for receiving a discharge command from a user, a dispenser nozzle 152 for discharging water or ice, a flow path 153 for guiding water from the external water supply to the dispenser nozzle 152, and discharge It may include a filter 154 for purifying the water to be dispensed and the dispenser display panel 155 for displaying the operating state of the dispenser 150.
  • the dispenser 150 may be installed outside the door 30 or the main body 10.
  • the dispenser 150 may be installed in the upper first door 30aa.
  • the dispenser 150 installed in the upper first door 30aa will be described, but the dispenser 150 is not limited to being installed in the upper first door 30aa.
  • the dispenser 40 draws water or ice from the user, such as the upper second door 30ab, the lower first door 30b, the lower second door 30c, and the external wound 12 of the body 10. It can be installed anywhere you can.
  • the door 30 or the outer box 12 may be provided with a cavity 150a recessed inwardly to form a water or ice extraction space, and the cavity 150a may have a dispenser nozzle 152 and a dispenser lever. 151 may be provided. When the user presses the dispenser lever 151, water or ice is discharged from the dispenser nozzle 152.
  • the water when water is discharged through the dispenser nozzle 152, the water may flow from an external water supply source (not shown) along the flow path 152 to the dispenser nozzle 152. Water may also be purified by filter 153 while flowing to dispenser nozzle 152.
  • the filter 153 may be detachably provided in the main body 10 or the door 30 so that the filter 153 may be replaced when the life of the filter 153 expires.
  • the cooling unit 160 may supply cold air to the storage chamber 20.
  • the cooling unit 160 may maintain the temperature of the storage compartment 20 within a predetermined range by using evaporation of the refrigerant.
  • the cooling unit 160 may include a compressor 161 for compressing the gas refrigerant, a condenser 162 for converting the compressed gas refrigerant into a liquid refrigerant, an expander 163 for reducing the liquid refrigerant, and a reduced liquid refrigerant in a gas state. It may include an evaporator 164 to change the state.
  • the cooling unit 160 may supply cold air to the storage compartment 20 by using a phenomenon in which the pressure-reduced liquid refrigerant absorbs thermal energy of ambient air while changing its state into a gas state.
  • the configuration of the cooling unit 160 is not limited to the compressor 161, the condenser 162, the expander 163, and the evaporator 164.
  • the cooling unit 160 may include a Peltier element using the Peltier effect.
  • the Peltier effect means that when a current flows through contact surfaces of different metals, an exothermic phenomenon occurs in one metal and an endothermic phenomenon occurs in another metal.
  • the cooling unit 160 may supply cold air to the storage compartment 102 using the Peltier element.
  • the cooling unit 160 may include a magnetic cooling device using a magneto-caloric effect.
  • the magnetocaloric effect means that heat is released when a specific material (magnetic calorie material) is magnetized, and absorbs heat when the specific material (magnetic calorie material) is demagnetized.
  • the cooling unit 160 may supply cold air to the storage compartment 20 by using a magnetic cooling device.
  • the temperature sensing unit 170 may be provided in the storage compartment 20 to sense a temperature inside the storage compartment 20.
  • the temperature sensing unit 170 may include a plurality of temperature sensors 171 installed in the plurality of storage chambers 20a, 20b, and 20c, respectively.
  • each of the plurality of temperature sensors 171 may include a thermistor whose electrical resistance changes with temperature.
  • the sound unit 180 converts an electrical signal received from the main controller 110 into an acoustic signal, converts an electrical signal into a speaker 181 for outputting the converted acoustic signal, and converts an electrical signal into a main controller ( It may include a microphone 182 that outputs to 110.
  • the main controller 110 may include a display 120, a storage 130, and a communication unit included in the refrigerator 1 according to a user input and / or a program and data stored in the storage 130 received through the display 120.
  • the controller 140 may control the dispenser 150, the cooling unit 150, the temperature sensing unit 170, and the acoustic unit 180.
  • the main controller 110 may include a microprocessor 111 that performs operations for controlling the refrigerator 1, and a memory 112 that stores / stores programs and data related to operations of the microprocessor 111. Can be.
  • the microprocessor 111 may load data stored / stored in the memory 112 according to a program stored / stored in the memory 112 and perform an arithmetic operation or a logical operation on the retrieved data. Also, the microprocessor 111 may output the result of an arithmetic operation or a logical operation to the memory 112.
  • the memory 112 may include a volatile memory that loses stored data when power supply is interrupted.
  • the volatile memory may load a program and data from the storage 130 described above, and temporarily store the loaded data.
  • the volatile memory can provide stored programs and data to the microprocessor 111, and can store data output from the microprocessor 111.
  • Such a volatile memory may include an S-RAM, a D-lap, and the like.
  • the memory 112 may include a nonvolatile memory as necessary.
  • the nonvolatile memory can retain the stored data when the power supply is interrupted.
  • the nonvolatile memory may store firmware for managing and initializing various components included in the refrigerator 1.
  • Such nonvolatile memories may include ROM, EPYROM, EPYROM, flash memory, and the like.
  • the main controller 110 may include a plurality of microprocessors 111 and a plurality of memories 112.
  • the first microprocessor and the first memory for controlling the temperature sensor 170, the dispenser 150, and the cooling unit 160 of the refrigerator 1, the display 120 of the refrigerator 1, and a storage unit. 130, a second microprocessor and a second memory for controlling the communicator 140 and the acoustic unit 180 may be included.
  • microprocessor 111 and the memory 112 have been described functionally and distinctly, the microprocessor 111 and the memory 112 are not necessarily physically distinguished.
  • the microprocessor 111 and the memory 112 may be implemented not only as separate chips but also as one chip.
  • the main controller 110 may control the overall operation of the refrigerator 1, and the operation of the refrigerator 1 described below may be interpreted as being performed by the control of the main controller 110.
  • the main controller 110, the storage unit 130, and the communication unit 140 have been functionally distinguished and described, but the main controller 110, the storage unit 130, and the communication unit 140 are not necessarily physically distinguished.
  • the main controller 110, the storage unit 130, and the communication unit 140 may be implemented as one chip, or each of the main controller 110, the storage unit 130, and the communication unit 140 may be a separate chip. Can be implemented.
  • the display 120, the storage unit 130, the communication unit 140, the dispenser 150, the cooling unit 150, the temperature sensing unit 170, and the sound unit 180 included in the refrigerator 1 are described above.
  • the main controller 110 has been described, a new configuration may be added or some components may be omitted as necessary.
  • FIG. 5 illustrates a home screen displayed on a display included in a refrigerator according to one embodiment.
  • the main controller 110 may control the display 120 to display the home screen 200 as illustrated in FIG. 5.
  • an application stored in the time / date display area 210 displaying the time and date, the operation information display area 220 displaying the operation information of the refrigerator 1, and the storage 130 is executed.
  • a plurality of launcher 230 for may be displayed.
  • time / date display area 210 In the time / date display area 210, current time information and today's date information may be displayed. In addition, the time / date display area 210 may display location information (eg, a name of a country or city) of a location where the refrigerator 1 is located.
  • location information eg, a name of a country or city
  • a storage map 221 related to operations of the plurality of storage rooms 20a, 20b, and 20c included in the refrigerator 1 may be displayed.
  • the storage map 221 may display information related to operations of the plurality of storage rooms 20a, 20b, and 20c included in the refrigerator 1. For example, as illustrated in FIG. 5, an upper storage compartment 20a, a lower first storage compartment 20b, and a lower second storage compartment 20c may be divided and displayed in the storage compartment map 221, and the upper storage compartment 20a may be displayed. ), A target temperature of the lower first storage compartment 20b and a target temperature of the lower second storage compartment 20c may be displayed.
  • the main controller 110 displays an image for displaying a target temperature of each of the storage compartments 20a, 20b, and 20c. Can be marked on. For example, when a user touches an area indicating the upper storage compartment 20a in the storage compartment map 221, an image for setting a target temperature of the upper storage compartment 20a may be displayed on the display 120.
  • the operation information display area 220 may display a timer setting icon 222 and a refrigerator setting icon 223 for executing an application for controlling the operation of the and refrigerator 1.
  • a timer setting screen for setting a target time of the timer may be displayed on the display 120.
  • the user may input the time to output the alarm or the time interval until the alarm is output through the timer setting image, and the refrigerator 1 outputs the alarm at the time input by the user or the user.
  • an alarm can be output.
  • the main controller 110 may display, on the display 120, an operation setting screen for inputting a setting value for controlling the operation of the refrigerator 1. For example, a user may set a target temperature of each of the plurality of storage compartments 20a, 20b, and 20c included in the refrigerator 1 through an operation setting screen, and which of the water and ice is discharged through the dispenser 150. You can set whether or not.
  • the home screen 200 may display a plurality of launcher 230 for executing various applications stored in the storage 130 by the user.
  • the home screen 200 includes an album launcher 231 for executing an album application displaying a picture stored in the storage 130, and a recipe launcher 232 for executing a recipe application providing a recipe for food.
  • the screen setting launcher 233 for executing the screen setting application for controlling the operation of the display 120 may be displayed.
  • the home screen 200 sets the operation of the home appliance control launcher 234 for executing the home appliance control application for controlling various home appliances through the refrigerator 1, and the operation of the voice output application for outputting various contents as a voice.
  • the voice output setting launcher 235 and the online shopping launcher 236 for executing a shopping application for online shopping may be displayed.
  • the home screen 200 of the refrigerator 1 may display main information related to the operation of the refrigerator 1 and a launcher for executing various applications.
  • FIG. 5 is only an example of the home screen 200, and the refrigerator 100 may display various types of home screens according to a user's setting, and the information and the launcher displayed on the home screen. Again not limited to the bar shown in FIG.
  • FIG. 6 illustrates a voice recognition UI displayed on the display 120 of the refrigerator according to one embodiment.
  • the main controller will be referred to as a controller for convenience.
  • the controller 110 may display a voice recognition user interface 250 (hereinafter, referred to as a UI) on the display 120.
  • the speech recognition UI 250 may be displayed on the display 120 as a UI having a configuration as illustrated in FIG. 6.
  • the wake word may be composed of a preset word or a combination of words, and may be changed to a desired expression by the user.
  • the controller 110 may execute the voice recognition function through the wake word uttered by the user. As shown in FIG. 7, when the microphone-shaped button 271 displayed on the public UI is touched, the voice recognition function is performed. You can also run
  • the shape of the launcher representing the button for executing the voice recognition function is not limited to the shape of the microphone, but may be implemented in various other forms of images.
  • the notification UI may be displayed on the display 120 when a touch gesture for swiping upward while a specific point at the bottom of the display 120 is touched is input.
  • the launcher for executing the voice recognition function may be displayed on the notification UI or may be displayed on the home screen 200 of the display 120.
  • the controller 110 recognizes and analyzes the user's voice in natural language using a voice recognition function to determine the user's intention, and can execute a command according to the user's intention. In addition, by outputting the voice process or the result of the command according to the user's intention so that the user can recognize the voice process or the result of the voice command.
  • the voice recognition UI 250 may be displayed on the display 120, as shown in FIGS. 6 and 7.
  • the speech recognition UI 250 may include a first region 252 in which a word or sentence output as speech through a text to speech (TTS) function is displayed, a second region 254 in which a state of a speech recognition function is displayed, and a TTS.
  • a third area 260 indicating that speech is output through the text to speech function, a setting object 256 for setting the speech recognition function, and a help object 258 for providing help for using the speech recognition function. ) May be included.
  • the second area 254 in which the status of the voice recognition function is displayed may display, in text, a state in which the user's voice is being input, a standby state for the user's voice input, or a state in which a process according to the voice input is being performed. Can be.
  • a state in which the user's voice is input may be displayed as ⁇ listening>, a standby state as ⁇ stand by>, and a state in which the process is performed as ⁇ processing>.
  • the above-described English text is merely an example, and the status may be displayed in Korean text, or may be displayed as an image instead of text.
  • the voice recognition UI 250 may be displayed in the form of a card UI popped up on the home screen 200 or displayed on the entire screen of the display 120. May be
  • the controller 110 transmits voice data to the first server SV1 to be described later, and receives information analyzed by the voice from the first server SV1. You can run the command.
  • FIGS. 9 and 10 illustrate communication between a refrigerator and a server according to an embodiment.
  • the refrigerator 1 may communicate with various electronic devices as well as the servers SV1 and SV2 through the communication unit 140.
  • the refrigerator 1 may be connected to an access point AP via the communication unit 140.
  • the refrigerator 1 may use a connection repeater (Wi-Fi TM, IEEE 802.11), Bluetooth (Bluetooth TM, IEEE 802.15.1), Zigbee (Zigbee, IEEE 802.15.4) and the like using a wireless communication standard.
  • AP connection repeater
  • the access relay may be called variously, such as “hub”, “router”, “switch”, “gateway”, a wide area network (Internet), such as the Internet (Wide Area) Network) (WAN).
  • Internet Wide Area Network
  • connection repeater AP not only a refrigerator 1 but also an air conditioner 2, a washing machine 3, an oven 4, a microwave oven 5, a robot cleaner 6, a security camera 7, an electric lamp ( 8) and various electronic devices such as the television 9 can be connected.
  • the electronic devices 1-9 connected to the access relay AP may form a local area network (LAN).
  • LAN local area network
  • the access relay AP may connect a local area network (LAN) formed by the electronic devices 1-9 connected to the access relay AP and a wide area network (WAN) such as the Internet.
  • LAN local area network
  • WAN wide area network
  • the wide area network WAN may be connected to a first server SV1 that provides information analyzed by voice data to a refrigerator and a second server SV2 operated by operators that provide information through an application mounted on the refrigerator.
  • the second server SV2 may be, for example, a server selling food online through a shopping application such as a mart application mounted in a refrigerator (hereinafter referred to as a mart server), and a server providing information to a weather application (hereinafter referred to as a weather server).
  • a server providing recipe related information to a recipe application hereinafter referred to as a recipe server
  • a music server hereinafter referred to as a music server
  • a mobile terminal MT may be connected to the wide area network WAN.
  • the mobile terminal MT may directly connect to the wide area network WAN or access the wide area network WAN through the access relay AP.
  • the mobile terminal MT may access the wide area network WAN through the access repeater AP.
  • the mobile terminal MT may directly connect to the wide area network WAN through a mobile communication service provided by the mobile communication provider.
  • the refrigerator 1 transmits data to the first server SV1 and / or the second server SV2 through the connection repeater AP, and the first server SV1 and / or the second server SV2 and the like. Data may be received from.
  • the refrigerator 1 may transmit voice data of a user input through the microphone 182 to the first server SV1 through the access repeater AP, and the first server SV1 may transmit the voice data.
  • the analysis information including the user's intention derived from the analysis may be transmitted to the refrigerator.
  • the refrigerator 1 may transmit information analyzing the voice data of the user to the second server SV2 through the access repeater AP, and may be associated with a specific application from the second server SV2 through the access repeater AP.
  • Information for example, food information associated with a mart application.
  • the refrigerator 1 may communicate with the first server SV1 and receive analysis information of the voice data of the user through the first server SV1.
  • the refrigerator 1 may communicate with the second server SV2 and receive information related to a specific application from the second server SV2.
  • the communication between the refrigerator and the server will be described in more detail with reference to FIGS. 9 and 10.
  • the refrigerator 1 transmits voice data to the first server SV1 (1000).
  • the first server SV1 may analyze the voice data to derive the intention of the user (1010).
  • the controller 110 of the refrigerator 1 modulates the input analog voice signal into voice data, which is a digital signal, and communicates with the first server SV1 through the communication unit 140. To send.
  • the controller 110 of the refrigerator 1 may modulate an analog voice signal into a digital voice signal by using pulse code modulation.
  • the controller 110 of the refrigerator 1 may input a voice command of ⁇ Notify me of the weather> from the user, and may modulate the voice command into a digital signal and transmit the same to the first server SV1.
  • the first server SV1 removes noise by performing preprocessing on the speech data transmitted from the refrigerator 1, and analyzes the speech data and converts the speech data into text, and automatic speech recognition (ASR) and automatic speech recognition. It may include a natural language understanding (NLU) to grasp the user's intention based on the text converted by the wealth. Alternatively, the first server SV1 may include a learning network model trained using an artificial intelligence algorithm. In this case, the first server SV1 may apply the voice data transmitted from the refrigerator 1 to the learning network model to determine (or recognize, estimate, infer, and predict) the intention of the user.
  • ASR automatic speech recognition
  • NLU natural language understanding
  • the first server SV1 may include a learning network model trained using an artificial intelligence algorithm. In this case, the first server SV1 may apply the voice data transmitted from the refrigerator 1 to the learning network model to determine (or recognize, estimate, infer, and predict) the intention of the user.
  • the automatic speech recognition unit and the natural language processing unit of the first server SV1 may determine the user's intention that the voice data requires information on the weather of the region where the user is currently located.
  • the first server SV1 transmits the information analyzing the voice data to the second server SV2 (1020), and the second server SV2 transmits the information requested by the voice data according to the analysis information of the voice data. In operation 1030, it may be transmitted to SV1.
  • the first server SV1 transmits the information transmitted from the second server SV2 to the refrigerator 1 (1040).
  • the first server SV1 determines the intention of the user included in the voice data
  • the first server SV1 determines the second server SV2 that provides information to an application related to the user's intention, and determines the voice data to the determined second server SV2. Transmit the analyzed information.
  • the weather server provides information to the weather application of the second server SV2.
  • the analysis information of the voice data can be transmitted.
  • the second server SV2 may search for information corresponding to the user's intention based on the analysis information of the voice data and transmit the information to the first server SV1.
  • the weather server may generate current weather information of the area where the user is based on the analysis information of the voice data, and transmit the generated weather information to the first server SV1.
  • the first server SV1 converts the information transmitted from the second server SV2 into a JSON (JavaScript Object Notation) file format and transmits the information to the refrigerator 1.
  • JSON JavaScript Object Notation
  • the refrigerator 1 displays the information transmitted from the first server SV1 on the display 120 using a card type UI and outputs it by voice (1050).
  • the controller 110 of the refrigerator 1 displays the received information as a card type UI (120). ), And the received information can be output by voice using the Text To Speech (TTS) function.
  • TTS Text To Speech
  • the controller 110 of the refrigerator 1 may receive information including the location, time, and weather of the user.
  • the card UI may display the display 120.
  • the controller 110 may output the weather information displayed by the card UI as a voice (for example, ⁇ today's current weather is 20 degrees, humidity is 50%, and it is clear without clouds>). have.
  • the refrigerator 1 transmits voice data to the first server SV1 (1100).
  • the first server SV1 may analyze the voice data to derive the intention of the user (1110).
  • the controller 110 of the refrigerator 1 modulates the input analog voice signal into voice data, which is a digital signal, and communicates with the first server SV1 through the communication unit 140. To send.
  • the controller 110 of the refrigerator 1 may modulate an analog voice signal into a digital voice signal by using pulse code modulation. For example, when the voice command “Listen to next song” is input from the user, the controller 110 of the refrigerator 1 may modulate the voice command into a digital signal and transmit the same to the first server SV1. Alternatively, when the voice command “tell me apple pie recipe” is input from the user, the controller 110 of the refrigerator 1 may modulate the voice command into a digital signal and transmit it to the first server SV1.
  • the first server SV1 performs preprocessing on the speech data transmitted from the refrigerator 1 to remove noise, analyzes the speech data, and converts the speech data into text, and automatic speech recognition (ASR) and automatic speech recognition. It may include a natural language understanding (NLU) to grasp the user's intention based on the text converted by the wealth. Alternatively, the first server SV1 may include a learning network model trained using an artificial intelligence algorithm. In this case, the first server SV1 may apply the voice data transmitted from the refrigerator 1 to the learning network model to determine (or recognize, estimate, infer, and predict) the intention of the user.
  • ASR automatic speech recognition
  • NLU natural language understanding
  • the first server SV1 may include a learning network model trained using an artificial intelligence algorithm. In this case, the first server SV1 may apply the voice data transmitted from the refrigerator 1 to the learning network model to determine (or recognize, estimate, infer, and predict) the intention of the user.
  • the automatic speech recognition unit and the natural language processing unit of the first server SV1 may grasp the user's intention that the speech data requires the reproduction of the next song of the song currently being played.
  • the automatic voice recognition unit and the natural language processing unit of the first server SV1 may recognize the user's intention that the voice data is required to inform the user of how to cook the apple pie.
  • the first server SV1 transmits information analyzing the voice data to the refrigerator 1 (1120), and the refrigerator 1 transmits analysis information of the voice data to the second server SV2 (1130).
  • the second server SV2 may transmit the information requested by the voice data to the refrigerator 1 according to the analysis information of the voice data (1140).
  • the first server SV1 analyzes the voice data, and when the user's intention is determined, converts the analysis information of the voice data of the user into a JSON (JavaScript Object Notation) file format and transmits it to the refrigerator 1.
  • JSON JavaScript Object Notation
  • the controller 110 of the refrigerator 1 determines a related application based on the analysis information, The analysis information can be output to the determined application.
  • the controller 110 of the refrigerator 1 determines that it is the user's intention to request the playback of the next song of the currently playing song based on the analysis information in the JSON file format, the user's intention of the music application is determined. This may be determined by an application that performs a function related to. The controller 110 may transmit a command for requesting reproduction of the next song to the determined music application.
  • the controller 110 of the refrigerator 1 determines the user's intention to inform the cooking method of the apple pie based on the analysis information in the JSON file format
  • the application performs a recipe application function related to the user's intention. Can be determined.
  • the controller 110 may transmit a command requesting information on a method of cooking an apple pie to the determined recipe application.
  • the determined application may transmit the analysis information to the second server SV2 that provides the information.
  • the second server SV2 may search for information corresponding to the user's intention based on the analysis information of the voice data and transmit the information to the refrigerator 1.
  • the music application when receiving the analysis information output from the controller 110, the music application may request information about the next song from the second server SV2 that provides the information to the music application, that is, the music server. have.
  • the music server may determine the next song of the song currently being played based on the analysis information of the voice data, and provide the music application with information about the determined next song.
  • the recipe application may request information on how to cook the apple pie to the second server (SV2) that provides information to the recipe application, that is, the recipe server.
  • the recipe server may provide a recipe application of a pre-stored apple pie to a recipe application based on analysis information of voice data, or provide a recipe application by searching for another cooking method of apple pie.
  • the refrigerator 1 may execute the related application according to the received information (1150).
  • the controller 110 executes the determined application according to the information transmitted from the second server SV2, and then executes the determined application.
  • Information corresponding to voice may be provided.
  • the controller 110 executes a music application when the information about the next song transmitted from the music server is received.
  • the music application may stop playing the currently playing song and play the next song according to the information about the next song transmitted from the music server.
  • the display 120 of the refrigerator 1 may display a UI of a music application to display information related to playing the next song, and the output of the song currently being played through the speaker 181 of the refrigerator 1 is stopped. The next song can be output.
  • the controller 110 executes the recipe application.
  • the display 120 of the refrigerator 1 may display a UI of a recipe application to display an apple pie cooking method, and a voice for reading the recipe of the apple pie through the speaker 181 of the refrigerator 1 may be output. Can be.
  • information related to a voice of a user may be received through communication between the first server SV1 and the second server SV2, and the refrigerator 1 and the second server may be received as illustrated in FIG. 10.
  • Information related to the voice of the user may be received through communication between the SV2s.
  • 11A to 14 illustrate a method of purchasing food by a user through a refrigerator according to one embodiment.
  • a voice for ordering a particular food item with a wake word may be input through the microphone 182 (900).
  • the controller 110 executes a voice recognition function to analyze the user's voice, and receives a list 300 of foods included in the user's voice from a related application (for example, a mart application).
  • a related application for example, a mart application.
  • the display 120 displays the display 120 in the form of a display.
  • the refrigerator 1 analyzes a user's voice through communication with the first server SV1 to determine the user's intention, and information related to the user's intention transmitted from the second server SV2. (Eg, a list of foods) may be received through the first server SV1. Then, using the received information as an input value of the application for displaying a food list, as shown in Figure 11A can display the food list 300 in the form of a card UI.
  • the voice analysis result of the user may be, for example, a food name.
  • the food name may include not only the exact name of the food, but also a idiom or slang that refers to the food, a portion of the food name, and a name similar to the food name.
  • the food name may be an alias registered or uttered mainly by the user of the refrigerator 1.
  • the food list 300 may be displayed in the form of a card UI at the bottom of the speech recognition UI 250.
  • the food list 300 includes a representative image 314 representing a food, food information 316 including a manufacturer, a food name, a dose, a quantity, a price, and the like, and an identification mark 312 to distinguish the food from other foods. , For example, a number.
  • a label for distinguishing each of the plurality of foods may be used, and as shown, may be indicated by a number or may be indicated by letters.
  • a selectable tab 318 may be displayed at the bottom of the food list 300.
  • Tab 318 may be represented by a letter indicating the function of tab 318 (see more food at the mart).
  • a mart application that provides food information may be executed and displayed on the display 120.
  • the controller 110 may output, through the speaker 181, a voice requesting confirmation regarding which food from among the foods included in the food list 300 is selected together with the food list 300, and a voice recognition UI.
  • a microphone image indicating that voice is being output through the speaker 181 may be displayed in the third region 260 of the 250.
  • the voice output through the speaker 181 may also be displayed as text in the first area 250 of the voice recognition UI.
  • the controller 110 may recognize the first voice including the food name through the microphone 182. For example, the controller 110 may modulate the input first voice into a digital voice signal. The controller 110 may acquire a food name corresponding to the first voice by recognizing a digital voice signal through a learning network model trained using an artificial intelligence algorithm. In this case, the food name included in the first voice and the food name acquired by recognizing may be the same as or different from each other. For example, when the food name included in the first voice is an idiom, alias or part of a food name, the food name recognized and acquired may be a full name, a sales name, or a brand name.
  • the learning network model that recognizes the first voice is being stored in the storage 130 or in the first server SV1 that receives the first voice data from the refrigerator 1 and recognizes the first voice data. Can be.
  • the controller 110 may transmit a first voice modulated by a digital voice signal to the first server SV1.
  • the first server SV1 may input (eg, estimate, infer, predict, or grasp) a food name corresponding to the first voice by applying the first voice as an input value to a learning network model trained using an artificial intelligence algorithm. Can be.
  • the first server SV1 may transmit the food name corresponding to the first voice to the controller 110.
  • the controller 110 applies the first voice modulated by the digital voice signal to the learning network model stored in the storage unit 130 to generate the first voice. It can recognize the food name corresponding to the.
  • the controller 110 When a food name is obtained using the learning network model, the controller 110 includes a food list including food information 316 related to the obtained food name and an identification mark 312 for distinguishing the food information 316 from other foods. 300 may be displayed on the display 120.
  • the controller 110 may recognize the second voice that refers to the identification mark 312 of the food list through the microphone 182. For example, the controller 110 may modulate the input second voice into a digital voice signal. In addition, the controller 110 may acquire the identification mark corresponding to the second voice by recognizing the digital voice signal through the learning network model trained using the artificial intelligence algorithm. In this case, the learning network model for recognizing the second voice may be stored in the storage 130 or in the first server SV1 analyzing the voice data.
  • the controller 110 may transmit a second voice modulated by a digital voice signal to the first server SV1.
  • the first server SV1 applies the second voice as an input value to the learning network model trained using an artificial intelligence algorithm to recognize (or estimate, infer, predict, or identify) an identification mark corresponding to the second voice. Can be.
  • the first server SV1 may transmit an identification mark corresponding to the second voice to the controller 110.
  • the controller 110 applies the second voice modulated by the digital voice signal to the learning network model stored in the storage unit 130 to generate the second voice. Recognize an identification mark corresponding to.
  • the controller 110 may display on the display 120 purchase information of at least one food indicated by the identification mark 312.
  • 11B illustrates a method for a user to purchase food through a refrigerator according to another embodiment.
  • voice may be input 950 for ordering a particular food through microphone 182.
  • the controller 110 may recognize the first voice including the food name. As described above, as an example, the controller 110 may recognize the first voice by applying the first voice input through the microphone 182 to a learning network model trained using an artificial intelligence algorithm.
  • the controller 110 may identify whether the food 951-1 associated with the food name corresponding to the first voice exists in the storage compartment 20 (951).
  • the food associated with the food name may be a food having the food name as at least part of the name, or a food similar or replaceable with the food name.
  • the refrigerator 1 may store the storage information list including the food names of the foods existing in the storage room 20 in the storage unit 130.
  • the food names of the storage information list may be generated by a user input when the user stores the food in the storage compartment 20 of the refrigerator 1.
  • the information input by the user by voice or text may be stored as the food name, or the food name included in the identification information may be stored as the user tags the food identification information (eg, a barcode).
  • the food names may be generated as a result of image recognition of the storage room 10 photographed using a camera provided in the refrigerator 1.
  • the refrigerator 1 may recognize a food name and store the recognized food name by applying an image photographed through a camera to a learning network model trained using an artificial intelligence algorithm.
  • the refrigerator 1 may search whether a food name corresponding to the first voice exists in the storage information list, and identify whether a food associated with the food name exists in the storage compartment 20.
  • the controller 110 May control the display 120 to display information indicating the presence of food 951-1 (952).
  • Information indicating the presence of food may be, for example, a video or image of the food (952-1), a notification text indicating the presence of food (e.g., 'there is a food similar to what you are looking for') 952-2, or It may include at least one of the notification sound indicating the presence of food.
  • the user may want to order food additionally.
  • the display 120 displaying information indicating the presence of the foods 951-1 may be displayed together with a UI (not shown) for further ordering of foods.
  • the controller 110 may control the display 120 to display the food list 300 as shown in FIG. 11A.
  • 11C illustrates a method of purchasing food by a user through a refrigerator according to another embodiment.
  • voice may be input 960 for ordering a particular food through microphone 182.
  • the controller 110 may recognize the first voice including the food name. As described above, as an example, the controller 110 may recognize the first voice by applying the first voice input through the microphone 182 to a learning network model trained using an artificial intelligence algorithm.
  • the controller 110 may identify whether food associated with the food name corresponding to the first voice exists in the storage compartment 20 (961). For example, the controller 110 may identify whether a food associated with a food name exists in the storage compartment 20 based on an image of the storage compartment 20 photographed using a camera provided in the refrigerator 1. Alternatively, a storage information list for foods existing in the storage compartment 20 may exist in the storage 130 of the refrigerator 1. In this case, the controller 110 may search whether the food name exists in the list of stored foods and identify whether the food associated with the food name exists in the storage compartment 20.
  • the controller 110 may purchase a food name to purchase the food.
  • the display 120 may display a food list 962-1 including food information having a name corresponding to and a label for identifying the food information (962).
  • the food list 962-1 may be displayed in a card UI form at the bottom of the voice recognition UI.
  • 11D illustrates a method for a user to purchase food through a refrigerator according to another embodiment.
  • voice may be input 970 to order a particular food through microphone 182.
  • the controller 110 may recognize the first voice including the food name. As described above, as an example, the controller 110 may recognize the first voice by applying the first voice input through the microphone 182 to a learning network model trained using an artificial intelligence algorithm. In addition, the controller 110 recognizes a user who uttered the first voice by applying the first voice input through the microphone 182 to the learning network model trained using an artificial intelligence algorithm, and utters the first voice. User information can be obtained.
  • the controller 110 may transmit a first voice modulated by a digital voice signal to the first server SV1.
  • the first server SV1 may recognize (or estimate, infer, predict, or grasp) a user who spoke the first voice by applying the first voice as an input value to a learning network model trained using an artificial intelligence algorithm. have.
  • the first server SV1 may transmit the user information that uttered the first voice to the controller 110.
  • the controller 110 applies the first voice modulated by the digital voice signal to the learning network model stored in the storage unit 130 to generate the first voice. It can recognize the user who uttered.
  • the controller 110 may obtain food information preferred by the user who spoke the first voice in relation to the food name (971).
  • the same food name may have different preferred foods for each user depending on the manufacturer of the food, the type of food or the capacity of the food, and the place of sale of the food.
  • the storage unit 130 of the refrigerator 1 may store a food information list 971-1 for each user, in which preferred food information for each user is registered for the same food name.
  • the food information for each user may be determined based on a purchase history purchased by each user, an input history registered directly by the user, a food information history corresponding to an identification mark selected from identification labels corresponding to the food name, and the like.
  • the food information list for each user may be stored in the food information according to the preferred priority for each user. For example, food information whose priority is determined according to the order of the purchase history of the user may be stored.
  • the controller 110 may acquire food information preferred by a specific user based on the food information list stored in the storage 130 (971). Alternatively, the controller 110 may apply food name and user information to a learning network model learned using an artificial intelligence algorithm to obtain food information preferred by a specific user.
  • the learning network model may be a learning network model trained using the above-described purchase history, input history, and food information history corresponding to the selected identification mark.
  • the controller 110 may include a food list including an identification mark for distinguishing the food information that the user prefers from other food information that the user prefers. 972-1) may be displayed on the display 120. In this case, the food list 972-1 may sort the food information preferred by the user according to the user's preferred ranking.
  • a user in response to a voice requesting confirmation of which food from among foods included in the food list, a user may utter a voice for selecting a specific food (902). For example, the user may utter the identification label to distinguish foods.
  • the user may utter the first or first one to select an identification label to select the first food in the food list (903, 904).
  • the controller 110 may analyze the user's voice through a voice recognition function.
  • the controller 110 transmits the voice data of the user to the first server SV1 and receives information analyzed by the first server SV1 to analyze the intention of the user. I can figure it out.
  • the controller 110 of the refrigerator 1 may analyze the user's voice and determine the user's intention using the voice recognition function.
  • the controller 110 may search for a food name corresponding to the identification mark uttered by the user (905) and display the searched food. As shown in FIG. 13, the controller 110 may display a UI including purchase information for purchasing selected foods by executing a mart application on the display 120 (906).
  • the controller 110 outputs information analyzing the voice data of the user to the mart application, and executes the mart application when receiving information on food from the second server SV2.
  • the UI as shown in FIG. 13 may be displayed on the display 120.
  • the food confirmation UI includes a representative image area 354 representing a food, a food information area 356 including a manufacturer, a food name, a capacity, a quantity, a price, and the like, a quantity control area 358 for controlling the quantity of food, A food purchase area 360 including food purchase information such as a shopping cart tab for putting food in a shopping cart, a direct purchase tab for purchasing food directly, a gift tab for presenting food, and a tab for bookmarking selected foods It may include.
  • the number or arrangement of regions included in the food purchase UI illustrated in FIG. 13 is merely an example and may include other numbers, arrangements or other contents.
  • the user may say ⁇ put in the shopping cart> to put the food displayed in the food confirmation UI in the shopping cart (907), or say ⁇ buy it> to immediately purchase the food displayed in the food confirmation UI. May be 912.
  • the controller 110 displays the voice recognition UI on the display 120 as shown in FIG. 14.
  • the text “Add the selected food to the shopping cart” may be displayed on the voice recognition UI.
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the voice recognition UI may be displayed as a card UI type on the mart application screen.
  • the mart application may display a shopping cart screen on the display 120 (911).
  • the controller 110 displays the voice recognition UI as shown in FIG. , An error occurred when adding the shopping cart. Check the content and try again> may be displayed on the speech recognition UI (910).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. In this case, the user may say "put it in the shopping cart" again to add the food to the shopping cart.
  • the controller 110 displays a voice recognition UI on the display 120 as shown in FIG. 14.
  • the text ⁇ Go to the purchase page of the selected food> may be displayed on the voice recognition UI.
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the voice recognition UI may be displayed as a card UI type on the mart application screen.
  • the mart application may display a food purchase screen on the display 120 when a text and a voice indicating that the function of directly purchasing a food is normally executed to move to a purchase page of the food are output.
  • the controller 110 displays a voice recognition UI on the display 120, as shown in FIG. An error occurred when moving the purchase page. Check the contents and try again> may be displayed on the speech recognition UI (916). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. In this case, the user may say ⁇ purchase> again to purchase the food again.
  • the user when the user does not want to select a food included in the food list or wants to cancel the food purchase process, the user may say ⁇ Cancel>.
  • the controller 110 may display the speech recognition UI on the display 120 in response to the user's speech called ⁇ cancel>, and display the text ⁇ food search canceled> on the speech recognition UI (918).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the voice recognition UI may be displayed as a card UI type on the mart application screen.
  • the user may say a food name that is not an identification mark (919). If foods of the same food name exist among the foods included in the food list, or if the user says the wrong food name, the food selected by the user may not be identified.
  • the controller 110 may display the voice recognition UI on the display 120 in response to the user's speech, and display the text ⁇ Please tell the number> on the voice recognition UI ( 920).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 provides information in the form of a card UI without immediately executing the related application in response to a user's utterance, and then selects a related application when specific information is selected from the information provided through the card UI. Can be run.
  • 15 and 16 illustrate a method in which a refrigerator performs a command in response to a voice command of a user.
  • the controller 110 of the refrigerator 1 may perform a command corresponding to the voice command as illustrated in FIG. 15.
  • the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function to intention of the user.
  • the controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.
  • the controller 110 of the refrigerator 1 determines whether a music application is currently running, that is, whether music is being played (1301). If the music is playing, the controller 110 immediately stops playing the music (1302).
  • the controller 110 may display the voice recognition UI on the display 120 as shown in FIG. 15, and display the text ⁇ Music not being played> on the voice recognition UI. (1303). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • a voice command for which an object is not specified may be input through the microphone 182. That is, as shown in FIG. 16, a voice of ⁇ stop>, in which an object of a command is omitted, may be input through the microphone 182 (1300).
  • the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function to determine the user's intention.
  • the controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.
  • the controller 110 determines whether the voice command can be performed on the currently running application displayed on the display 120 (1310).
  • the controller 110 When the controller 110 can perform a voice command on the currently running application displayed on the display 120, the controller 110 performs a voice command on the application displayed on the display 120 (1320).
  • the controller 110 immediately stops the function of reading a recipe application. You can.
  • the controller 110 when a voice command in which the command object is omitted is input, the controller 110 is displayed on the display 120 to determine whether the command can be executed first with respect to the currently running application.
  • the controller 110 When the controller 110 is unable to perform a voice command on the currently running application displayed on the display 120, the controller 110 displays a voice recognition UI on the display 120 to inform that the voice command cannot be performed (1330). .
  • the controller 110 may display the voice recognition UI on the display 120 to display the text ⁇ not reading the recipe> on the voice recognition UI.
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 determines whether a predetermined priority exists for the application related to the input voice command (1340). . If there is a predetermined priority, the controller 110 performs a voice command for the application according to the priority (1350).
  • the controller 110 performs a voice command for the application ranked in the highest rank according to the priority, and if it is unable to perform the voice command for the application ranked in the highest rank, it is ranked in the next rank. Perform voice commands for the application.
  • the controller 110 stops playing the music being played by the music application. If the music is not playing, the controller 110 determines whether it is possible to perform a voice command for the application ranked in the next rank. The controller 110 may perform a voice command for the applications ranked in the priority in this way.
  • the controller 110 may request confirmation of whether to perform the voice command for another application (1360).
  • the controller 110 determines whether a voice command can be performed for another application.
  • the controller 110 determines the radio application as a target of a voice command as shown in FIG. 16, and displays a voice recognition UI before performing the voice command for the radio application. And request confirmation of whether or not the voice command is performed.
  • the controller 110 may display the voice recognition UI on the display 120 and display the text ⁇ Shall I stop the radio> on the voice recognition UI.
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 When the answer to the confirmation request is input through the microphone 182, the controller 110 performs a command according to the input answer (1370).
  • the controller 110 stops the execution of the radio application when an affirmative voice (eg ⁇ yes>) is input to the confirmation request " Stop radio? &Quot;.
  • an affirmative voice eg ⁇ yes>
  • the refrigerator 1 may perform a command corresponding to the intention of the user according to the method shown in FIG. 16.
  • 17 to 20 provide a method of identifying a user in a refrigerator according to one embodiment and providing information corresponding to the identified user.
  • a user may refer to a specific user and request the provision of information related to the user.
  • the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function. Understand your intentions.
  • the controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.
  • a calendar application As an example of an application for providing information specific to each user, a calendar application is mentioned. However, the present invention is not limited thereto, and a voice output setting application (eg, morning briefing in FIG. Yes) may provide the user-specific information according to the method according to Figs.
  • a voice output setting application eg, morning briefing in FIG. Yes
  • the controller 110 of the refrigerator 1 When the controller 110 of the refrigerator 1 recognizes the name of a specific user included in the voice and the intention of the user included in the voice is determined, the controller 110 performs a command according to the voice. That is, if it is determined that the user's intention is to show the mother's calendar among the plurality of users and the calendar of this month among the mother's calendars, as shown in FIG.
  • the moon calendar 290 is displayed on the display 120 (2030).
  • a calendar provided by a calendar application may include a calendar portion 291 in which a date and a day are displayed, and a calendar portion 292 in which a today's schedule is displayed.
  • the configuration of the calendar shown in FIG. 18 is merely an example and may have different configurations and arrangements.
  • the controller 110 of the refrigerator 1 cannot recognize the name of a specific user included in the voice, the controller 110 performs a process of selecting a specific user from among prestored users (2020).
  • the controller 110 of the refrigerator 1 cannot recognize the name of a specific user included in the voice, as shown in FIG. 19, the controller 110 displays the voice recognition UI on the display 120 and selects a profile.
  • the text may be displayed on the voice recognition UI (2021).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 may display a profile list of a user registered in advance in the refrigerator 1 together with the voice recognition UI.
  • the profile list includes registered user names and an indication of an identification mark to identify each user name.
  • the identification mark may be indicated by a number, as shown in FIG. 19, but is not limited thereto. In the case of the identification mark, it may be a mark for distinguishing a plurality of users, respectively, and as shown, may be indicated by a number or may be indicated by a letter.
  • the user in response to a voice requesting confirmation of which user among users included in the profile list, the user may utter a voice for selecting a specific user (2022). For example, the user may speak the identification mark negatively.
  • the user may select an identification mark by uttering a second or second to select a second user in the profile list (2023, 2024).
  • the controller 110 may analyze the user's voice through a voice recognition function.
  • the controller 110 transmits the voice data of the user to the first server SV1 and receives information analyzed by the first server SV1 to analyze the intention of the user. I can figure it out.
  • the controller 110 of the refrigerator 1 may analyze the user's voice and determine the user's intention using the voice recognition function.
  • the controller 110 may determine a user name corresponding to the identification mark uttered by the user, and display the determined user's calendar on the display 120. That is, the controller 110 determines the mother as the user corresponding to the identification mark of the second or the second uttered by the user, and displays the mother's current month's calendar on the display 120 as shown in FIG. 18 (2030). ).
  • the user may say a user name that is not an identification mark (2025).
  • the controller 110 may display the voice recognition UI on the display 120 in response to the user's speech, and display the text ⁇ Please tell the profile number> on the voice recognition UI. (2026).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 displays the voice recognition UI on the display 120 to display the text " Correct information cannot be obtained. &Quot; May be displayed on the voice recognition UI (2028).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 may display the speech recognition UI on the display 120 in response to the user's speech called ⁇ cancel>, and display the text ⁇ canceled> on the speech recognition UI.
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • a user may request the provision of information stored differently for each user without referring to a specific user.
  • the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function. To grasp the intentions.
  • the controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.
  • the controller 110 of the refrigerator 1 compares the voice data with voice data of the pre-stored users and determines whether there is a matching voice data (2041).
  • the controller 110 of the refrigerator 1 compares a parameter such as amplitude, waveform, frequency, or period of the voice data input through the microphone 182 with a parameter of pre-stored voice data, and selects a microphone ( In operation 182, voice data that may be considered to match the input voice data may be determined.
  • the controller 110 may display the user's calendar indicated by the determined voice data on the display 120. May be 2042. For example, if the matched voice data among the pre-stored voice data is determined to be the mother's, the controller 110 may display the mother's current month's calendar on the display 120.
  • the controller 110 determines that there is no voice data that can be considered to be matched (NO in 2041), as shown in FIG. 19, the controller 110 displays the voice recognition UI on the display 120 to display the text ⁇ Please select a profile>. It may be displayed on the voice recognition UI (2021). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. The controller 110 may display a profile list of a user registered in advance in the refrigerator 1 together with the voice recognition UI.
  • the user in response to a voice requesting confirmation of which user among users included in the profile list, the user may utter a voice for selecting a specific user (2022). For example, the user may speak the identification mark negatively.
  • the user may select an identification mark by uttering a second or second to select a second user in the profile list (2023, 2024).
  • the controller 110 may determine a user name corresponding to the identification mark uttered by the user, and display the determined user's calendar on the display 120. That is, the controller 110 determines the mother as the user corresponding to the identification mark of the second or the second uttered by the user, and displays the mother's current month's calendar on the display 120 as shown in FIG. 18 (2030). ).
  • the user may say a user name that is not an identification mark (2025).
  • the controller 110 may display the voice recognition UI on the display 120 in response to the user's speech, and display the text ⁇ Please tell the profile number> on the voice recognition UI. (2026).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the controller 110 displays the voice recognition UI on the display 120 to display the text " Correct information cannot be obtained. &Quot; May be displayed on the voice recognition UI (2028).
  • the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.
  • the refrigerator 1 identifies and identifies the user according to the method shown in FIG. 20 even if the user does not refer to a specific user but requires the provision of information stored differently for each user. It can provide information suitable for the user.
  • 21A is a block diagram of a controller for data learning and recognition according to an embodiment.
  • the controller 2100 may include a data learner 2110 and a data recognizer 2120.
  • the controller 2100 may be the controller 110 of the refrigerator 1.
  • the controller 2100 may be a controller (not shown) of the first server SV1.
  • the data learner 2110 may train the learning network model to have a criterion for recognizing voice data. For example, the data learner 2110 may train the learning network model to have a criterion for recognizing a food name corresponding to the first voice or an identification mark corresponding to the second voice. Alternatively, the data learner 2110 may train the learning network model to have a reference for recognizing an image. For example, the data learner 2110 may train the learning network model to have a criterion for recognizing food stored in the storage room from an image of the storage room of the refrigerator 1.
  • the data learner 2110 may train a learning network model (or a data recognition model) using learning data according to a supervised learning method or an unsupervised learning method based on an artificial intelligence algorithm.
  • the data learner 2110 may train the learning network model using the voice data and the text corresponding to the voice data (for example, a food name and an identification mark) as the training data.
  • the data learner 2110 may train the learning network model using the voice data and the user information that uttered the voice data as the training data.
  • the data learner 2110 may train the learning network model using image data and text (eg, a food name) corresponding to the image data as learning data.
  • the data recognizer 2120 may recognize the voice data by applying the voice data as the recognition data to the learned learning network model.
  • the data recognizer 2120 may recognize food names or identification marks corresponding to the voice data by applying voice data related to the first voice or the second voice as recognition data to the learning network model.
  • the recognition result of the learning network model (or data recognition model) may be used again to update the learning network model.
  • the data recognizer 2120 may recognize the user who spoke the voice data by applying the voice data as the recognition data to the learning network model.
  • the data recognizing unit 2120 may recognize the image data by applying the image data of the storage room 20 as the recognition data to the learned learning network model.
  • the data recognizing unit 2120 may apply image data as recognition data to the learning network model to recognize a food name corresponding to the image data.
  • At least one of the data learner 2110 and the data recognizer 2120 may be manufactured in the form of at least one hardware chip and mounted on the electronic device.
  • at least one of the data learner 2110 and the data recognizer 2120 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general purpose processor (eg, a CPU).
  • the electronic device may be manufactured as a part of an application processor or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various electronic devices.
  • a dedicated hardware chip for artificial intelligence is a dedicated processor specialized for probability calculation, and has higher parallelism performance than a conventional general-purpose processor, so that a computational task in artificial intelligence such as machine learning can be processed quickly.
  • the data learner 2110 and the data recognizer 2120 may be mounted on one electronic device or may be mounted on separate electronic devices, respectively.
  • one of the data learner 2110 and the data recognizer 2120 may be included in the refrigerator 1, and the other may be included in the server SV1.
  • the data learner 2110 and the data recognizer 2120 may provide at least a part of the learning network model constructed by the data learner 2110 to the data recognizer 2120 through wired or wirelessly. Data input to the data recognizer 2120 may be provided to the data learner 2110 as additional learning data.
  • At least one of the data learner 2110 and the data recognizer 2120 may be implemented as a software module.
  • the software module may be a computer readable non-transitory computer. It may be stored in a non-transitory computer readable media.
  • at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • OS operating system
  • OS operating system
  • others may be provided by a predetermined application.
  • 21B is a detailed block diagram of the data learner 2110 and the data recognizer 2120, according to an exemplary embodiment.
  • FIG. 21B is a block diagram of the data learner 2110, according to an exemplary embodiment.
  • the data learner 2110 may include a data acquirer 2110-1, a preprocessor 2110-2, a training data selector 2110-3, and a model.
  • the learning unit 2110-4 and the model evaluating unit 2110-5 may be included.
  • the data learner 2110 according to some exemplary embodiments may include a data acquirer 2110-1 and a model learner 2110-4, and include a preprocessor 2110-2 and a training data selector 2110-. 3) and at least one of the model evaluator 2110-5 may optionally be further included or not included at all.
  • the data acquirer 2110-1 may acquire data necessary for learning a criterion for recognizing voice data or image data.
  • the data acquirer 2110-1 may acquire audio data or video data.
  • the data acquirer 2110-1 may acquire audio data or image data from the refrigerator 1.
  • the data acquisition unit 2110-1 may acquire audio data or video data from a third device (eg, a mobile terminal or a server) connected to the refrigerator 1.
  • the data acquirer 2110-1 may acquire audio data or image data from a device or database storing or managing learning data.
  • the preprocessor 2110-2 may preprocess audio data or video data.
  • the preprocessor 2110-2 may process the acquired audio data or the image data into a preset format so that the model learner 2110-4 to be described later can use the acquired data for learning for situation determination. Can be.
  • the preprocessor 2110-2 may remove noise or process the audio data or the image data acquired by the data acquirer 2110-1 so as to select meaningful data or to process a predetermined form. Can be.
  • the preprocessor 2110-2 may process the acquired data in the form of data suitable for learning.
  • the training data selector 2110-3 may select voice data or video data required for learning from the preprocessed data according to a preset criterion or randomly.
  • the selected training data may be provided to the model learner 2110-4.
  • the preset criterion may include, for example, at least one of a property of data, a time for generating data, a place for generating data, a device for generating data, reliability of data, and a size of data.
  • training data selector 2110-3 may select the training data according to a preset criterion by the training by the model learner 2110-4 to be described later.
  • the model learner 2110-4 may train the learning network model to have a criterion for recognizing the voice data or the image data based on the training data. For example, the model learner 2110-4 may train the learning network model to have a criterion for recognizing a food name or an identification mark corresponding to the voice data. For example, the model learner 2110-4 may train the learning network model to have a criterion for recognizing food stored in the storage room from image data photographing the storage room of the refrigerator 1.
  • the learning network model may be a pre-built model.
  • the learning network model may be a model built in advance by receiving basic learning data (for example, sample image data or audio data).
  • the learning network model may be set to recognize (or determine, estimate, infer, predict) a food name or an identification mark corresponding to the voice.
  • the learning network model may be set to recognize (or determine, estimate, reason, predict) the user who spoke the voice.
  • the learning network model may be set to recognize (or determine, estimate, infer, predict) the food stored in the storage room from the image data.
  • the learning network model may be, for example, a model based on a neural network.
  • the learning network model can be designed to simulate the human brain structure on a computer.
  • the learning network model may include a plurality of weighted network nodes that simulate neurons in a human neural network.
  • a plurality of network nodes may form a connection relationship to simulate the synaptic activity of neurons that send and receive signals through synapses.
  • the learning network model may include, for example, a neural network model or a deep learning model developed from the neural network model.
  • a plurality of network nodes may be located at different depths (or layers) and exchange data according to a convolutional connection relationship.
  • models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the learning network model, but are not limited thereto.
  • the model learning unit 2110-4 may be a learning network model for learning a learning network model having a large correlation between input learning data and basic learning data. You can decide.
  • the model learner 2110-4 may train the learning network model using, for example, a learning algorithm including an error back-propagation method or a gradient descent method. .
  • the model learner 2110-4 may train the learning network model through, for example, supervised learning using learning data as an input value.
  • the model learner 2110-4 may train the learning network model through unsupervised learning that finds a criterion by learning by itself without any guidance.
  • the model learner 2110-4 may train the learning network model through, for example, reinforcement learning using feedback on whether the result of learning is correct.
  • the model learner 2110-4 may store the learned learning network model.
  • the model learner 2110-4 may store the learned learning network model in the storage 130 of the refrigerator 1.
  • the model learner 2110-4 may store the learned learning network model in a memory of the server SV1 or the server SV2 connected to the refrigerator 1 through a wired or wireless network.
  • the storage unit 130 in which the learned learning network model is stored may store, for example, instructions or data related to at least one other component of the refrigerator 10.
  • the memory may also store software and / or programs.
  • the program may include, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or “application”), and the like.
  • the model evaluator 2110-5 may input the evaluation data into the learning network model, and cause the model learner 2110-4 to relearn if the recognition result output from the evaluation data does not satisfy a predetermined criterion. have.
  • the evaluation data may be preset data for evaluating the learning network model.
  • the model evaluator 2110-5 may determine a predetermined criterion when the number or ratio of the evaluation data that is not accurate among the recognition results of the learned learning network model for the evaluation data exceeds a preset threshold. It can be evaluated as not satisfied. For example, when a predetermined criterion is defined at a ratio of 2%, the model evaluation unit 2110-5 when the learned learning network model outputs an incorrect recognition result for more than 20 evaluation data out of a total of 1000 evaluation data. Can be judged that the learned learning network model is not suitable.
  • the model evaluator 2110-5 evaluates whether a predetermined criterion is satisfied for each learned learning network model, and the model that satisfies the predetermined criterion is a final learning network. Can be determined as a model. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 2110-5 may determine any one or a predetermined number of models which are preset in the order of the highest evaluation score as the final learning network model.
  • At least one of -5) may be manufactured in the form of at least one hardware chip and mounted on the electronic device.
  • at least one of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5 One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general purpose processor (eg, a CPU or application processor) or a graphics dedicated processor (eg, a GPU). It may be mounted on various electronic devices.
  • AI artificial intelligence
  • the data obtaining unit 2110-1, the preprocessor 2110-2, the training data selecting unit 2110-3, the model learning unit 2110-4, and the model evaluating unit 2110-5 are electronic components. It may be mounted on the device, or may be mounted on separate electronic devices, respectively.
  • some of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5. May be included in the refrigerator 1 and the remaining part may be included in the server SV1 or the server SV2.
  • At least one of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5 is provided. It may be implemented as a software module. At least one of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5 is a software module. (Or a program module including instructions), the software module may be stored in a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.
  • OS operating system
  • OS operating system
  • some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.
  • FIG. 21B is a block diagram of the data recognition unit 2120 according to an embodiment.
  • the data recognizer 2120 may include a data acquirer 2120-1, a preprocessor 2120-2, a recognition data selector 2120-3, and a recognizer.
  • the result providing unit 2120-4 and the model updating unit 2120-5 may be included.
  • the data recognizer 2120 essentially includes a data acquirer 2120-1 and a recognition result provider 2120-4, and includes a preprocessor 2120-2 and a recognition data selector ( 2120-3) and the model updater 2120-5 may optionally be further included.
  • the data recognizer 2120 may apply the voice data to the learned learning network model as the recognition data, and may recognize a food name corresponding to the voice data, a food label, or a user who utters the voice data. Alternatively, the data recognizing unit 2120 may recognize the food stored in the storage room 20 of the refrigerator 1 by applying the image data to the learned learning network model as the recognition data.
  • the data acquirer 2120-1 may acquire voice data necessary for recognizing a user who spoke a food name, a food label, or a voice from the voice data.
  • the data acquirer 2120-1 may acquire image data necessary for recognizing food from the image data.
  • the data acquirer 2120-1 may acquire various sensing information that is directly input by the user or data selected by the user, or sensed using various sensors of the refrigerator 1.
  • the data acquisition unit 2120-1 may acquire data from an external device (eg, a mobile terminal or a server) that communicates with the refrigerator 1.
  • the preprocessor 2120-2 may preprocess audio data or video data.
  • the preprocessor 2120-2 may process the acquired audio data or the image data into a predetermined format so that the recognition result providing unit 2120-4 to be described later can use the data for situation determination.
  • the preprocessor 2120-2 may remove noise or process the audio data or the image data acquired by the data acquirer 2120-1 so that meaningful data can be selected or processed into a predetermined form. Can be.
  • the preprocessor 2120-2 may process the acquired data into a form of data suitable for learning.
  • the recognition data selector 2120-3 may select voice data or image data required for recognition from the preprocessed data according to a preset criterion or randomly.
  • the selected recognition data may be provided to the model learner 2110-4.
  • the preset criterion may include, for example, at least one of a property of data, a time for generating data, a place for generating data, a device for generating data, reliability of data, and a size of data.
  • the recognition result providing unit 2120-4 may apply the selected recognition data to the learning network model to recognize the recognition network.
  • the recognition result providing unit 2120-4 may apply the selected voice data to the learning network model to recognize a food name or an identification mark corresponding to the voice.
  • the recognition result providing unit 2120-4 may recognize the user who spoke the voice by applying the selected voice data to the learning network model.
  • the recognition result providing unit 2120-4 may recognize the food corresponding to the image data by applying the selected image data to the learning network model.
  • the model updater 2120-5 may cause the learning network model to be updated based on the evaluation of the recognition result provided by the recognition result provider 2120-4.
  • the model updater 2120-5 provides the model learner 2110-4 by providing the food name, the identification mark, or the user information provided by the recognition result provider 2120-4 to the model learner 2110-4.
  • 2110-4 may update the learning network model.
  • At least one of 2120-5 may be manufactured in the form of at least one hardware chip and mounted on the electronic device.
  • the preprocessor 2120-2, the recognition data selector 2120-3, the recognition result provider 2120-4, and the model updater 2120-5 may be fabricated in the form of a dedicated hardware chip for artificial intelligence (AI), or may be fabricated as part of an existing general purpose processor (e.g., CPU or application processor) or graphics dedicated processor (e.g., GPU). It may be mounted on various electronic devices.
  • AI artificial intelligence
  • the data acquisition unit 2120-1, the preprocessor 2120-2, the recognition data selection unit 2120-3, the recognition result providing unit 2120-4, and the model updater 2120-5 may be mounted on an electronic device, or may be mounted on separate electronic devices, respectively.
  • the preprocessor 2120-2, the recognition data selector 2120-3, the recognition result provider 2120-4, and the model updater 2120-5 may be included in the electronic device, and others may be included in the server.
  • At least one of the data acquirer 2120-1, the preprocessor 2120-2, the recognition data selector 2120-3, the recognition result provider 2120-4, and the model updater 2120-5 May be implemented as a software module.
  • At least one of the data acquisition unit 2120-1, the preprocessor 2120-2, the recognition data selection unit 2120-3, the recognition result providing unit 2120-4, and the model updater 2120-5 is software.
  • the software module When implemented as a module (or a program module including instructions), the software module may be stored on a computer readable non-transitory computer readable media.
  • at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.
  • FIG. 22 is a flowchart of a refrigerator 1 displaying information according to an embodiment of the present disclosure.
  • the refrigerator 1 may determine whether a first voice including a food name is input through the microphone 182 (2201).
  • the refrigerator 1 displays the food list including food information corresponding to the food name and an identification mark for identifying the food information, based on the recognition of the first voice.
  • the display 120 may display the same.
  • the refrigerator 1 acquires a food name by recognizing the first voice by using a learning network model trained according to an artificial intelligence algorithm, and includes food information corresponding to the food name.
  • the list may be displayed on the display 120.
  • the learning network model may be a learning network model trained using a plurality of voices and a plurality of words corresponding to each of the plurality of voices.
  • the refrigerator 1 acquires a food name and user information uttering the first voice by recognizing the first voice by using a network model trained according to an artificial intelligence algorithm.
  • the display 120 may display a food list corresponding to the food information including food information related to the user information (for example, food information preferred by the user).
  • the learning network model may be a learning network model trained using a plurality of voices and user information corresponding to each of the plurality of voices.
  • the refrigerator 1 may display a food list including food information corresponding to the food name on the display. On the other hand, if food is present in the storage compartment 20, the refrigerator 1 may display on the display 120 information indicating that the food associated with the recognized food name exists in the storage compartment of the refrigerator 1.
  • the refrigerator 1 may determine whether a second voice indicating an identification mark is input through the microphone 182 (2203).
  • the refrigerator 1 transmits purchase information of at least one food item corresponding to the identification mark to the display 120 located in front of the refrigerator 1 based on the recognition of the second voice. It may be displayed (2204).
  • the refrigerator 1 acquires an identification mark by recognizing the second voice using a learning network model trained according to an artificial intelligence algorithm, and purchases food information corresponding to the identification mark. Can be displayed on the display.
  • FIG. 23 is a flowchart of a network system according to an embodiment of the present disclosure.
  • the network system may include a first component 2301 and a second component 2302.
  • the first component 2301 is the refrigerator 1
  • the second component 2302 may be a server SV1 in which a data recognition model is stored or a cloud including at least one server.
  • the first component 2301 may be a general purpose processor
  • the second component 2302 may be an artificial intelligence dedicated processor.
  • the first component 2301 may be at least one application
  • the second component 2302 may be an operating system (OS). That is, the second component 2302 is a component that is more integrated, dedicated, has a smaller delay, has a higher performance, or has more resources than the first component 2301. It can be a component that can process many operations required at the time of generation, update or application of the faster than the first component 2301.
  • an interface for transmitting / receiving data between the first component 2301 and the second component 2302 may be defined.
  • an application program interface having recognition data to be applied to the data recognition model as an argument value (or, a parameter value or a transfer value) may be defined.
  • An API is defined as a set of subroutines or functions that can be called for any processing of one protocol (e.g., a protocol defined in the refrigerator 1) to another protocol (e.g., a protocol defined in the server SV1). Can be. That is, an environment in which an operation of another protocol may be performed in one protocol through an API may be provided.
  • the first component 2301 may receive a first voice of a user including a food name through the microphone 182 (2310).
  • the first component 2301 may modulate the input analog voice signal into voice data which is a digital signal and transmit the same to the second component 2302. In this case, the first component 2301 may change the voice data according to the promised communication format and transmit the voice data to the second component 2302 (2311).
  • the second component 2302 may recognize the received voice data (2312). For example, the second component 2302 may apply the received voice data to the learned learning network model to recognize the voice data and obtain recognition information as a recognition result. For example, the second component 2302 may obtain the food name and the user information that spoke the voice from the voice data. The second component 2302 may transmit the obtained recognition information to the first component 2301 (2313).
  • the first component 2301 may obtain information requested by the voice data, based on the received recognition information (2314). For example, the first component 2301 may obtain food information corresponding to the food name from the storage of the first component 2301 or the third component 2303 as information requested by the voice data. It may be.
  • the third component 2303 may be, for example, a server in communication connection with the first component 2301.
  • the server may be a server SV2 transmitting information requested by voice data in FIGS. 9 and 10.
  • the second component 2302 may directly transmit the recognized recognition information to the third component 2303 without passing through the first component 2301 (2323).
  • the third component 2303 may acquire the information requested by the voice data based on the received recognition information, and transmit the information requested by the acquired voice data to the first component 2301 (2324).
  • the first component 2301 may obtain information requested by the voice data from the third component 2303.
  • the first component 2301 may include food information corresponding to a food name and an identification mark for identifying the food information as the information requested by the voice data.
  • the list of foods to be included may be displayed (2315).
  • the first component 2301 displays purchase information of at least one food corresponding to the identification mark. can do.
  • 24 is a view illustrating a method of purchasing food by a user through a refrigerator according to another embodiment of the present disclosure.
  • a first voice for ordering a specific food may be input through the microphone 182 (980).
  • the controller 110 may recognize the first voice including the food name. As a result of the recognition, the controller 110 may obtain user information that uttered the recognized food name and the first voice.
  • the controller 110 may obtain food information preferred by the user who spoke the first voice in relation to the food name (981).
  • the controller 110 may display on the display 120 a food list including an identification mark for distinguishing the plurality of food information and the plurality of food information that the user prefers ( 982).
  • a second voice indicating a specific identification mark among the identification marks included in the food list may be input (983).
  • the controller 110 may recognize the second voice including the identification mark. As a result of the recognition, the controller 110 may display the purchase information of the food corresponding to the identification mark on the display 120 (984).
  • a third voice for requesting purchase of food through the microphone 182 may be input (985).
  • the third voice requesting the purchase of food may be, for example, a voice spoken by the user as " purchase immediately ".
  • the controller 110 may recognize the third voice.
  • the controller 110 may display, on the display 120, a UI (for example, a UI for payment) for purchasing food as a result of recognizing the third voice.
  • the controller 110 may control the communication unit 140 to transmit a message informing the purchase situation to the external device 986-1 (986).
  • the message indicating the purchase status may include, for example, a message requesting confirmation of whether to purchase the food before purchasing the food, a message requesting input of a password for purchasing the food, or a message indicating a purchase result after the purchase of the food. have.
  • the controller 110 may acquire at least one of the first voice to the third voice to obtain user information uttering the voice.
  • the controller 110 may transmit a message informing the purchase situation to the user's device corresponding to the user information by using the obtained user information.
  • the controller 110 may transmit a message informing the information about the stored food to the user's device.
  • the controller 110 may transmit a message to the device of the guardian of the user who spoke the voice. For example, if the person who ordered the food is a child, the controller 110 may transmit a message informing the above-described purchase situation to the parent device of the child.
  • the external device 986-1 receiving the message may display the purchase status as text, provide a voice, display an image, display haptic or emit light.
  • the manner of notifying the purchase situation is not limited to the above example.
  • the controller 110 may notify a purchase situation to an external device 986-1 that is previously set or communicates with the refrigerator 1. Can also be transmitted. Alternatively, the controller 110 may transmit a message informing the purchase situation to the external device 986-1 that satisfies the preset criteria. At this time, the external device 986-1 that satisfies the preset criteria is located at a predetermined place (for example, a living room, a room, etc.), the signal strength communicating with the refrigerator 1 is greater than or equal to a predetermined value, or the refrigerator 1
  • the target device may be an external device having a distance from the predetermined distance.
  • the preset reference may be an external device 986-1 connected to the refrigerator 1 according to a specific communication method.
  • the specific communication method may include, for example, a short range communication method (eg, Wi-Fi, Bluetooth, Zigbee, etc.) or a direct communication method.
  • the controller 110 sends a push message indicating a purchase status through a communication channel established according to the short range communication method. (986-1).
  • module includes, for example, a unit composed of a combination of one or more of hardware, software, or firmware, for example, a logic, a logical block.
  • block may be used interchangeably.
  • the module may be an integrally formed part or a minimum unit or part of performing one or more functions.
  • Modules may be implemented mechanically or electronically, for example, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) or programs that perform some operations, known or will be developed in the future. It may include a programmable-logic device.
  • ASIC application-specific integrated circuit
  • FPGAs field-programmable gate arrays
  • the methods described above may be embodied in the form of program instructions that may be executed by various computer means and may be recorded in a computer readable medium.
  • the computer readable medium may include program instructions, data files, data structures, etc. alone or in combination.
  • Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts.
  • Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks.
  • Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.
  • the hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.
  • the method according to the disclosed embodiments may be provided as a computer program product.
  • the computer program product may include a S / W program, a computer readable storage medium on which the S / W program is stored, or a food product traded between a seller and a buyer.
  • a computer program product may be a food product (eg, a downloader) in the form of a S / W program distributed electronically through a refrigerator 1, a server SV1, a server SV2 or a manufacturer of the device, or an application store.
  • a loadable app For electronic distribution, at least a part of the S / W program may be stored in a storage medium or temporarily generated.
  • the storage medium may be a server of a manufacturer or an application store or a storage medium of a relay server.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • General Engineering & Computer Science (AREA)
  • Thermal Sciences (AREA)
  • Mechanical Engineering (AREA)
  • Combustion & Propulsion (AREA)
  • Chemical & Material Sciences (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

L'invention concerne un réfrigérateur comprenant: une chambre de stockage pour stocker des aliments; une unité de détection de température pour détecter la température à l'intérieur de la chambre de stockage; une unité de refroidissement pour fournir de l'air froid à la chambre de stockage; un dispositif d'affichage pour afficher des informations; un microphone destiné à recevoir une voix; au moins un processeur connecté électriquement à l'unité de détection de température, au microphone et au dispositif d'affichage; et une mémoire connectée électriquement à l'au moins un processeur. La mémoire stocke au moins un ensemble d'instructions de telle sorte que le processeur affiche, sur le dispositif d'affichage: une liste d'aliments comprenant des informations d'aliment, correspondant à un nom d'aliment, et un indicateur d'identification capable d'identifier les informations d'aliment, lorsqu'une première voix comprenant le nom d'aliment est reconnue à travers le microphone; et acquérir des informations d'au moins un aliment correspondant à l'indicateur d'identification lorsqu'une seconde voix indiquant l'indicateur d'identification est reconnue à travers le microphone.
PCT/KR2017/015548 2017-01-03 2017-12-27 Réfrigérateur et son procédé d'affichage d'informations WO2018128317A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201780082150.3A CN110168298B (zh) 2017-01-03 2017-12-27 冰箱及其信息显示方法
US16/476,008 US11521606B2 (en) 2017-01-03 2017-12-27 Refrigerator and information display method thereof
EP17890733.3A EP3546869A4 (fr) 2017-01-03 2017-12-27 Réfrigérateur et son procédé d'affichage d'informations

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR10-2017-0000880 2017-01-03
KR20170000880 2017-01-03
KR1020170179174A KR102412202B1 (ko) 2017-01-03 2017-12-26 냉장고 및 이의 정보 표시 방법
KR10-2017-0179174 2017-12-26

Publications (1)

Publication Number Publication Date
WO2018128317A1 true WO2018128317A1 (fr) 2018-07-12

Family

ID=62789537

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2017/015548 WO2018128317A1 (fr) 2017-01-03 2017-12-27 Réfrigérateur et son procédé d'affichage d'informations

Country Status (1)

Country Link
WO (1) WO2018128317A1 (fr)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110473542A (zh) * 2019-09-06 2019-11-19 北京安云世纪科技有限公司 语音指令执行功能的唤醒方法、装置及电子设备
WO2020181748A1 (fr) * 2019-03-13 2020-09-17 青岛海尔电冰箱有限公司 Procédé d'interaction de données de parole et système d'interaction destinés à être utilisés lors de l'ajout d'une matière alimentaire dans un réfrigérateur
CN112771901A (zh) * 2018-09-19 2021-05-07 伊莱克斯电器股份公司 可自定义的食物显示器
IT202000004741A1 (it) * 2020-03-05 2021-09-05 Nanolever S R L Apparecchiatura per trattare prodotti alimentari e corrispondente procedimento
EP3775733A4 (fr) * 2018-09-18 2021-09-22 Samsung Electronics Co., Ltd. Réfrigérateur et procédé de commande associé
CN114442964A (zh) * 2020-10-30 2022-05-06 青岛海尔电冰箱有限公司 食材信息打印系统
US11353259B2 (en) 2018-09-18 2022-06-07 Samsung Electronics Co., Ltd. Augmented-reality refrigerator and method of controlling thereof
EP3957938A4 (fr) * 2019-04-15 2022-06-15 Sony Group Corporation Dispositif de traitement d'informations, procédé de traitement d'informations et dispositif de stockage d'aliments

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080105933A (ko) * 2007-06-01 2008-12-04 엘지전자 주식회사 냉장고의 정보 입력 방법
KR100885584B1 (ko) * 2007-08-23 2009-02-24 엘지전자 주식회사 냉장고 및 그 제어방법
KR20120118376A (ko) * 2011-04-18 2012-10-26 엘지전자 주식회사 냉장고의 제어방법
US20140169640A1 (en) * 2012-09-28 2014-06-19 Lg Electronics Inc. Electric product
KR20140139736A (ko) * 2013-05-28 2014-12-08 삼성전자주식회사 냉장고 및 그 제어 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080105933A (ko) * 2007-06-01 2008-12-04 엘지전자 주식회사 냉장고의 정보 입력 방법
KR100885584B1 (ko) * 2007-08-23 2009-02-24 엘지전자 주식회사 냉장고 및 그 제어방법
KR20120118376A (ko) * 2011-04-18 2012-10-26 엘지전자 주식회사 냉장고의 제어방법
US20140169640A1 (en) * 2012-09-28 2014-06-19 Lg Electronics Inc. Electric product
KR20140139736A (ko) * 2013-05-28 2014-12-08 삼성전자주식회사 냉장고 및 그 제어 방법

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3546869A4 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3775733A4 (fr) * 2018-09-18 2021-09-22 Samsung Electronics Co., Ltd. Réfrigérateur et procédé de commande associé
US11353259B2 (en) 2018-09-18 2022-06-07 Samsung Electronics Co., Ltd. Augmented-reality refrigerator and method of controlling thereof
US11732961B2 (en) 2018-09-18 2023-08-22 Samsung Electronics Co., Ltd. Augmented-reality refrigerator and method of controlling thereof
CN112771901A (zh) * 2018-09-19 2021-05-07 伊莱克斯电器股份公司 可自定义的食物显示器
WO2020181748A1 (fr) * 2019-03-13 2020-09-17 青岛海尔电冰箱有限公司 Procédé d'interaction de données de parole et système d'interaction destinés à être utilisés lors de l'ajout d'une matière alimentaire dans un réfrigérateur
EP3957938A4 (fr) * 2019-04-15 2022-06-15 Sony Group Corporation Dispositif de traitement d'informations, procédé de traitement d'informations et dispositif de stockage d'aliments
CN110473542A (zh) * 2019-09-06 2019-11-19 北京安云世纪科技有限公司 语音指令执行功能的唤醒方法、装置及电子设备
CN110473542B (zh) * 2019-09-06 2022-04-15 北京安云世纪科技有限公司 语音指令执行功能的唤醒方法、装置及电子设备
IT202000004741A1 (it) * 2020-03-05 2021-09-05 Nanolever S R L Apparecchiatura per trattare prodotti alimentari e corrispondente procedimento
CN114442964A (zh) * 2020-10-30 2022-05-06 青岛海尔电冰箱有限公司 食材信息打印系统

Similar Documents

Publication Publication Date Title
WO2018128317A1 (fr) Réfrigérateur et son procédé d'affichage d'informations
WO2020060090A1 (fr) Procédé et système de fourniture d'informations concernant l'état d'un objet dans un réfrigérateur
WO2021010509A1 (fr) Appareil de cuisson à intelligence artificielle
WO2019156499A1 (fr) Dispositif électronique et procédé d'exécution de fonction de dispositif électronique
WO2018088794A2 (fr) Procédé de correction d'image au moyen d'un dispositif et dispositif associé
WO2020204221A1 (fr) Dispositif de conditionnement d'air
WO2020060380A1 (fr) Réfrigérateur et procédé de commande associé
WO2018084576A1 (fr) Dispositif électronique et procédé de commande associé
WO2019027240A1 (fr) Dispositif électronique et procédé pour fournir un résultat de recherche de celui-ci
WO2019225961A1 (fr) Dispositif électronique permettant de générer une réponse à une entrée vocale à l'aide d'une application, et procédé de fonctionnement associé
WO2020091210A1 (fr) Système et procédé d'intégration de bases de données d'après un graphe de connaissances
WO2019107945A1 (fr) Procédé de fourniture de service sur la base de l'emplacement d'une source sonore et dispositif de reconnaissance vocale associé
WO2018199483A1 (fr) Procédé et appareil de gestion d'un agent intelligent
WO2016108660A1 (fr) Procédé et dispositif pour commander un dispositif domestique
WO2021125453A1 (fr) Réfrigérateur à intelligence artificielle
WO2021137345A1 (fr) Réfrigérateur à intelligence artificielle et son procédé de fonctionnement
WO2021006404A1 (fr) Serveur d'intelligence artificielle
WO2018182270A1 (fr) Dispositif électronique et procédé de commande d'écran pour traitement d'entrée d'utilisateur à l'aide de celui-ci
WO2020085794A1 (fr) Dispositif électronique et son procédé de commande
WO2020184748A1 (fr) Dispositif d'intelligence artificielle et procédé de commande d'un système d'arrêt automatique sur la base d'informations de trafic
WO2021029457A1 (fr) Serveur d'intelligence artificielle et procédé permettant de fournir des informations à un utilisateur
WO2020230933A1 (fr) Dispositif d'intelligence artificielle pour reconnaître la voix d'un utilisateur et procédé associé
WO2020213758A1 (fr) Dispositif d'intelligence artificielle à interactivité locutoire et procédé associé
WO2019198865A1 (fr) Réfrigérateur et procédé d'affichage d'une interface utilisateur sur un réfrigérateur, terminal d'utilisateur et procédé pour exécuter une fonction dans un terminal d'utilisateur
EP3523709A1 (fr) Dispositif électronique et procédé de commande associé

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17890733

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017890733

Country of ref document: EP

Effective date: 20190626