WO2018128317A1

WO2018128317A1 - Refrigerator and information display method thereof

Info

Publication number: WO2018128317A1
Application number: PCT/KR2017/015548
Authority: WO
Inventors: 천은진; 도영수; 이형진; 조남국; 황도형
Original assignee: 삼성전자주식회사
Priority date: 2017-01-03
Filing date: 2017-12-27
Publication date: 2018-07-12

Abstract

A refrigerator comprises: a storage chamber for storing food; a temperature sensing unit for sensing the temperature inside the storage chamber; a cooling unit for supplying cool air to the storage chamber; a display for displaying information; a microphone for receiving a voice; at least one processor electrically connected with the temperature sensing unit, the microphone, and the display; and a memory electrically connected with the at least one processor. The memory stores at least one instruction set such that the processor displays, on the display: a food list including food information, corresponding to a food name, and an identification indicator capable of identifying the food information, when a first voice including the food name is recognized through the microphone; and purchase information of at least one food corresponding to the identification indicator when a second voice indicating the identification indicator is recognized through the microphone.

Description

Refrigerator and its information display method

The disclosed invention relates to a refrigerator and a method of displaying information of a refrigerator, and more particularly, to a refrigerator and a method of displaying information of a refrigerator that can communicate with an external device.

In addition, the present disclosure relates to an artificial intelligence (AI) system and its application for simulating functions such as recognition and judgment of the human brain using a machine learning algorithm.

Recently, a refrigerator includes a display for displaying a temperature of a storage compartment and an operation mode of the refrigerator.

Such a display not only enables a user to easily acquire image information using a graphical user interface, but also allows a user to intuitively input a control command using a touch panel. In other words, the recent display performs not only an information display function but also an information input function.

In addition, recent refrigerators include a communication module for connecting to an external device (eg, a server connected to the Internet).

The refrigerator may access the Internet through a communication module, obtain various information from various servers connected to the Internet, and provide various services based on the obtained information. For example, the refrigerator may provide not only information related to food such as food information and recipe of food through the Internet, but also various services such as internet shopping.

As such, the refrigerator provides more various services to the user through the display and the communication module.

Recently, artificial intelligence technology, which is a system in which a machine learns itself, judges itself, and becomes smart, unlike a conventional rule-based smart system, is spreading. An artificial intelligence system is a computer system that implements human-level intelligence. Unlike conventional rule-based smart systems, an artificial intelligence system is a machine that learns, judges, and becomes smart. The more artificial intelligence systems are used, the better the recognition rate and the more accurate the user's taste will be. Therefore, the existing rule-based smart system is gradually being replaced by artificial intelligence system based on deep learning.

Artificial intelligence techniques may include elementary techniques utilizing machine learning (eg, deep learning) and machine learning. Machine learning is an algorithmic technique for classifying and learning features of input data. Element technology is a technology using a machine learning algorithm, and may be composed of technical fields such as linguistic understanding, visual understanding, inference / prediction, knowledge representation, and motion control.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technique for recognizing and applying / processing human language / characters, including natural language processing, machine translation, dialogue system, question and answer, speech recognition / synthesis, and the like. Visual understanding is a technology that recognizes and processes objects as human vision, and includes object recognition, object tracking, image retrieval, person recognition, scene understanding, spatial understanding, and image enhancement. Inference / prediction is a technique for determining, logically inferring, and predicting information. It includes knowledge / probability-based inference, optimization prediction, preference-based planning, and recommendation. Knowledge representation is a technology that automates human experience information into knowledge data, and includes knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle, movement of a robot, and the like, and includes motion control (navigation, collision, driving), operation control (action control), and the like.

One aspect of the disclosed invention provides a refrigerator capable of receiving a command through voice using voice recognition technology and outputting content in voice.

One aspect of the disclosed invention provides a refrigerator capable of providing internet shopping by command via voice.

One aspect of the disclosed invention provides a refrigerator capable of identifying a plurality of users and providing content suitable for the identified user.

A refrigerator according to the disclosed embodiment may include a microphone; A display for displaying information according to a voice input through the microphone; And when a first voice including a food name is recognized, a list including food information having a name corresponding to the food name and a label for identifying the food, on the display, and displaying at least one of the labels. And a controller for displaying food information indicated by a label included in the second voice on the display when the second voice to be referred to is recognized.

In addition, when the first voice is recognized, the controller may display the list in the form of a card type user interface on the user interface of the application displayed on the display.

In addition, when the second voice is recognized, the controller may execute an application for providing food information indicated by a label included in the second voice and display the food information provided by the application on a display.

The controller may further include a communication unit, and when the first voice is recognized, the controller transmits the first voice data to the server through the communication unit, and transmits information about the list transmitted from the server in the communication unit. Upon receipt, the list can be displayed on the display.

The controller may further include a communication unit. When the second voice is recognized, the controller transmits the data of the second voice to the server through the communication unit, and receives the food information transmitted from the server in the communication unit. The food information may be displayed on the display.

The controller may further include a communication unit, and when the first voice is recognized, the controller transmits the first voice data to the server through the communication unit, and analyzes the analysis information on the first voice transmitted from the server. When received by the communication unit, an instruction according to the analysis information may be transmitted to an application related to performing a task according to the analysis information, so that the application may display the list.

The controller may further include a communication unit, and when the second voice is recognized, the controller transmits the data of the second voice to the server through the communication unit, and analyzes the analysis information on the second voice transmitted from the server. When received by the communication unit, an instruction according to the analysis information may be transmitted to an application related to performing work according to the analysis information, so that the application may display the food information.

In addition, when the first voice is recognized and the list is displayed on the display, a voice indicating that the list is displayed is output. When a second voice is recognized and the food information is displayed on the display, the food information is displayed. It may further include a speaker for outputting a voice indicating.

A refrigerator according to the disclosed embodiment may include a microphone; A display for displaying information according to a voice input through the microphone; A speaker for outputting voice; And when a voice is input through the microphone, performs a command indicated by the input voice for the application displayed on the display, and if the command indicated by the input voice cannot be performed for the application, to the applications related to the command. And a controller for performing the command with respect to another application according to a predetermined priority.

In addition, when the controller cannot perform a command indicated by the input voice to the application, the controller may output a voice indicating that the command cannot be performed to the application through the speaker.

In addition, when there is no priority, the controller may output a voice requesting confirmation of whether to execute the command with respect to another application through a speaker.

In addition, when an answer to the confirmation request is input, the controller may execute a command indicated by the voice for an application indicated by the input answer.

In addition, when the controller cannot execute a command indicated by the input voice to the application, the controller determines an application to execute the command according to a predetermined priority for applications related to the command, and determines the application to the determined application. The command can be performed for

The controller may perform a command indicated by the input voice for an application displayed on the display if the object inputted by the microphone does not include a target to perform the command indicated by the voice.

A refrigerator according to the disclosed embodiment may include a microphone; A display for displaying information according to a voice input through the microphone; A speaker for outputting voice; And when a voice is input through a microphone, when user identification is required in performing a command indicated by the voice, displaying a user registered in advance on the display and outputting a voice through the speaker requesting to select from the displayed users. Controller;

In addition, when a voice for selecting at least one user from among the displayed users is input, the controller may perform a command indicated by the voice in relation to the selected user, and display the result on the display.

In addition, when the user identification is required to perform the command indicated by the voice, the controller may display a user mark which is identified in advance on the display and each user.

In addition, when a voice for selecting at least one of the marks is input, the controller may perform a command indicated by the voice in relation to the user indicated by the mark and display the result on the display.

In addition, when a voice is input through a microphone, when the voice includes a representation representing a user, the controller performs a command indicated by the voice in relation to a user included in the voice and displays the result on the display. can do.

In addition, when a voice is input through a microphone, the controller may perform a command indicated by the voice in relation to a user represented by voice data matching the input voice among prestored voice data and display the result on the display. have.

An aspect of the disclosed invention can provide a refrigerator capable of receiving a command through voice using voice recognition technology and outputting content in voice.

One aspect of the disclosed invention may provide a refrigerator capable of providing internet shopping with a command via voice.

One aspect of the disclosed invention may provide a refrigerator capable of identifying a plurality of users and providing content suitable for the identified user.

A refrigerator according to the disclosed embodiment includes a storage compartment for storing food; A temperature sensor for sensing a temperature inside the storage compartment; Cooling unit for supplying cold air to the storage compartment; A display for displaying information; A microphone for receiving a voice; A display for displaying information; At least one processor electrically connected to the temperature sensing unit, the microphone, and the display; And a memory electrically connected to the at least one processor.

The memory may include a food list including food information corresponding to the food name and an identification mark for identifying the food information, when the processor recognizes the first voice including the food name through the microphone. If the second voice indicating the identification mark is recognized on the display and the microphone is recognized, the at least one instruction set to display purchase information of at least one food item corresponding to the identification mark is stored on the display. .

In addition, when the first voice including the food name is input through the microphone, the memory acquires the food name by recognizing the first voice using a learning network model trained according to an artificial intelligence algorithm. And at least one instruction set to display the food list including the food information corresponding to the food name on the display. In this case, the learning network model may be trained using a plurality of voices and a plurality of words corresponding to each of the plurality of voices.

The memory may be configured to recognize the food name and the food by recognizing the first voice using a learning network model trained according to an artificial intelligence algorithm when the first voice including the food name is input through the microphone. Obtain at least one instruction configured to acquire user information uttering a first voice and display the food list corresponding to the food name on the display, the food list including the food information related to the user information. In this case, the learning network model may be trained using a plurality of voices and user information corresponding to each of the plurality of voices.

In addition, the memory, if the first voice is recognized, the processor identifies whether the food associated with the food name in the storage compartment exists in the storage compartment, and if the food is present in the storage compartment, the food is the At least one instruction set to display on the display information indicating what is present in the storage compartment may be stored.

The memory may be further configured to identify whether or not a food associated with the food name exists in the storage compartment when the processor recognizes the first voice, and if the food does not exist in the storage compartment, the processor identifies the food name. At least one instruction set to display the food list including the corresponding food information on the display may be stored.

The memory may store at least one instruction configured to display the food list on a user interface of an application displayed on the display in a form of a card type user interface when the first voice is recognized. Can be.

The memory may be configured such that, when the second voice is recognized, the processor executes an application for providing food information indicated by a label included in the second voice to display food information provided by the application on a display. At least one instruction may be stored.

The memory device may further include a communication unit, wherein the processor transmits data of the first voice to a server through the communication unit when the first voice is recognized, and is transmitted from the server in the communication unit. When receiving the information on the food list, it is possible to store at least one instruction set to display the food list on the display.

The apparatus may further include a communication unit, wherein the memory transmits the data of the second voice to the server through the communication unit when the second voice is recognized, and is transmitted from the server in the communication unit. Upon receiving food information, at least one instruction set to display the food information on the display may be stored.

The apparatus may further include a communication unit, and, when the processor recognizes the first voice, the memory transmits the first voice data to the server through the communication unit, and transmits the first voice to the server. When the communication information is received by the communication unit, an instruction according to the analysis information is transmitted to an application related to performing the work according to the analysis information, and the application may store at least one instruction set to display the food list. .

The apparatus may further include a communication unit, and, when the processor recognizes the second voice, transmits data of the second voice to the server through the communication unit, and transmits the second voice to the server. When receiving the analysis information on the communication unit, and transmits an instruction according to the analysis information to the application associated with the performance of the operation according to the analysis information, the application may store at least one instruction set to display the food information. .

The apparatus may further include a speaker, wherein the memory outputs a voice indicating that the list is displayed through the speaker when the processor recognizes the first voice and displays the food list on the display. When a voice is recognized and the food information is displayed on the display, at least one instruction set to output a voice indicating that the food information is displayed through the speaker may be stored.

A refrigerator according to the disclosed embodiment includes a storage compartment for storing food; A temperature sensor for sensing a temperature inside the storage compartment; Cooling unit for supplying cold air to the storage compartment; A microphone for receiving a voice; A display for displaying information; At least one processor electrically connected to the temperature sensing unit, the microphone, and the display; And a memory electrically connected to the at least one processor.

Further, when the voice is input through the microphone, the memory performs a command indicated by the input voice for the application displayed on the display, and performs a command indicated by the input voice for the application. If not, store at least one instruction set to execute the command for another application according to a predetermined priority for the applications associated with the command.

The apparatus may further include a speaker, wherein the memory is further configured to generate a voice indicating that the processor cannot perform the command for the application if the processor cannot perform a command indicated by the input voice for the application. At least one instruction set to be output through the speaker may be stored.

The apparatus may further include a speaker, wherein the memory is configured to output, through the speaker, a voice requesting confirmation whether the processor performs the command for another application when the processor does not have the priority. You can store one instruction.

The memory may store at least one instruction set by the processor to execute a command indicated by the voice to an application indicated by the input response when the controller receives an answer to the confirmation request.

In addition, the memory, if the processor is unable to perform the command indicated by the input voice for the application, determines the application to perform the command according to a predetermined priority for the applications associated with the command and At least one instruction configured to execute the command with respect to the determined application may be stored.

Also, the memory may be configured to cause the processor to perform a command indicated by the input voice for an application displayed on the display, when the voice input through the microphone does not include a target to perform the command indicated by the voice. At least one set instruction may be stored.

A refrigerator according to the disclosed embodiment includes a storage compartment for storing food; A temperature sensor for sensing a temperature inside the storage compartment; Cooling unit for supplying cold air to the storage compartment; A microphone for receiving a voice; A display for displaying information; A speaker for outputting voice; And at least one processor electrically connected to the temperature sensing unit, the microphone, the speaker, and the display. And a memory electrically connected to the at least one processor.

In addition, when the voice is input through the microphone, the memory is configured to display a user registered in advance on the display and select from among the displayed users, when user identification is required to perform a command indicated by the voice. Store at least one instruction configured to output the desired voice through the speaker.

The memory is further configured to perform a command indicated by the voice in relation to the selected user when the processor inputs a voice for selecting at least one user among the displayed users, and display the result on the display. At least one instruction may be stored.

The memory may further include at least one instruction configured to display, by the processor, a user identifying a user who is registered in advance on the display and a beacon for identifying each user when user identification is required to perform a command indicated by the voice. Can be stored.

The memory may further include at least one set by the processor to perform a command indicated by the voice in relation to a user indicated by the mark and to display a result on the display when a voice for selecting at least one of the marks is input. Instructions can be stored.

In addition, when the voice is input through the microphone, the memory performs a command indicated by the voice in relation to a user included in the voice when the voice includes a representation representing a user. At least one instruction set to display a result on the display may be stored.

In addition, when the voice is input through the microphone, the memory is further configured to perform a command indicated by the voice in relation to a user represented by voice data matching the input voice among prestored voice data, and output the result. At least one instruction set to be displayed on the display may be stored.

An information display method of a refrigerator according to an exemplary embodiment of the present disclosure may include: receiving a first voice including a food name through a microphone; Displaying a food list on a display in front of the refrigerator, the food list including food information corresponding to the food name and an identification mark for identifying the food information, based on the recognition of the first voice; Receiving a second voice indicating the identification mark through the microphone; And displaying, on the display, purchase information of at least one food item corresponding to the identification mark, based on the recognition of the second voice.

According to an aspect of the disclosed invention, it is possible to provide a refrigerator capable of receiving a command through voice using voice recognition technology and outputting a content by voice.

According to an aspect of the disclosed invention, it is possible to provide a refrigerator capable of providing internet shopping by a command via voice.

According to an aspect of the disclosed invention, it is possible to provide a refrigerator capable of identifying a plurality of users and providing content suitable for the identified user.

1 illustrates an exterior of a refrigerator according to one embodiment.

2 illustrates a front surface of a refrigerator according to one embodiment.

3 illustrates a configuration of a refrigerator according to one embodiment.

4 illustrates a display included in a refrigerator according to one embodiment.

5 illustrates a home screen displayed on a display included in a refrigerator according to one embodiment.

6 and 7 illustrate a voice recognition application displayed on a display included in a refrigerator according to one embodiment.

8 illustrates communication with an external device through a communication unit included in a refrigerator according to one embodiment.

9 and 10 illustrate a communication between a refrigerator and a server according to an embodiment.

11A to 14 illustrate a method of purchasing food by a user through a refrigerator according to one embodiment.

15 and 16 illustrate a method in which a refrigerator performs a command in response to a voice command of a user.

17 to 20 provide a method of identifying a user in a refrigerator according to one embodiment and providing information corresponding to the identified user.

21A is a block diagram of a controller for data learning and recognition according to an embodiment.

21B is a detailed block diagram of a data learner and a data recognizer, according to an exemplary embodiment.

22 is a flowchart of a refrigerator displaying information according to an embodiment.

23 is a flowchart of a network system according to an embodiment.

24 is a view illustrating a method of purchasing food by a user through a refrigerator according to another embodiment.

Configurations shown in the embodiments and drawings described herein are only exemplary embodiments of the disclosed invention, there may be various modifications that can replace the embodiments and drawings of the present specification at the time of filing of the present application.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting and / or limiting the disclosed invention.

For example, the singular forms herein may include the plural forms unless the context clearly indicates otherwise.

In addition, the terms "comprise" or "have" are intended to express the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, one or more other features, It does not exclude the possibility of further presence or addition of numbers, steps, actions, components, parts or combinations thereof.

In addition, terms including an ordinal number such as "first", "second", etc. are used to distinguish one element from another element and do not limit the one element.

In addition, terms such as "~ part", "~ group", "~ block", "~ member", "~ module" may refer to a unit for processing at least one function or operation. For example, the terms may refer to at least one hardware processed by at least one piece of hardware such as a field-programmable gate array (FPGA) / application specific integrated circuit (ASIC), at least one software stored in a memory, or a processor. have.

In addition, the term "food" in the present specification may include industrial products, products produced or hunted by the user manufactured and traded by a person or a machine.

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of the disclosed invention. Like reference numerals or signs in the accompanying drawings may refer to parts or components that perform substantially the same function.

1 illustrates an exterior of a refrigerator according to one embodiment, and FIG. 2 illustrates a front surface of the refrigerator according to one embodiment. 3 illustrates a configuration of a refrigerator according to one embodiment, and FIG. 4 illustrates a display included in the refrigerator according to one embodiment.

As shown in FIGS. 1, 2, and 3, the refrigerator 1 may include a main body 10 having a front face open and a door 30 opening and closing the open front face of the body 10. have. In addition, the main body 10 is provided with a storage compartment 20 whose front surface is opened so that food can be refrigerated or frozen.

The main body 10 may form an appearance of the refrigerator 1. The main body 10 may include an inner wound 11 forming the storage compartment 20 and an outer wound 12 coupled to an outer side of the inner wound 11 to form an outer appearance. In this case, a heat insulating material (not shown) may be filled between the inner wound 11 and the outer wound 12 of the main body 10 to prevent leakage of cold air from the storage compartment 20.

The storage compartment 20 may be divided into a plurality of compartments by the horizontal barrier rib 21 and the vertical barrier rib 22. For example, as illustrated in FIG. 1, the storage compartment 20 may be divided into an upper storage compartment 20a, a lower first storage compartment 20b, and a lower second storage compartment 20c. The storage compartment 20 may be provided with a shelf 23 on which food can be placed and a sealed container 24 for hermetically storing food.

The storage compartment 20 may be opened and closed by the door 30. For example, as shown in FIG. 1, the upper storage compartment 20a may be opened and closed by the upper first door 30aa and the upper second door 30ab, and the lower first storage compartment 20b may be the lower first. It may be opened and closed by the door 30b, and the lower second storage compartment 20c may be opened and closed by the lower second door 30c.

A handle 31 may be provided at the door 30 to easily open and close the door 30. The handle 31 may be formed long in the vertical direction along the upper first door 30aa and the upper second door 30ab and between the lower first door 30b and the lower second door 30c. . Thereby, when the door 30 is closed, the handle 31 can be seen as being integrally provided.

In addition, the refrigerator 1 may include a display 120, a storage 130, a communication unit 140, a dispenser 150, a cooling unit 150, a temperature sensing unit 170, an acoustic unit 180, and a main controller ( 110) may be included.

The display 120 can interact with the user. For example, the user may receive a user input from the user and display an image according to the received user input.

The display 120 includes a display panel 121 for displaying an image, a touch panel 122 for receiving a user input, and a touch screen controller 123 for controlling / driving the display panel 121 and the touch panel 122. can do.

The display panel 121 may convert image data received from the main controller 110 through the touch screen controller 123 into an optical image that can be viewed by the user.

The display panel 101 includes a cathode ray tube (CRT) display panel, a liquid crystal display (LCD) panel, a light emitting diode (LED) panel, and an organic light emitting diode (OLED). Diode (PDP) panels, plasma display panels (PDPs), field emission display (FED) panels and the like can be employed. However, the display panel 101 is not limited thereto, and the display panel 101 may employ various display means capable of visually displaying an optical image corresponding to the image data.

The touch panel 122 may receive a user's touch input and transmit an electrical signal corresponding to the received touch input to the touch screen controller 123.

In detail, the touch panel 122 senses a user's touch on the touch panel 122 and transmits an electrical signal corresponding to the coordinates of the user's touch point to the touch screen controller 123. As described below, the touch screen controller 123 may acquire the coordinates of the touch point of the user based on the electrical signal received from the touch panel 122.

In addition, the touch panel 122 may be positioned in front of the display panel 121. In other words, the touch panel 122 may be provided on a surface on which an image is displayed. Therefore, the touch panel 122 may be made of a transparent material so that the image displayed on the display panel 121 is not distorted.

The touch panel 122 may employ a resistive touch panel or a capacitive touch panel. However, the touch panel 122 is not limited thereto, and the touch panel 122 may detect a user's touch or approach, and output various electrical signals corresponding to the detected coordinates of the touch point or the coordinates of the access point. An input means can be employed.

The touch screen controller 123 may drive / control the operations of the display panel 121 and the touch panel 122. In detail, the touch screen controller 123 drives the display panel 121 to display an optical image corresponding to the image data received from the main controller 110, and detects the coordinates of the touch point of the user. ) Can be controlled.

According to an embodiment, the touch screen controller 123 may determine the coordinates of the user's touch point based on the electrical signal output from the touch panel 122, and transmit the coordinates of the user's touch point to the main controller 110. .

In addition, according to the embodiment, the touch screen controller 123 may transmit an electrical signal output from the touch panel 122 to the main controller 110 so that the main controller 110 determines the coordinates of the user's touch point.

The touch screen controller 123 may include a memory (not shown) that stores programs and data for controlling operations of the display panel 121 and the touch panel 122, and the display panel 121 according to the programs and data stored in the memory. And a microprocessor (not shown) that performs an operation for controlling the operation of the touch panel 122. In addition, the memory and the processor of the touch screen controller 123 may be provided as separate chips or may be provided as a single chip.

The display 120 may be installed in the door 30 for the convenience of the user. For example, as shown in FIG. 2, the display 120 may be installed in the upper second door 30ab. Hereinafter, the display 120 installed in the upper second door 30ab will be described, but the installation position of the display 120 is not limited to the upper second door 30ab. For example, if the dispenser 150 is a position that can be viewed by the user, such as the upper first door 30aa, the lower first door 30b, the lower second door 30c, and the trauma 12 of the main body 10. Can be installed anywhere.

In addition, the display 120 may have a wake up function that is automatically activated when the user approaches within a certain range. For example, when the user approaches within a certain range, the display 120 may be activated. In other words, the display 120 may be in an on state. On the other hand, if the user deviates within a certain range, the display 120 may be deactivated. In other words, the display 120 may be in an off state.

The display 120 may display various screens or images. The screen or image displayed on the display 120 is described in detail below.

The storage unit 130 may store a control program and control data for controlling the operation of the refrigerator 1 and various application programs and application data for performing various functions according to a user input. For example, the storage unit 130 may include an operating system (OS) program that manages the configuration and resources (software and hardware) included in the refrigerator 1, an image display application that displays a pre-stored image, and a pre-stored video. Video playback application to play music, music application to play music, radio application to play radio, calendar application to manage the schedule, memo application to store and display notes, online shopping mall application to buy food online and food Recipe applications that provide recipes, and the like.

In addition, the storage unit 130 may include a nonvolatile memory in which a program or data is not lost even when power is cut off. For example, the storage 130 may include a large capacity flash memory or a solid state drive (SSD) 131.

The communication unit 140 may transmit data to or receive data from an external device under the control of the main controller 110.

In addition, the communication unit 140 may include at least one of

communication modules

141, 142, and 143 that transmit and receive data according to a predetermined communication protocol. For example, the communication unit 140 may connect to an external device such as a WiFi Fidelity (WiFi®) module 141 that connects to a local area network (LAN) through a wireless access device. Bluetooth® module 142 for one-to-one communication or one-to-many communication with a few external devices, and a Zigbee module for forming a short-range communication network between a plurality of electronic devices (mainly home appliances) 143) and the like.

In addition, each of the plurality of

communication modules

141, 142, and 143 transmits a radio signal to or receives radio signals from free space, an antenna for modulating data to be transmitted, and a demodulator / demodulator for demodulating the received radio signals. It may include.

Operation of the refrigerator 1 through the communication unit 140 is described in more detail below.

The dispenser 150 may discharge water or ice according to a user input. In other words, the user may directly take out water or ice to the outside without opening the door 30 through the dispenser 150.

The dispenser 150 includes a dispenser lever 151 for receiving a discharge command from a user, a dispenser nozzle 152 for discharging water or ice, a flow path 153 for guiding water from the external water supply to the dispenser nozzle 152, and discharge It may include a filter 154 for purifying the water to be dispensed and the dispenser display panel 155 for displaying the operating state of the dispenser 150.

The dispenser 150 may be installed outside the door 30 or the main body 10. For example, as illustrated in FIG. 2, the dispenser 150 may be installed in the upper first door 30aa. Hereinafter, the dispenser 150 installed in the upper first door 30aa will be described, but the dispenser 150 is not limited to being installed in the upper first door 30aa. For example, the dispenser 40 draws water or ice from the user, such as the upper second door 30ab, the lower first door 30b, the lower second door 30c, and the external wound 12 of the body 10. It can be installed anywhere you can.

For example, the door 30 or the outer box 12 may be provided with a cavity 150a recessed inwardly to form a water or ice extraction space, and the cavity 150a may have a dispenser nozzle 152 and a dispenser lever. 151 may be provided. When the user presses the dispenser lever 151, water or ice is discharged from the dispenser nozzle 152.

In particular, when water is discharged through the dispenser nozzle 152, the water may flow from an external water supply source (not shown) along the flow path 152 to the dispenser nozzle 152. Water may also be purified by filter 153 while flowing to dispenser nozzle 152.

In this case, the filter 153 may be detachably provided in the main body 10 or the door 30 so that the filter 153 may be replaced when the life of the filter 153 expires.

The cooling unit 160 may supply cold air to the storage chamber 20.

Specifically, the cooling unit 160 may maintain the temperature of the storage compartment 20 within a predetermined range by using evaporation of the refrigerant.

The cooling unit 160 may include a compressor 161 for compressing the gas refrigerant, a condenser 162 for converting the compressed gas refrigerant into a liquid refrigerant, an expander 163 for reducing the liquid refrigerant, and a reduced liquid refrigerant in a gas state. It may include an evaporator 164 to change the state.

In particular, the cooling unit 160 may supply cold air to the storage compartment 20 by using a phenomenon in which the pressure-reduced liquid refrigerant absorbs thermal energy of ambient air while changing its state into a gas state.

However, the configuration of the cooling unit 160 is not limited to the compressor 161, the condenser 162, the expander 163, and the evaporator 164.

For example, the cooling unit 160 may include a Peltier element using the Peltier effect. The Peltier effect means that when a current flows through contact surfaces of different metals, an exothermic phenomenon occurs in one metal and an endothermic phenomenon occurs in another metal. The cooling unit 160 may supply cold air to the storage compartment 102 using the Peltier element.

As another example, the cooling unit 160 may include a magnetic cooling device using a magneto-caloric effect. The magnetocaloric effect means that heat is released when a specific material (magnetic calorie material) is magnetized, and absorbs heat when the specific material (magnetic calorie material) is demagnetized. The cooling unit 160 may supply cold air to the storage compartment 20 by using a magnetic cooling device.

The temperature sensing unit 170 may be provided in the storage compartment 20 to sense a temperature inside the storage compartment 20. The temperature sensing unit 170 may include a plurality of temperature sensors 171 installed in the plurality of

storage chambers

20a, 20b, and 20c, respectively. In addition, each of the plurality of temperature sensors 171 may include a thermistor whose electrical resistance changes with temperature.

The sound unit 180 converts an electrical signal received from the main controller 110 into an acoustic signal, converts an electrical signal into a speaker 181 for outputting the converted acoustic signal, and converts an electrical signal into a main controller ( It may include a microphone 182 that outputs to 110.

The main controller 110 may include a display 120, a storage 130, and a communication unit included in the refrigerator 1 according to a user input and / or a program and data stored in the storage 130 received through the display 120. The controller 140 may control the dispenser 150, the cooling unit 150, the temperature sensing unit 170, and the acoustic unit 180.

The main controller 110 may include a microprocessor 111 that performs operations for controlling the refrigerator 1, and a memory 112 that stores / stores programs and data related to operations of the microprocessor 111. Can be.

The microprocessor 111 may load data stored / stored in the memory 112 according to a program stored / stored in the memory 112 and perform an arithmetic operation or a logical operation on the retrieved data. Also, the microprocessor 111 may output the result of an arithmetic operation or a logical operation to the memory 112.

The memory 112 may include a volatile memory that loses stored data when power supply is interrupted. The volatile memory may load a program and data from the storage 130 described above, and temporarily store the loaded data. In addition, the volatile memory can provide stored programs and data to the microprocessor 111, and can store data output from the microprocessor 111. Such a volatile memory may include an S-RAM, a D-lap, and the like.

In addition, the memory 112 may include a nonvolatile memory as necessary. The nonvolatile memory can retain the stored data when the power supply is interrupted. The nonvolatile memory may store firmware for managing and initializing various components included in the refrigerator 1. Such nonvolatile memories may include ROM, EPYROM, EPYROM, flash memory, and the like.

In addition, the main controller 110 may include a plurality of microprocessors 111 and a plurality of memories 112. For example, the first microprocessor and the first memory for controlling the temperature sensor 170, the dispenser 150, and the cooling unit 160 of the refrigerator 1, the display 120 of the refrigerator 1, and a storage unit. 130, a second microprocessor and a second memory for controlling the communicator 140 and the acoustic unit 180 may be included.

Although the microprocessor 111 and the memory 112 have been described functionally and distinctly, the microprocessor 111 and the memory 112 are not necessarily physically distinguished. For example, the microprocessor 111 and the memory 112 may be implemented not only as separate chips but also as one chip.

As such, the main controller 110 may control the overall operation of the refrigerator 1, and the operation of the refrigerator 1 described below may be interpreted as being performed by the control of the main controller 110.

In the above description, the main controller 110, the storage unit 130, and the communication unit 140 have been functionally distinguished and described, but the main controller 110, the storage unit 130, and the communication unit 140 are not necessarily physically distinguished. . For example, the main controller 110, the storage unit 130, and the communication unit 140 may be implemented as one chip, or each of the main controller 110, the storage unit 130, and the communication unit 140 may be a separate chip. Can be implemented.

In addition, the display 120, the storage unit 130, the communication unit 140, the dispenser 150, the cooling unit 150, the temperature sensing unit 170, and the sound unit 180 included in the refrigerator 1 are described above. Although the main controller 110 has been described, a new configuration may be added or some components may be omitted as necessary.

When power is supplied to the refrigerator 1, the main controller 110 may control the display 120 to display the home screen 200 as illustrated in FIG. 5.

On the home screen 200, an application stored in the time / date display area 210 displaying the time and date, the operation information display area 220 displaying the operation information of the refrigerator 1, and the storage 130 is executed. A plurality of launcher 230 for may be displayed.

In the time / date display area 210, current time information and today's date information may be displayed. In addition, the time / date display area 210 may display location information (eg, a name of a country or city) of a location where the refrigerator 1 is located.

In the operation information display area 220, a storage map 221 related to operations of the plurality of

storage rooms

20a, 20b, and 20c included in the refrigerator 1 may be displayed.

The storage map 221 may display information related to operations of the plurality of

storage rooms

20a, 20b, and 20c included in the refrigerator 1. For example, as illustrated in FIG. 5, an upper storage compartment 20a, a lower first storage compartment 20b, and a lower second storage compartment 20c may be divided and displayed in the storage compartment map 221, and the upper storage compartment 20a may be displayed. ), A target temperature of the lower first storage compartment 20b and a target temperature of the lower second storage compartment 20c may be displayed.

In this case, when the user touches an area representing each of the

storage compartments

20a, 20b, and 20c, the main controller 110 displays an image for displaying a target temperature of each of the

storage compartments

20a, 20b, and 20c. Can be marked on. For example, when a user touches an area indicating the upper storage compartment 20a in the storage compartment map 221, an image for setting a target temperature of the upper storage compartment 20a may be displayed on the display 120.

In addition, the operation information display area 220 may display a timer setting icon 222 and a refrigerator setting icon 223 for executing an application for controlling the operation of the and refrigerator 1.

When the timer setting icon 222 is touched by the user, a timer setting screen for setting a target time of the timer may be displayed on the display 120. For example, the user may input the time to output the alarm or the time interval until the alarm is output through the timer setting image, and the refrigerator 1 outputs the alarm at the time input by the user or the user. When the input time interval elapses, an alarm can be output.

When the refrigerator setting icon 223 is touched by the user, the main controller 110 may display, on the display 120, an operation setting screen for inputting a setting value for controlling the operation of the refrigerator 1. For example, a user may set a target temperature of each of the plurality of

storage compartments

20a, 20b, and 20c included in the refrigerator 1 through an operation setting screen, and which of the water and ice is discharged through the dispenser 150. You can set whether or not.

In addition, the home screen 200 may display a plurality of launcher 230 for executing various applications stored in the storage 130 by the user.

For example, the home screen 200 includes an album launcher 231 for executing an album application displaying a picture stored in the storage 130, and a recipe launcher 232 for executing a recipe application providing a recipe for food. The screen setting launcher 233 for executing the screen setting application for controlling the operation of the display 120 may be displayed.

In addition, the home screen 200 sets the operation of the home appliance control launcher 234 for executing the home appliance control application for controlling various home appliances through the refrigerator 1, and the operation of the voice output application for outputting various contents as a voice. The voice output setting launcher 235 and the online shopping launcher 236 for executing a shopping application for online shopping may be displayed.

As described above, the home screen 200 of the refrigerator 1 may display main information related to the operation of the refrigerator 1 and a launcher for executing various applications.

However, the drawing illustrated in FIG. 5 is only an example of the home screen 200, and the refrigerator 100 may display various types of home screens according to a user's setting, and the information and the launcher displayed on the home screen. Again not limited to the bar shown in FIG.

6 illustrates a voice recognition UI displayed on the display 120 of the refrigerator according to one embodiment. Hereinafter, the main controller will be referred to as a controller for convenience.

When a preset wake word is input through the microphone 182, the controller 110 may display a voice recognition user interface 250 (hereinafter, referred to as a UI) on the display 120. The speech recognition UI 250 may be displayed on the display 120 as a UI having a configuration as illustrated in FIG. 6. The wake word may be composed of a preset word or a combination of words, and may be changed to a desired expression by the user.

As described above, the controller 110 may execute the voice recognition function through the wake word uttered by the user. As shown in FIG. 7, when the microphone-shaped button 271 displayed on the public UI is touched, the voice recognition function is performed. You can also run

The shape of the launcher representing the button for executing the voice recognition function is not limited to the shape of the microphone, but may be implemented in various other forms of images.

The notification UI may be displayed on the display 120 when a touch gesture for swiping upward while a specific point at the bottom of the display 120 is touched is input. The launcher for executing the voice recognition function may be displayed on the notification UI or may be displayed on the home screen 200 of the display 120.

The controller 110 recognizes and analyzes the user's voice in natural language using a voice recognition function to determine the user's intention, and can execute a command according to the user's intention. In addition, by outputting the voice process or the result of the command according to the user's intention so that the user can recognize the voice process or the result of the voice command.

When the voice recognition function is executed, the voice recognition UI 250 may be displayed on the display 120, as shown in FIGS. 6 and 7. The speech recognition UI 250 may include a first region 252 in which a word or sentence output as speech through a text to speech (TTS) function is displayed, a second region 254 in which a state of a speech recognition function is displayed, and a TTS. A third area 260 indicating that speech is output through the text to speech function, a setting object 256 for setting the speech recognition function, and a help object 258 for providing help for using the speech recognition function. ) May be included.

The second area 254 in which the status of the voice recognition function is displayed may display, in text, a state in which the user's voice is being input, a standby state for the user's voice input, or a state in which a process according to the voice input is being performed. Can be. For example, a state in which the user's voice is input may be displayed as <listening>, a standby state as <stand by>, and a state in which the process is performed as <processing>. The above-described English text is merely an example, and the status may be displayed in Korean text, or may be displayed as an image instead of text.

As illustrated in FIGS. 6 and 7, the voice recognition UI 250 may be displayed in the form of a card UI popped up on the home screen 200 or displayed on the entire screen of the display 120. May be

When the user's voice is input through the microphone 182, the controller 110 transmits voice data to the first server SV1 to be described later, and receives information analyzed by the voice from the first server SV1. You can run the command.

8 illustrates communication with an external device through a communication unit included in a refrigerator according to an embodiment, and FIGS. 9 and 10 illustrate communication between a refrigerator and a server according to an embodiment.

The refrigerator 1 may communicate with various electronic devices as well as the servers SV1 and SV2 through the communication unit 140.

For example, as illustrated in FIG. 8, the refrigerator 1 may be connected to an access point AP via the communication unit 140. Specifically, the refrigerator 1 may use a connection repeater (Wi-Fi ™, IEEE 802.11), Bluetooth (Bluetooth ™, IEEE 802.15.1), Zigbee (Zigbee, IEEE 802.15.4) and the like using a wireless communication standard. AP).

At this time, the access relay (AP) may be called variously, such as "hub", "router", "switch", "gateway", a wide area network (Internet), such as the Internet (Wide Area) Network) (WAN).

In the connection repeater AP, not only a refrigerator 1 but also an air conditioner 2, a washing machine 3, an oven 4, a microwave oven 5, a robot cleaner 6, a security camera 7, an electric lamp ( 8) and various electronic devices such as the television 9 can be connected. In addition, the electronic devices 1-9 connected to the access relay AP may form a local area network (LAN).

As such, the access relay AP may connect a local area network (LAN) formed by the electronic devices 1-9 connected to the access relay AP and a wide area network (WAN) such as the Internet.

The wide area network WAN may be connected to a first server SV1 that provides information analyzed by voice data to a refrigerator and a second server SV2 operated by operators that provide information through an application mounted on the refrigerator. have. The second server SV2 may be, for example, a server selling food online through a shopping application such as a mart application mounted in a refrigerator (hereinafter referred to as a mart server), and a server providing information to a weather application (hereinafter referred to as a weather server). A server providing recipe related information to a recipe application (hereinafter referred to as a recipe server), a server providing music related information to a music application (hereinafter referred to as a music server), and the like.

In addition, a mobile terminal MT may be connected to the wide area network WAN. In this case, the mobile terminal MT may directly connect to the wide area network WAN or access the wide area network WAN through the access relay AP. For example, when the mobile terminal MT is located near the access repeater AP, the mobile terminal MT may access the wide area network WAN through the access repeater AP. In addition, when the mobile terminal MT is located far from the access relay AP, the mobile terminal MT may directly connect to the wide area network WAN through a mobile communication service provided by the mobile communication provider.

The refrigerator 1 transmits data to the first server SV1 and / or the second server SV2 through the connection repeater AP, and the first server SV1 and / or the second server SV2 and the like. Data may be received from.

For example, the refrigerator 1 may transmit voice data of a user input through the microphone 182 to the first server SV1 through the access repeater AP, and the first server SV1 may transmit the voice data. The analysis information including the user's intention derived from the analysis may be transmitted to the refrigerator.

The refrigerator 1 may transmit information analyzing the voice data of the user to the second server SV2 through the access repeater AP, and may be associated with a specific application from the second server SV2 through the access repeater AP. Information, for example, food information associated with a mart application.

As described above, the refrigerator 1 may communicate with the first server SV1 and receive analysis information of the voice data of the user through the first server SV1. In addition, the refrigerator 1 may communicate with the second server SV2 and receive information related to a specific application from the second server SV2. Hereinafter, the communication between the refrigerator and the server will be described in more detail with reference to FIGS. 9 and 10.

As illustrated in FIG. 9, when the voice command uttered by the user is input, the refrigerator 1 transmits voice data to the first server SV1 (1000). When the first server SV1 receives the voice data, the first server SV1 may analyze the voice data to derive the intention of the user (1010).

When the user's voice command is input through the microphone 182, the controller 110 of the refrigerator 1 modulates the input analog voice signal into voice data, which is a digital signal, and communicates with the first server SV1 through the communication unit 140. To send.

The controller 110 of the refrigerator 1 may modulate an analog voice signal into a digital voice signal by using pulse code modulation. For example, the controller 110 of the refrigerator 1 may input a voice command of <Notify me of the weather> from the user, and may modulate the voice command into a digital signal and transmit the same to the first server SV1.

The first server SV1 removes noise by performing preprocessing on the speech data transmitted from the refrigerator 1, and analyzes the speech data and converts the speech data into text, and automatic speech recognition (ASR) and automatic speech recognition. It may include a natural language understanding (NLU) to grasp the user's intention based on the text converted by the wealth. Alternatively, the first server SV1 may include a learning network model trained using an artificial intelligence algorithm. In this case, the first server SV1 may apply the voice data transmitted from the refrigerator 1 to the learning network model to determine (or recognize, estimate, infer, and predict) the intention of the user.

For example, the automatic speech recognition unit and the natural language processing unit of the first server SV1 may determine the user's intention that the voice data requires information on the weather of the region where the user is currently located.

The first server SV1 transmits the information analyzing the voice data to the second server SV2 (1020), and the second server SV2 transmits the information requested by the voice data according to the analysis information of the voice data. In operation 1030, it may be transmitted to SV1. The first server SV1 transmits the information transmitted from the second server SV2 to the refrigerator 1 (1040).

When the first server SV1 determines the intention of the user included in the voice data, the first server SV1 determines the second server SV2 that provides information to an application related to the user's intention, and determines the voice data to the determined second server SV2. Transmit the analyzed information.

For example, if it is determined that the intention of the user included in the voice data requires information on the weather of the region where the user is currently located, the weather server provides information to the weather application of the second server SV2. The analysis information of the voice data can be transmitted.

The second server SV2 may search for information corresponding to the user's intention based on the analysis information of the voice data and transmit the information to the first server SV1. For example, the weather server may generate current weather information of the area where the user is based on the analysis information of the voice data, and transmit the generated weather information to the first server SV1. The first server SV1 converts the information transmitted from the second server SV2 into a JSON (JavaScript Object Notation) file format and transmits the information to the refrigerator 1.

The refrigerator 1 displays the information transmitted from the first server SV1 on the display 120 using a card type UI and outputs it by voice (1050).

When the communication unit 140 of the refrigerator 1 receives the JSON file format information transmitted from the first server SV1, the controller 110 of the refrigerator 1 displays the received information as a card type UI (120). ), And the received information can be output by voice using the Text To Speech (TTS) function.

For example, when the controller 110 of the refrigerator 1 receives information about the current weather of the location where the user is from the first server SV1 as a JSON file, the controller 110 may receive information including the location, time, and weather of the user. The card UI may display the display 120. In addition, the controller 110 may output the weather information displayed by the card UI as a voice (for example, <today's current weather is 20 degrees, humidity is 50%, and it is clear without clouds>). have.

Meanwhile, referring to FIG. 10, when a voice command uttered by a user is input, the refrigerator 1 transmits voice data to the first server SV1 (1100). When the first server SV1 receives the voice data, the first server SV1 may analyze the voice data to derive the intention of the user (1110).

The controller 110 of the refrigerator 1 may modulate an analog voice signal into a digital voice signal by using pulse code modulation. For example, when the voice command “Listen to next song” is input from the user, the controller 110 of the refrigerator 1 may modulate the voice command into a digital signal and transmit the same to the first server SV1. Alternatively, when the voice command “tell me apple pie recipe” is input from the user, the controller 110 of the refrigerator 1 may modulate the voice command into a digital signal and transmit it to the first server SV1.

The first server SV1 performs preprocessing on the speech data transmitted from the refrigerator 1 to remove noise, analyzes the speech data, and converts the speech data into text, and automatic speech recognition (ASR) and automatic speech recognition. It may include a natural language understanding (NLU) to grasp the user's intention based on the text converted by the wealth. Alternatively, the first server SV1 may include a learning network model trained using an artificial intelligence algorithm. In this case, the first server SV1 may apply the voice data transmitted from the refrigerator 1 to the learning network model to determine (or recognize, estimate, infer, and predict) the intention of the user.

For example, the automatic speech recognition unit and the natural language processing unit of the first server SV1 may grasp the user's intention that the speech data requires the reproduction of the next song of the song currently being played. Alternatively, the automatic voice recognition unit and the natural language processing unit of the first server SV1 may recognize the user's intention that the voice data is required to inform the user of how to cook the apple pie.

The first server SV1 transmits information analyzing the voice data to the refrigerator 1 (1120), and the refrigerator 1 transmits analysis information of the voice data to the second server SV2 (1130). The second server SV2 may transmit the information requested by the voice data to the refrigerator 1 according to the analysis information of the voice data (1140).

The first server SV1 analyzes the voice data, and when the user's intention is determined, converts the analysis information of the voice data of the user into a JSON (JavaScript Object Notation) file format and transmits it to the refrigerator 1.

When the communication unit 140 of the refrigerator 1 receives the analysis information in the JSON file format transmitted from the first server SV1, the controller 110 of the refrigerator 1 determines a related application based on the analysis information, The analysis information can be output to the determined application.

For example, if the controller 110 of the refrigerator 1 determines that it is the user's intention to request the playback of the next song of the currently playing song based on the analysis information in the JSON file format, the user's intention of the music application is determined. This may be determined by an application that performs a function related to. The controller 110 may transmit a command for requesting reproduction of the next song to the determined music application.

Alternatively, when the controller 110 of the refrigerator 1 determines the user's intention to inform the cooking method of the apple pie based on the analysis information in the JSON file format, the application performs a recipe application function related to the user's intention. Can be determined. The controller 110 may transmit a command requesting information on a method of cooking an apple pie to the determined recipe application.

When the analysis information is output to the determined application, the determined application may transmit the analysis information to the second server SV2 that provides the information. The second server SV2 may search for information corresponding to the user's intention based on the analysis information of the voice data and transmit the information to the refrigerator 1.

For example, in the music application, when receiving the analysis information output from the controller 110, the music application may request information about the next song from the second server SV2 that provides the information to the music application, that is, the music server. have.

The music server may determine the next song of the song currently being played based on the analysis information of the voice data, and provide the music application with information about the determined next song.

Alternatively, when receiving the analysis information output from the controller 110 in the recipe application, the recipe application may request information on how to cook the apple pie to the second server (SV2) that provides information to the recipe application, that is, the recipe server. have. The recipe server may provide a recipe application of a pre-stored apple pie to a recipe application based on analysis information of voice data, or provide a recipe application by searching for another cooking method of apple pie.

When the information transmitted from the second server SV2 is received, the refrigerator 1 may execute the related application according to the received information (1150).

When the communication unit 140 of the refrigerator 1 receives the information transmitted from the second server SV2, the controller 110 executes the determined application according to the information transmitted from the second server SV2, and then executes the determined application. Information corresponding to voice may be provided.

For example, the controller 110 executes a music application when the information about the next song transmitted from the music server is received. The music application may stop playing the currently playing song and play the next song according to the information about the next song transmitted from the music server. The display 120 of the refrigerator 1 may display a UI of a music application to display information related to playing the next song, and the output of the song currently being played through the speaker 181 of the refrigerator 1 is stopped. The next song can be output.

Alternatively, when the controller 110 receives the information on the method of cooking the apple pie transmitted from the recipe server, the controller 110 executes the recipe application. The display 120 of the refrigerator 1 may display a UI of a recipe application to display an apple pie cooking method, and a voice for reading the recipe of the apple pie through the speaker 181 of the refrigerator 1 may be output. Can be.

As illustrated in FIG. 9, information related to a voice of a user may be received through communication between the first server SV1 and the second server SV2, and the refrigerator 1 and the second server may be received as illustrated in FIG. 10. Information related to the voice of the user may be received through communication between the SV2s.

As shown in FIG. 11A, a voice for ordering a particular food item with a wake word may be input through the microphone 182 (900). When the wake word is recognized, the controller 110 executes a voice recognition function to analyze the user's voice, and receives a list 300 of foods included in the user's voice from a related application (for example, a mart application). In operation 901, the display 120 displays the display 120 in the form of a display.

In the method illustrated in FIG. 9, the refrigerator 1 analyzes a user's voice through communication with the first server SV1 to determine the user's intention, and information related to the user's intention transmitted from the second server SV2. (Eg, a list of foods) may be received through the first server SV1. Then, using the received information as an input value of the application for displaying a food list, as shown in Figure 11A can display the food list 300 in the form of a card UI. Here, the voice analysis result of the user may be, for example, a food name. The food name may include not only the exact name of the food, but also a idiom or slang that refers to the food, a portion of the food name, and a name similar to the food name. In addition, the food name may be an alias registered or uttered mainly by the user of the refrigerator 1.

The food list 300 may be displayed in the form of a card UI at the bottom of the speech recognition UI 250. The food list 300 includes a representative image 314 representing a food, food information 316 including a manufacturer, a food name, a dose, a quantity, a price, and the like, and an identification mark 312 to distinguish the food from other foods. , For example, a number.

In the case of the identification label, a label for distinguishing each of the plurality of foods may be used, and as shown, may be indicated by a number or may be indicated by letters.

In addition, when a user wants to check more foods than the foods included in the food list 300, a selectable tab 318 may be displayed at the bottom of the food list 300. Tab 318 may be represented by a letter indicating the function of tab 318 (see more food at the mart). When the user touches the tab 318, a mart application that provides food information may be executed and displayed on the display 120.

The controller 110 may output, through the speaker 181, a voice requesting confirmation regarding which food from among the foods included in the food list 300 is selected together with the food list 300, and a voice recognition UI. A microphone image indicating that voice is being output through the speaker 181 may be displayed in the third region 260 of the 250. In addition, the voice output through the speaker 181 may also be displayed as text in the first area 250 of the voice recognition UI.

In various embodiments, the controller 110 may recognize the first voice including the food name through the microphone 182. For example, the controller 110 may modulate the input first voice into a digital voice signal. The controller 110 may acquire a food name corresponding to the first voice by recognizing a digital voice signal through a learning network model trained using an artificial intelligence algorithm. In this case, the food name included in the first voice and the food name acquired by recognizing may be the same as or different from each other. For example, when the food name included in the first voice is an idiom, alias or part of a food name, the food name recognized and acquired may be a full name, a sales name, or a brand name.

The learning network model that recognizes the first voice is being stored in the storage 130 or in the first server SV1 that receives the first voice data from the refrigerator 1 and recognizes the first voice data. Can be.

For example, when the learning network model is being stored in the first server SV1, the controller 110 may transmit a first voice modulated by a digital voice signal to the first server SV1. The first server SV1 may input (eg, estimate, infer, predict, or grasp) a food name corresponding to the first voice by applying the first voice as an input value to a learning network model trained using an artificial intelligence algorithm. Can be. As a result of the recognition, the first server SV1 may transmit the food name corresponding to the first voice to the controller 110. As another example, when the learning network model is being stored in the storage unit 130, the controller 110 applies the first voice modulated by the digital voice signal to the learning network model stored in the storage unit 130 to generate the first voice. It can recognize the food name corresponding to the.

When a food name is obtained using the learning network model, the controller 110 includes a food list including food information 316 related to the obtained food name and an identification mark 312 for distinguishing the food information 316 from other foods. 300 may be displayed on the display 120.

Next, the controller 110 may recognize the second voice that refers to the identification mark 312 of the food list through the microphone 182. For example, the controller 110 may modulate the input second voice into a digital voice signal. In addition, the controller 110 may acquire the identification mark corresponding to the second voice by recognizing the digital voice signal through the learning network model trained using the artificial intelligence algorithm. In this case, the learning network model for recognizing the second voice may be stored in the storage 130 or in the first server SV1 analyzing the voice data.

For example, when the learning network model is being stored in the first server SV1, the controller 110 may transmit a second voice modulated by a digital voice signal to the first server SV1. The first server SV1 applies the second voice as an input value to the learning network model trained using an artificial intelligence algorithm to recognize (or estimate, infer, predict, or identify) an identification mark corresponding to the second voice. Can be. The first server SV1 may transmit an identification mark corresponding to the second voice to the controller 110. As another example, when the learning network model is being stored in the storage unit 130, the controller 110 applies the second voice modulated by the digital voice signal to the learning network model stored in the storage unit 130 to generate the second voice. Recognize an identification mark corresponding to.

When the identification mark is recognized using the learning network model, the controller 110 may display on the display 120 purchase information of at least one food indicated by the identification mark 312.

11B illustrates a method for a user to purchase food through a refrigerator according to another embodiment.

As shown in FIG. 11B, voice may be input 950 for ordering a particular food through microphone 182.

The controller 110 may recognize the first voice including the food name. As described above, as an example, the controller 110 may recognize the first voice by applying the first voice input through the microphone 182 to a learning network model trained using an artificial intelligence algorithm.

When the first voice is recognized, the controller 110 may identify whether the food 951-1 associated with the food name corresponding to the first voice exists in the storage compartment 20 (951). The food associated with the food name may be a food having the food name as at least part of the name, or a food similar or replaceable with the food name.

For example, the refrigerator 1 may store the storage information list including the food names of the foods existing in the storage room 20 in the storage unit 130. For example, the food names of the storage information list may be generated by a user input when the user stores the food in the storage compartment 20 of the refrigerator 1. Specifically, the information input by the user by voice or text may be stored as the food name, or the food name included in the identification information may be stored as the user tags the food identification information (eg, a barcode). Alternatively, the food names may be generated as a result of image recognition of the storage room 10 photographed using a camera provided in the refrigerator 1. For example, the refrigerator 1 may recognize a food name and store the recognized food name by applying an image photographed through a camera to a learning network model trained using an artificial intelligence algorithm.

The refrigerator 1 may search whether a food name corresponding to the first voice exists in the storage information list, and identify whether a food associated with the food name exists in the storage compartment 20.

As a result of identifying whether the food 951-1 associated with the food name corresponding to the first voice exists in the storage compartment 20, if the food 951-1 associated with the food name exists in the storage compartment 20, the controller 110 ) May control the display 120 to display information indicating the presence of food 951-1 (952). Information indicating the presence of food may be, for example, a video or image of the food (952-1), a notification text indicating the presence of food (e.g., 'there is a food similar to what you are looking for') 952-2, or It may include at least one of the notification sound indicating the presence of food.

On the other hand, the user may want to order food additionally. For example, the display 120 displaying information indicating the presence of the foods 951-1 may be displayed together with a UI (not shown) for further ordering of foods. In this case, in response to a user input of selecting a UI (not shown), the controller 110 may control the display 120 to display the food list 300 as shown in FIG. 11A.

11C illustrates a method of purchasing food by a user through a refrigerator according to another embodiment.

As shown in FIG. 11C, voice may be input 960 for ordering a particular food through microphone 182.

When the first voice is recognized, the controller 110 may identify whether food associated with the food name corresponding to the first voice exists in the storage compartment 20 (961). For example, the controller 110 may identify whether a food associated with a food name exists in the storage compartment 20 based on an image of the storage compartment 20 photographed using a camera provided in the refrigerator 1. Alternatively, a storage information list for foods existing in the storage compartment 20 may exist in the storage 130 of the refrigerator 1. In this case, the controller 110 may search whether the food name exists in the list of stored foods and identify whether the food associated with the food name exists in the storage compartment 20.

As a result of identifying whether a food associated with a food name corresponding to the first voice exists in the storage compartment 20, and if a food associated with the food name does not exist in the storage compartment 20, the controller 110 may purchase a food name to purchase the food. The display 120 may display a food list 962-1 including food information having a name corresponding to and a label for identifying the food information (962). For example, the food list 962-1 may be displayed in a card UI form at the bottom of the voice recognition UI.

11D illustrates a method for a user to purchase food through a refrigerator according to another embodiment.

As shown in FIG. 11D, voice may be input 970 to order a particular food through microphone 182.

The controller 110 may recognize the first voice including the food name. As described above, as an example, the controller 110 may recognize the first voice by applying the first voice input through the microphone 182 to a learning network model trained using an artificial intelligence algorithm. In addition, the controller 110 recognizes a user who uttered the first voice by applying the first voice input through the microphone 182 to the learning network model trained using an artificial intelligence algorithm, and utters the first voice. User information can be obtained.

For example, when the learning network model is being stored in the first server SV1, the controller 110 may transmit a first voice modulated by a digital voice signal to the first server SV1. The first server SV1 may recognize (or estimate, infer, predict, or grasp) a user who spoke the first voice by applying the first voice as an input value to a learning network model trained using an artificial intelligence algorithm. have. The first server SV1 may transmit the user information that uttered the first voice to the controller 110.

As another example, when the learning network model is being stored in the storage unit 130, the controller 110 applies the first voice modulated by the digital voice signal to the learning network model stored in the storage unit 130 to generate the first voice. It can recognize the user who uttered.

As a result of the application of the learning network model, when the food name and the user information are obtained, the controller 110 may obtain food information preferred by the user who spoke the first voice in relation to the food name (971). For example, when there are a plurality of users who use the refrigerator 1, the same food name may have different preferred foods for each user depending on the manufacturer of the food, the type of food or the capacity of the food, and the place of sale of the food. Accordingly, the storage unit 130 of the refrigerator 1 may store a food information list 971-1 for each user, in which preferred food information for each user is registered for the same food name. The food information for each user may be determined based on a purchase history purchased by each user, an input history registered directly by the user, a food information history corresponding to an identification mark selected from identification labels corresponding to the food name, and the like. In addition, the food information list for each user may be stored in the food information according to the preferred priority for each user. For example, food information whose priority is determined according to the order of the purchase history of the user may be stored.

In this case, the controller 110 may acquire food information preferred by a specific user based on the food information list stored in the storage 130 (971). Alternatively, the controller 110 may apply food name and user information to a learning network model learned using an artificial intelligence algorithm to obtain food information preferred by a specific user. In this case, the learning network model may be a learning network model trained using the above-described purchase history, input history, and food information history corresponding to the selected identification mark.

When the food name and food information preferred by the user who uttered the first voice are obtained, the controller 110 may include a food list including an identification mark for distinguishing the food information that the user prefers from other food information that the user prefers. 972-1) may be displayed on the display 120. In this case, the food list 972-1 may sort the food information preferred by the user according to the user's preferred ranking.

As illustrated in FIG. 12, in response to a voice requesting confirmation of which food from among foods included in the food list, a user may utter a voice for selecting a specific food (902). For example, the user may utter the identification label to distinguish foods.

The user may utter the first or first one to select an identification label to select the first food in the food list (903, 904).

The controller 110 may analyze the user's voice through a voice recognition function. In this case, as shown in FIG. 10, the controller 110 transmits the voice data of the user to the first server SV1 and receives information analyzed by the first server SV1 to analyze the intention of the user. I can figure it out. Alternatively, the controller 110 of the refrigerator 1 may analyze the user's voice and determine the user's intention using the voice recognition function.

The controller 110 may search for a food name corresponding to the identification mark uttered by the user (905) and display the searched food. As shown in FIG. 13, the controller 110 may display a UI including purchase information for purchasing selected foods by executing a mart application on the display 120 (906).

In this case, as shown in FIG. 11, the controller 110 outputs information analyzing the voice data of the user to the mart application, and executes the mart application when receiving information on food from the second server SV2. The UI as shown in FIG. 13 may be displayed on the display 120.

Referring to FIG. 13, a food confirmation UI 350 specifically showing information of a food selected by a user is shown. The food confirmation UI includes a representative image area 354 representing a food, a food information area 356 including a manufacturer, a food name, a capacity, a quantity, a price, and the like, a quantity control area 358 for controlling the quantity of food, A food purchase area 360 including food purchase information such as a shopping cart tab for putting food in a shopping cart, a direct purchase tab for purchasing food directly, a gift tab for presenting food, and a tab for bookmarking selected foods It may include. The number or arrangement of regions included in the food purchase UI illustrated in FIG. 13 is merely an example and may include other numbers, arrangements or other contents.

As shown in FIG. 14, the user may say <put in the shopping cart> to put the food displayed in the food confirmation UI in the shopping cart (907), or say <buy it> to immediately purchase the food displayed in the food confirmation UI. May be 912.

When the function of putting the food into the shopping cart in response to the utterance of the user `` put in the shopping cart '' is normally executed (YES in 908), the controller 110 displays the voice recognition UI on the display 120 as shown in FIG. 14. In operation 909, the text “Add the selected food to the shopping cart” may be displayed on the voice recognition UI. In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. The voice recognition UI may be displayed as a card UI type on the mart application screen.

When the text and voice indicating that the food is normally added to the shopping cart is output, the mart application may display a shopping cart screen on the display 120 (911).

If an error occurs in the function of putting food into the shopping cart in response to the user's utterance of <put in the shopping cart> (No at 908), the controller 110 displays the voice recognition UI as shown in FIG. , An error occurred when adding the shopping cart. Check the content and try again> may be displayed on the speech recognition UI (910). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. In this case, the user may say "put it in the shopping cart" again to add the food to the shopping cart.

If the function of immediately purchasing food in response to a user's utterance of <buy it> is normally executed (YES in 913), the controller 110 displays a voice recognition UI on the display 120 as shown in FIG. 14. In operation 914, the text <Go to the purchase page of the selected food> may be displayed on the voice recognition UI. In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. The voice recognition UI may be displayed as a card UI type on the mart application screen.

In operation 915, the mart application may display a food purchase screen on the display 120 when a text and a voice indicating that the function of directly purchasing a food is normally executed to move to a purchase page of the food are output.

If an error occurs in the function of immediately purchasing food in response to a user's utterance of <buy it> (NO in 913), the controller 110 displays a voice recognition UI on the display 120, as shown in FIG. An error occurred when moving the purchase page. Check the contents and try again> may be displayed on the speech recognition UI (916). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. In this case, the user may say <purchase> again to purchase the food again.

Meanwhile, as illustrated in FIG. 12, when the user does not want to select a food included in the food list or wants to cancel the food purchase process, the user may say <Cancel>.

The controller 110 may display the speech recognition UI on the display 120 in response to the user's speech called <cancel>, and display the text <food search canceled> on the speech recognition UI (918). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. The voice recognition UI may be displayed as a card UI type on the mart application screen.

In addition, as illustrated in FIG. 12, the user may say a food name that is not an identification mark (919). If foods of the same food name exist among the foods included in the food list, or if the user says the wrong food name, the food selected by the user may not be identified. In this case, as shown in FIG. 12, the controller 110 may display the voice recognition UI on the display 120 in response to the user's speech, and display the text <Please tell the number> on the voice recognition UI ( 920). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

As described above, the controller 110 provides information in the form of a card UI without immediately executing the related application in response to a user's utterance, and then selects a related application when specific information is selected from the information provided through the card UI. Can be run.

When the user utters a voice by specifying a target of the voice command, the controller 110 of the refrigerator 1 may perform a command corresponding to the voice command as illustrated in FIG. 15.

As shown in FIG. 15, when a voice of <Stop Music> is input through the microphone 182 (1300), the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function to intention of the user. Figure out. The controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.

When the user's intention to stop the music is determined, the controller 110 of the refrigerator 1 determines whether a music application is currently running, that is, whether music is being played (1301). If the music is playing, the controller 110 immediately stops playing the music (1302).

If music is not being played, the controller 110 may display the voice recognition UI on the display 120 as shown in FIG. 15, and display the text <Music not being played> on the voice recognition UI. (1303). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

Unlike the example shown in FIG. 15, a voice command for which an object is not specified may be input through the microphone 182. That is, as shown in FIG. 16, a voice of <stop>, in which an object of a command is omitted, may be input through the microphone 182 (1300).

When a voice of <Stop>, in which the object of the command is omitted, is input through the microphone 182, the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function to determine the user's intention. The controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.

When the voice input through the microphone 182 does not include the target of the command, the controller 110 determines whether the voice command can be performed on the currently running application displayed on the display 120 (1310).

When the controller 110 can perform a voice command on the currently running application displayed on the display 120, the controller 110 performs a voice command on the application displayed on the display 120 (1320).

For example, if a recipe application is currently displayed on the display 120 and a voice reading a recipe is output through the speaker 181, the controller 110 immediately stops the function of reading a recipe application. You can.

In other words, when a voice command in which the command object is omitted is input, the controller 110 is displayed on the display 120 to determine whether the command can be executed first with respect to the currently running application.

When the controller 110 is unable to perform a voice command on the currently running application displayed on the display 120, the controller 110 displays a voice recognition UI on the display 120 to inform that the voice command cannot be performed (1330). .

That is, as illustrated in FIG. 16, the controller 110 may display the voice recognition UI on the display 120 to display the text <not reading the recipe> on the voice recognition UI. In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

If the controller 110 notifies that the voice command cannot be performed for the application displayed on the display 120 through the voice recognition UI, the controller 110 determines whether a predetermined priority exists for the application related to the input voice command (1340). . If there is a predetermined priority, the controller 110 performs a voice command for the application according to the priority (1350).

For example, the controller 110 performs a voice command for the application ranked in the highest rank according to the priority, and if it is unable to perform the voice command for the application ranked in the highest rank, it is ranked in the next rank. Perform voice commands for the application.

When the music application is ranked highest in the priority of the application related to the voice command " Stop ", the controller 110 stops playing the music being played by the music application. If the music is not playing, the controller 110 determines whether it is possible to perform a voice command for the application ranked in the next rank. The controller 110 may perform a voice command for the applications ranked in the priority in this way.

If a predetermined priority does not exist for the application related to the input voice command, the controller 110 may request confirmation of whether to perform the voice command for another application (1360).

As described above, if a recipe application is currently displayed on the display 120 but the function of reading a recipe is not executed, the controller 110 determines whether a voice command can be performed for another application.

When the radio application is currently running in the background, the controller 110 determines the radio application as a target of a voice command as shown in FIG. 16, and displays a voice recognition UI before performing the voice command for the radio application. And request confirmation of whether or not the voice command is performed.

That is, as shown in FIG. 16, the controller 110 may display the voice recognition UI on the display 120 and display the text <Shall I stop the radio> on the voice recognition UI. In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

When the answer to the confirmation request is input through the microphone 182, the controller 110 performs a command according to the input answer (1370).

For example, the controller 110 stops the execution of the radio application when an affirmative voice (eg <yes>) is input to the confirmation request " Stop radio? &Quot;.

As described above, even if the object to be executed in the command is omitted in the voice command, the refrigerator 1 according to the disclosed embodiment may perform a command corresponding to the intention of the user according to the method shown in FIG. 16.

As illustrated in FIGS. 17 to 19, a user may refer to a specific user and request the provision of information related to the user.

As shown in FIG. 17, when a voice of “show mother's calendar this month” is input through the microphone 182 (2000), the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function. Understand your intentions.

The controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.

As an example of an application for providing information specific to each user, a calendar application is mentioned. However, the present invention is not limited thereto, and a voice output setting application (eg, morning briefing in FIG. Yes) may provide the user-specific information according to the method according to Figs.

When the controller 110 of the refrigerator 1 recognizes the name of a specific user included in the voice and the intention of the user included in the voice is determined, the controller 110 performs a command according to the voice. That is, if it is determined that the user's intention is to show the mother's calendar among the plurality of users and the calendar of this month among the mother's calendars, as shown in FIG. The moon calendar 290 is displayed on the display 120 (2030).

As illustrated in FIG. 18, a calendar provided by a calendar application may include a calendar portion 291 in which a date and a day are displayed, and a calendar portion 292 in which a today's schedule is displayed. The configuration of the calendar shown in FIG. 18 is merely an example and may have different configurations and arrangements.

If the controller 110 of the refrigerator 1 cannot recognize the name of a specific user included in the voice, the controller 110 performs a process of selecting a specific user from among prestored users (2020).

If the controller 110 of the refrigerator 1 cannot recognize the name of a specific user included in the voice, as shown in FIG. 19, the controller 110 displays the voice recognition UI on the display 120 and selects a profile. The text may be displayed on the voice recognition UI (2021). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. The controller 110 may display a profile list of a user registered in advance in the refrigerator 1 together with the voice recognition UI.

The profile list includes registered user names and an indication of an identification mark to identify each user name. The identification mark may be indicated by a number, as shown in FIG. 19, but is not limited thereto. In the case of the identification mark, it may be a mark for distinguishing a plurality of users, respectively, and as shown, may be indicated by a number or may be indicated by a letter.

As illustrated in FIG. 19, in response to a voice requesting confirmation of which user among users included in the profile list, the user may utter a voice for selecting a specific user (2022). For example, the user may speak the identification mark negatively.

The user may select an identification mark by uttering a second or second to select a second user in the profile list (2023, 2024).

The controller 110 may determine a user name corresponding to the identification mark uttered by the user, and display the determined user's calendar on the display 120. That is, the controller 110 determines the mother as the user corresponding to the identification mark of the second or the second uttered by the user, and displays the mother's current month's calendar on the display 120 as shown in FIG. 18 (2030). ).

In addition, the user may say a user name that is not an identification mark (2025).

When the user speaks the wrong user name or utters another expression that is not related to the user name, a case may occur in which the profile selected by the user cannot be identified. In this case, as illustrated in FIG. 19, the controller 110 may display the voice recognition UI on the display 120 in response to the user's speech, and display the text <Please tell the profile number> on the voice recognition UI. (2026). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

If the controller 110 has already requested the user to select a profile more than once (2027), as shown in FIG. 19, the controller 110 displays the voice recognition UI on the display 120 to display the text " Correct information cannot be obtained. &Quot; May be displayed on the voice recognition UI (2028). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

Although not shown in the drawing, when the user does not want to select a user included in the profile list or wants to cancel the calendar display itself, the user may say <cancel>. The controller 110 may display the speech recognition UI on the display 120 in response to the user's speech called <cancel>, and display the text <canceled> on the speech recognition UI. In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

On the other hand, as shown in Figure 20, a user may request the provision of information stored differently for each user without referring to a specific user.

As shown in FIG. 20, when the voice of <Show this month's calendar> is input through the microphone 182 (2040), the controller 110 of the refrigerator 1 analyzes the input voice through a voice recognition function. To grasp the intentions. The controller 110 of the refrigerator 1 may analyze a user's voice by using a voice recognition function, and transmit voice data to the first server SV1 to obtain analysis information of the voice data from the first server SV1. May be received.

If the voice data does not include a representation indicating a specific user, the controller 110 of the refrigerator 1 compares the voice data with voice data of the pre-stored users and determines whether there is a matching voice data (2041).

The controller 110 of the refrigerator 1 compares a parameter such as amplitude, waveform, frequency, or period of the voice data input through the microphone 182 with a parameter of pre-stored voice data, and selects a microphone ( In operation 182, voice data that may be considered to match the input voice data may be determined.

When the controller 110 determines that there is voice data that can be considered to match voice data input through the microphone 182 among the pre-stored voice data, the controller 110 may display the user's calendar indicated by the determined voice data on the display 120. May be 2042. For example, if the matched voice data among the pre-stored voice data is determined to be the mother's, the controller 110 may display the mother's current month's calendar on the display 120.

If the controller 110 determines that there is no voice data that can be considered to be matched (NO in 2041), as shown in FIG. 19, the controller 110 displays the voice recognition UI on the display 120 to display the text <Please select a profile>. It may be displayed on the voice recognition UI (2021). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function. The controller 110 may display a profile list of a user registered in advance in the refrigerator 1 together with the voice recognition UI.

In addition, the user may say a user name that is not an identification mark (2025). When the user speaks the wrong user name or utters another expression that is not related to the user name, a case may occur in which the profile selected by the user cannot be identified. In this case, as illustrated in FIG. 19, the controller 110 may display the voice recognition UI on the display 120 in response to the user's speech, and display the text <Please tell the profile number> on the voice recognition UI. (2026). In addition, the controller 110 may output the text displayed on the voice recognition UI as a voice through the speaker 181 using the TTS function.

As described above, the refrigerator 1 according to the disclosed embodiment identifies and identifies the user according to the method shown in FIG. 20 even if the user does not refer to a specific user but requires the provision of information stored differently for each user. It can provide information suitable for the user.

Referring to FIG. 21A, the controller 2100 may include a data learner 2110 and a data recognizer 2120. The controller 2100 may be the controller 110 of the refrigerator 1. Alternatively, the controller 2100 may be a controller (not shown) of the first server SV1.

The data learner 2110 may train the learning network model to have a criterion for recognizing voice data. For example, the data learner 2110 may train the learning network model to have a criterion for recognizing a food name corresponding to the first voice or an identification mark corresponding to the second voice. Alternatively, the data learner 2110 may train the learning network model to have a reference for recognizing an image. For example, the data learner 2110 may train the learning network model to have a criterion for recognizing food stored in the storage room from an image of the storage room of the refrigerator 1.

The data learner 2110 may train a learning network model (or a data recognition model) using learning data according to a supervised learning method or an unsupervised learning method based on an artificial intelligence algorithm.

In an embodiment, the data learner 2110 may train the learning network model using the voice data and the text corresponding to the voice data (for example, a food name and an identification mark) as the training data. Alternatively, the data learner 2110 may train the learning network model using the voice data and the user information that uttered the voice data as the training data. Alternatively, the data learner 2110 may train the learning network model using image data and text (eg, a food name) corresponding to the image data as learning data.

The data recognizer 2120 may recognize the voice data by applying the voice data as the recognition data to the learned learning network model. For example, the data recognizer 2120 may recognize food names or identification marks corresponding to the voice data by applying voice data related to the first voice or the second voice as recognition data to the learning network model. On the other hand, the recognition result of the learning network model (or data recognition model) may be used again to update the learning network model. Alternatively, the data recognizer 2120 may recognize the user who spoke the voice data by applying the voice data as the recognition data to the learning network model.

Alternatively, the data recognizing unit 2120 may recognize the image data by applying the image data of the storage room 20 as the recognition data to the learned learning network model. For example, the data recognizing unit 2120 may apply image data as recognition data to the learning network model to recognize a food name corresponding to the image data.

At least one of the data learner 2110 and the data recognizer 2120 may be manufactured in the form of at least one hardware chip and mounted on the electronic device. For example, at least one of the data learner 2110 and the data recognizer 2120 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general purpose processor (eg, a CPU). Alternatively, the electronic device may be manufactured as a part of an application processor or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various electronic devices. In this case, a dedicated hardware chip for artificial intelligence is a dedicated processor specialized for probability calculation, and has higher parallelism performance than a conventional general-purpose processor, so that a computational task in artificial intelligence such as machine learning can be processed quickly.

In this case, the data learner 2110 and the data recognizer 2120 may be mounted on one electronic device or may be mounted on separate electronic devices, respectively. For example, one of the data learner 2110 and the data recognizer 2120 may be included in the refrigerator 1, and the other may be included in the server SV1. In addition, the data learner 2110 and the data recognizer 2120 may provide at least a part of the learning network model constructed by the data learner 2110 to the data recognizer 2120 through wired or wirelessly. Data input to the data recognizer 2120 may be provided to the data learner 2110 as additional learning data.

Meanwhile, at least one of the data learner 2110 and the data recognizer 2120 may be implemented as a software module. When at least one of the data learner 2110 and the data recognizer 2120 is implemented as a software module (or a program module including instructions), the software module may be a computer readable non-transitory computer. It may be stored in a non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

21B is a detailed block diagram of the data learner 2110 and the data recognizer 2120, according to an exemplary embodiment.

FIG. 21B is a block diagram of the data learner 2110, according to an exemplary embodiment.

Referring to (a) of FIG. 21B, the data learner 2110 may include a data acquirer 2110-1, a preprocessor 2110-2, a training data selector 2110-3, and a model. The learning unit 2110-4 and the model evaluating unit 2110-5 may be included. The data learner 2110 according to some exemplary embodiments may include a data acquirer 2110-1 and a model learner 2110-4, and include a preprocessor 2110-2 and a training data selector 2110-. 3) and at least one of the model evaluator 2110-5 may optionally be further included or not included at all.

The data acquirer 2110-1 may acquire data necessary for learning a criterion for recognizing voice data or image data.

For example, the data acquirer 2110-1 may acquire audio data or video data. The data acquirer 2110-1 may acquire audio data or image data from the refrigerator 1. Alternatively, the data acquisition unit 2110-1 may acquire audio data or video data from a third device (eg, a mobile terminal or a server) connected to the refrigerator 1. Alternatively, the data acquirer 2110-1 may acquire audio data or image data from a device or database storing or managing learning data.

The preprocessor 2110-2 may preprocess audio data or video data. The preprocessor 2110-2 may process the acquired audio data or the image data into a preset format so that the model learner 2110-4 to be described later can use the acquired data for learning for situation determination. Can be.

For example, the preprocessor 2110-2 may remove noise or process the audio data or the image data acquired by the data acquirer 2110-1 so as to select meaningful data or to process a predetermined form. Can be. Alternatively, the preprocessor 2110-2 may process the acquired data in the form of data suitable for learning.

The training data selector 2110-3 may select voice data or video data required for learning from the preprocessed data according to a preset criterion or randomly. The selected training data may be provided to the model learner 2110-4. The preset criterion may include, for example, at least one of a property of data, a time for generating data, a place for generating data, a device for generating data, reliability of data, and a size of data.

In addition, the training data selector 2110-3 may select the training data according to a preset criterion by the training by the model learner 2110-4 to be described later.

The model learner 2110-4 may train the learning network model to have a criterion for recognizing the voice data or the image data based on the training data. For example, the model learner 2110-4 may train the learning network model to have a criterion for recognizing a food name or an identification mark corresponding to the voice data. For example, the model learner 2110-4 may train the learning network model to have a criterion for recognizing food stored in the storage room from image data photographing the storage room of the refrigerator 1.

In this case, the learning network model may be a pre-built model. For example, the learning network model may be a model built in advance by receiving basic learning data (for example, sample image data or audio data).

As a learning result of the model learner 2110-4, the learning network model may be set to recognize (or determine, estimate, infer, predict) a food name or an identification mark corresponding to the voice. Alternatively, the learning network model may be set to recognize (or determine, estimate, reason, predict) the user who spoke the voice. Alternatively, the learning network model may be set to recognize (or determine, estimate, infer, predict) the food stored in the storage room from the image data.

The learning network model may be, for example, a model based on a neural network. The learning network model can be designed to simulate the human brain structure on a computer. The learning network model may include a plurality of weighted network nodes that simulate neurons in a human neural network. A plurality of network nodes may form a connection relationship to simulate the synaptic activity of neurons that send and receive signals through synapses. The learning network model may include, for example, a neural network model or a deep learning model developed from the neural network model. In the deep learning model, a plurality of network nodes may be located at different depths (or layers) and exchange data according to a convolutional connection relationship. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the learning network model, but are not limited thereto.

According to various embodiments of the present disclosure, when there are a plurality of pre-built learning network models, the model learning unit 2110-4 may be a learning network model for learning a learning network model having a large correlation between input learning data and basic learning data. You can decide. In addition, the model learner 2110-4 may train the learning network model using, for example, a learning algorithm including an error back-propagation method or a gradient descent method. . In addition, the model learner 2110-4 may train the learning network model through, for example, supervised learning using learning data as an input value. Alternatively, the model learner 2110-4 may train the learning network model through unsupervised learning that finds a criterion by learning by itself without any guidance. In addition, the model learner 2110-4 may train the learning network model through, for example, reinforcement learning using feedback on whether the result of learning is correct.

When the learning network model is trained, the model learner 2110-4 may store the learned learning network model. In this case, the model learner 2110-4 may store the learned learning network model in the storage 130 of the refrigerator 1. Alternatively, the model learner 2110-4 may store the learned learning network model in a memory of the server SV1 or the server SV2 connected to the refrigerator 1 through a wired or wireless network.

In this case, the storage unit 130 in which the learned learning network model is stored may store, for example, instructions or data related to at least one other component of the refrigerator 10. The memory may also store software and / or programs. The program may include, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or “application”), and the like.

The model evaluator 2110-5 may input the evaluation data into the learning network model, and cause the model learner 2110-4 to relearn if the recognition result output from the evaluation data does not satisfy a predetermined criterion. have. In this case, the evaluation data may be preset data for evaluating the learning network model.

For example, the model evaluator 2110-5 may determine a predetermined criterion when the number or ratio of the evaluation data that is not accurate among the recognition results of the learned learning network model for the evaluation data exceeds a preset threshold. It can be evaluated as not satisfied. For example, when a predetermined criterion is defined at a ratio of 2%, the model evaluation unit 2110-5 when the learned learning network model outputs an incorrect recognition result for more than 20 evaluation data out of a total of 1000 evaluation data. Can be judged that the learned learning network model is not suitable.

On the other hand, when there are a plurality of learned learning network models, the model evaluator 2110-5 evaluates whether a predetermined criterion is satisfied for each learned learning network model, and the model that satisfies the predetermined criterion is a final learning network. Can be determined as a model. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 2110-5 may determine any one or a predetermined number of models which are preset in the order of the highest evaluation score as the final learning network model.

Meanwhile, the data acquisition unit 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110 in the data learner 2110. At least one of -5) may be manufactured in the form of at least one hardware chip and mounted on the electronic device. For example, at least one of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5. One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general purpose processor (eg, a CPU or application processor) or a graphics dedicated processor (eg, a GPU). It may be mounted on various electronic devices.

In addition, the data obtaining unit 2110-1, the preprocessor 2110-2, the training data selecting unit 2110-3, the model learning unit 2110-4, and the model evaluating unit 2110-5 are electronic components. It may be mounted on the device, or may be mounted on separate electronic devices, respectively. For example, some of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5. May be included in the refrigerator 1 and the remaining part may be included in the server SV1 or the server SV2.

In addition, at least one of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5 is provided. It may be implemented as a software module. At least one of the data acquirer 2110-1, the preprocessor 2110-2, the training data selector 2110-3, the model learner 2110-4, and the model evaluator 2110-5 is a software module. (Or a program module including instructions), the software module may be stored in a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

(B) of FIG. 21B is a block diagram of the data recognition unit 2120 according to an embodiment.

Referring to (b) of FIG. 21B, the data recognizer 2120 according to some embodiments may include a data acquirer 2120-1, a preprocessor 2120-2, a recognition data selector 2120-3, and a recognizer. The result providing unit 2120-4 and the model updating unit 2120-5 may be included. According to some embodiments, the data recognizer 2120 essentially includes a data acquirer 2120-1 and a recognition result provider 2120-4, and includes a preprocessor 2120-2 and a recognition data selector ( 2120-3) and the model updater 2120-5 may optionally be further included.

The data recognizer 2120 may apply the voice data to the learned learning network model as the recognition data, and may recognize a food name corresponding to the voice data, a food label, or a user who utters the voice data. Alternatively, the data recognizing unit 2120 may recognize the food stored in the storage room 20 of the refrigerator 1 by applying the image data to the learned learning network model as the recognition data.

First, the data acquirer 2120-1 may acquire voice data necessary for recognizing a user who spoke a food name, a food label, or a voice from the voice data. Alternatively, the data acquirer 2120-1 may acquire image data necessary for recognizing food from the image data.

For example, the data acquirer 2120-1 may acquire various sensing information that is directly input by the user or data selected by the user, or sensed using various sensors of the refrigerator 1. In addition, the data acquisition unit 2120-1 may acquire data from an external device (eg, a mobile terminal or a server) that communicates with the refrigerator 1.

The preprocessor 2120-2 may preprocess audio data or video data. The preprocessor 2120-2 may process the acquired audio data or the image data into a predetermined format so that the recognition result providing unit 2120-4 to be described later can use the data for situation determination.

For example, the preprocessor 2120-2 may remove noise or process the audio data or the image data acquired by the data acquirer 2120-1 so that meaningful data can be selected or processed into a predetermined form. Can be. Alternatively, the preprocessor 2120-2 may process the acquired data into a form of data suitable for learning.

The recognition data selector 2120-3 may select voice data or image data required for recognition from the preprocessed data according to a preset criterion or randomly. The selected recognition data may be provided to the model learner 2110-4. The preset criterion may include, for example, at least one of a property of data, a time for generating data, a place for generating data, a device for generating data, reliability of data, and a size of data.

The recognition result providing unit 2120-4 may apply the selected recognition data to the learning network model to recognize the recognition network. For example, the recognition result providing unit 2120-4 may apply the selected voice data to the learning network model to recognize a food name or an identification mark corresponding to the voice. Alternatively, the recognition result providing unit 2120-4 may recognize the user who spoke the voice by applying the selected voice data to the learning network model. Alternatively, the recognition result providing unit 2120-4 may recognize the food corresponding to the image data by applying the selected image data to the learning network model.

The model updater 2120-5 may cause the learning network model to be updated based on the evaluation of the recognition result provided by the recognition result provider 2120-4. For example, the model updater 2120-5 provides the model learner 2110-4 by providing the food name, the identification mark, or the user information provided by the recognition result provider 2120-4 to the model learner 2110-4. 2110-4 may update the learning network model.

Meanwhile, the data acquisition unit 2120-1, the preprocessor 2120-2, the recognition data selection unit 2120-3, the recognition result providing unit 2120-4, and the model updater 2 in the data recognition unit 2120 At least one of 2120-5 may be manufactured in the form of at least one hardware chip and mounted on the electronic device. For example, among the data acquirer 2120-1, the preprocessor 2120-2, the recognition data selector 2120-3, the recognition result provider 2120-4, and the model updater 2120-5. At least one may be fabricated in the form of a dedicated hardware chip for artificial intelligence (AI), or may be fabricated as part of an existing general purpose processor (e.g., CPU or application processor) or graphics dedicated processor (e.g., GPU). It may be mounted on various electronic devices.

In addition, the data acquisition unit 2120-1, the preprocessor 2120-2, the recognition data selection unit 2120-3, the recognition result providing unit 2120-4, and the model updater 2120-5 It may be mounted on an electronic device, or may be mounted on separate electronic devices, respectively. For example, among the data acquirer 2120-1, the preprocessor 2120-2, the recognition data selector 2120-3, the recognition result provider 2120-4, and the model updater 2120-5. Some may be included in the electronic device, and others may be included in the server.

In addition, at least one of the data acquirer 2120-1, the preprocessor 2120-2, the recognition data selector 2120-3, the recognition result provider 2120-4, and the model updater 2120-5. May be implemented as a software module. At least one of the data acquisition unit 2120-1, the preprocessor 2120-2, the recognition data selection unit 2120-3, the recognition result providing unit 2120-4, and the model updater 2120-5 is software. When implemented as a module (or a program module including instructions), the software module may be stored on a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

22 is a flowchart of a refrigerator 1 displaying information according to an embodiment of the present disclosure.

The refrigerator 1 may determine whether a first voice including a food name is input through the microphone 182 (2201).

As a result of the determination, when the first voice is identified, the refrigerator 1 displays the food list including food information corresponding to the food name and an identification mark for identifying the food information, based on the recognition of the first voice. In operation 2202, the display 120 may display the same.

For example, when the first voice is identified, the refrigerator 1 acquires a food name by recognizing the first voice by using a learning network model trained according to an artificial intelligence algorithm, and includes food information corresponding to the food name. The list may be displayed on the display 120. In this case, the learning network model may be a learning network model trained using a plurality of voices and a plurality of words corresponding to each of the plurality of voices.

As another example, when the first voice is identified, the refrigerator 1 acquires a food name and user information uttering the first voice by recognizing the first voice by using a network model trained according to an artificial intelligence algorithm. The display 120 may display a food list corresponding to the food information including food information related to the user information (for example, food information preferred by the user). In this case, the learning network model may be a learning network model trained using a plurality of voices and user information corresponding to each of the plurality of voices.

As another example, when the first voice is identified, it may be identified whether a food associated with the recognized food name exists in the storage compartment 20 of the refrigerator 1. If the food is not present in the storage compartment 20, the refrigerator 1 may display a food list including food information corresponding to the food name on the display. On the other hand, if food is present in the storage compartment 20, the refrigerator 1 may display on the display 120 information indicating that the food associated with the recognized food name exists in the storage compartment of the refrigerator 1.

Next, the refrigerator 1 may determine whether a second voice indicating an identification mark is input through the microphone 182 (2203).

As a result of the determination, when the second voice is identified, the refrigerator 1 transmits purchase information of at least one food item corresponding to the identification mark to the display 120 located in front of the refrigerator 1 based on the recognition of the second voice. It may be displayed (2204).

For example, when the second voice is identified, the refrigerator 1 acquires an identification mark by recognizing the second voice using a learning network model trained according to an artificial intelligence algorithm, and purchases food information corresponding to the identification mark. Can be displayed on the display.

23 is a flowchart of a network system according to an embodiment of the present disclosure.

In FIG. 23, the network system may include a first component 2301 and a second component 2302. Here, the first component 2301 is the refrigerator 1, and the second component 2302 may be a server SV1 in which a data recognition model is stored or a cloud including at least one server. Alternatively, the first component 2301 may be a general purpose processor, and the second component 2302 may be an artificial intelligence dedicated processor. Alternatively, the first component 2301 may be at least one application, and the second component 2302 may be an operating system (OS). That is, the second component 2302 is a component that is more integrated, dedicated, has a smaller delay, has a higher performance, or has more resources than the first component 2301. It can be a component that can process many operations required at the time of generation, update or application of the faster than the first component 2301.

Meanwhile, an interface for transmitting / receiving data between the first component 2301 and the second component 2302 may be defined.

For example, an application program interface (API) having recognition data to be applied to the data recognition model as an argument value (or, a parameter value or a transfer value) may be defined. An API is defined as a set of subroutines or functions that can be called for any processing of one protocol (e.g., a protocol defined in the refrigerator 1) to another protocol (e.g., a protocol defined in the server SV1). Can be. That is, an environment in which an operation of another protocol may be performed in one protocol through an API may be provided.

In FIG. 23, the first component 2301 may receive a first voice of a user including a food name through the microphone 182 (2310).

The first component 2301 may modulate the input analog voice signal into voice data which is a digital signal and transmit the same to the second component 2302. In this case, the first component 2301 may change the voice data according to the promised communication format and transmit the voice data to the second component 2302 (2311).

The second component 2302 may recognize the received voice data (2312). For example, the second component 2302 may apply the received voice data to the learned learning network model to recognize the voice data and obtain recognition information as a recognition result. For example, the second component 2302 may obtain the food name and the user information that spoke the voice from the voice data. The second component 2302 may transmit the obtained recognition information to the first component 2301 (2313).

The first component 2301 may obtain information requested by the voice data, based on the received recognition information (2314). For example, the first component 2301 may obtain food information corresponding to the food name from the storage of the first component 2301 or the third component 2303 as information requested by the voice data. It may be. The third component 2303 may be, for example, a server in communication connection with the first component 2301. For example, the server may be a server SV2 transmitting information requested by voice data in FIGS. 9 and 10.

In operation 2312, the second component 2302 may directly transmit the recognized recognition information to the third component 2303 without passing through the first component 2301 (2323). In this case, the third component 2303 may acquire the information requested by the voice data based on the received recognition information, and transmit the information requested by the acquired voice data to the first component 2301 (2324). The first component 2301 may obtain information requested by the voice data from the third component 2303.

When the information requested by the voice data is obtained in operation 2314 or 2324, the first component 2301 may include food information corresponding to a food name and an identification mark for identifying the food information as the information requested by the voice data. The list of foods to be included may be displayed (2315).

On the other hand, when the food list including the identification mark is checked, and when a second voice of the user referring to the one identification mark is recognized, the first component 2301 displays purchase information of at least one food corresponding to the identification mark. can do.

24 is a view illustrating a method of purchasing food by a user through a refrigerator according to another embodiment of the present disclosure.

As illustrated in FIG. 24, a first voice for ordering a specific food may be input through the microphone 182 (980).

The controller 110 may recognize the first voice including the food name. As a result of the recognition, the controller 110 may obtain user information that uttered the recognized food name and the first voice.

When the food name and the user information are obtained, the controller 110 may obtain food information preferred by the user who spoke the first voice in relation to the food name (981).

Based on the obtained food information, the controller 110 may display on the display 120 a food list including an identification mark for distinguishing the plurality of food information and the plurality of food information that the user prefers ( 982).

Next, via the microphone 182, a second voice indicating a specific identification mark among the identification marks included in the food list may be input (983).

The controller 110 may recognize the second voice including the identification mark. As a result of the recognition, the controller 110 may display the purchase information of the food corresponding to the identification mark on the display 120 (984).

Next, a third voice for requesting purchase of food through the microphone 182 may be input (985). The third voice requesting the purchase of food may be, for example, a voice spoken by the user as " purchase immediately ".

The controller 110 may recognize the third voice. The controller 110 may display, on the display 120, a UI (for example, a UI for payment) for purchasing food as a result of recognizing the third voice. In this situation, the controller 110 may control the communication unit 140 to transmit a message informing the purchase situation to the external device 986-1 (986). The message indicating the purchase status may include, for example, a message requesting confirmation of whether to purchase the food before purchasing the food, a message requesting input of a password for purchasing the food, or a message indicating a purchase result after the purchase of the food. have.

For example, the controller 110 may acquire at least one of the first voice to the third voice to obtain user information uttering the voice. The controller 110 may transmit a message informing the purchase situation to the user's device corresponding to the user information by using the obtained user information. In addition, when the food requested by the user exists in the storage compartment 20 of the refrigerator 1, the controller 110 may transmit a message informing the information about the stored food to the user's device. On the other hand, the controller 110 may transmit a message to the device of the guardian of the user who spoke the voice. For example, if the person who ordered the food is a child, the controller 110 may transmit a message informing the above-described purchase situation to the parent device of the child.

In response to the received message, the external device 986-1 receiving the message may display the purchase status as text, provide a voice, display an image, display haptic or emit light. The manner of notifying the purchase situation is not limited to the above example.

According to various embodiments of the present disclosure, when a situation in which food is purchased through the refrigerator 1 occurs, the controller 110 may notify a purchase situation to an external device 986-1 that is previously set or communicates with the refrigerator 1. Can also be transmitted. Alternatively, the controller 110 may transmit a message informing the purchase situation to the external device 986-1 that satisfies the preset criteria. At this time, the external device 986-1 that satisfies the preset criteria is located at a predetermined place (for example, a living room, a room, etc.), the signal strength communicating with the refrigerator 1 is greater than or equal to a predetermined value, or the refrigerator 1 The target device may be an external device having a distance from the predetermined distance. In addition, the preset reference may be an external device 986-1 connected to the refrigerator 1 according to a specific communication method. The specific communication method may include, for example, a short range communication method (eg, Wi-Fi, Bluetooth, Zigbee, etc.) or a direct communication method. When the refrigerator 1 is communicating with the external device 986-1 according to the short range communication method, the controller 110 sends a push message indicating a purchase status through a communication channel established according to the short range communication method. (986-1).

The term "module" as used in the present disclosure includes, for example, a unit composed of a combination of one or more of hardware, software, or firmware, for example, a logic, a logical block. The terms "block", "component", or "circuit" may be used interchangeably. The module may be an integrally formed part or a minimum unit or part of performing one or more functions. "Modules" may be implemented mechanically or electronically, for example, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) or programs that perform some operations, known or will be developed in the future. It may include a programmable-logic device.

The methods described above may be embodied in the form of program instructions that may be executed by various computer means and may be recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the method according to the disclosed embodiments may be provided as a computer program product.

The computer program product may include a S / W program, a computer readable storage medium on which the S / W program is stored, or a food product traded between a seller and a buyer.

For example, a computer program product may be a food product (eg, a downloader) in the form of a S / W program distributed electronically through a refrigerator 1, a server SV1, a server SV2 or a manufacturer of the device, or an application store. A loadable app). For electronic distribution, at least a part of the S / W program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a server of a manufacturer or an application store or a storage medium of a relay server.

Although one embodiment of the disclosed invention has been illustrated and described above, the disclosed invention is not limited to the specific embodiments described above, and those skilled in the art without departing from the scope of the claims. Various modifications are possible as well as such variations are not individually understood from the disclosed invention.

Claims

A storage room for storing food;

A temperature sensor for sensing a temperature inside the storage compartment;

Cooling unit for supplying cold air to the storage compartment;

A microphone for receiving a voice;

A display for displaying information;

At least one processor electrically connected to the temperature sensing unit, the microphone, and the display; And

A memory electrically connected to the at least one processor;

The memory, the processor,

When the first voice including a food name is recognized through the microphone, a food list including food information corresponding to the food name and an identification mark for identifying the food information is displayed on the display, and through the microphone. When the second voice referring to the identification mark is recognized, storing at least one instruction set to display purchase information of the food corresponding to the identification mark on the display;

Refrigerator.
The method of claim 1,

The memory, the processor,

When the first voice including the food name is input through the microphone, the food name is obtained by recognizing the first voice using a learning network model trained according to an artificial intelligence algorithm, and the food information corresponding to the food name. Store at least one instruction configured to display the food list on the display;

The learning network model,

Learned using a plurality of voices and a plurality of words corresponding to each of the plurality of voices,

Refrigerator.
The method of claim 1,

The memory, the processor,

When the first voice including the food name is input through the microphone, the food name and the user information that utter the first voice are obtained by recognizing the first voice using a learning network model trained according to an artificial intelligence algorithm. Store at least one instruction corresponding to the food name and configured to display the food list on the display, the food list including the food information associated with the user information;

The learning network model,

Learned using a plurality of voices and user information corresponding to each of the plurality of voices,

Refrigerator.
The method of claim 1,

The memory, the processor,

If the first voice is recognized, the storage compartment identifies whether a food associated with the food name exists in the storage compartment, and if the food is present in the storage compartment, displays information indicating that the food exists in the storage compartment. Storing at least one instruction set to display in

Refrigerator.
The method of claim 1,

The memory, the processor,

If the first voice is recognized, identifying whether a food associated with the food name exists in the storage compartment in the storage compartment, and if the food does not exist in the storage compartment, the food including the food information corresponding to the food name Storing at least one instruction set to display a list on the display,

Refrigerator.
The method of claim 1,

The memory, the processor,

When the first voice is recognized, storing at least one instruction configured to display the food list in the form of a user interface of a card type on the user interface of the application displayed on the display.

Refrigerator.
The method of claim 1,

The memory, the processor,

When the second voice is recognized, executing an application for providing food information indicated by a label included in the second voice to store at least one instruction set to display purchase information of the food provided by the application on a display; ,

Refrigerator.
The method of claim 1,

It further comprises a communication unit;

The memory, the processor,

When the first voice is recognized, the first voice data is transmitted to the server through the communication unit, and when the communication unit receives information about the food list transmitted from the server, the food list is displayed on the display. Storing at least one instruction set to:

Refrigerator.
The method of claim 1,

It further comprises a communication unit;

The memory, the processor,

When the second voice is recognized, the second voice data is transmitted to the server through the communication unit, and when the communication unit receives the purchase information of the food transmitted from the server, the purchase information of the food is displayed on the display. Storing at least one instruction set to display,

Refrigerator.
The method of claim 1,

It further comprises a communication unit;

The memory, the processor,

When the first voice is recognized, the data of the first voice is transmitted to the server through the communication unit, and when the communication unit receives analysis information on the first voice transmitted from the server, the operation according to the analysis information. Transmitting an instruction according to the analysis information to an application related to the execution, wherein the application stores at least one instruction configured to display the food list;

Refrigerator.
The method of claim 1,

It further comprises a communication unit;

The memory, the processor,

When the second voice is recognized, the second voice data is transmitted to the server through the communication unit, and when the communication unit receives analysis information on the second voice transmitted from the server, the operation according to the analysis information. Transmitting an instruction according to the analysis information to an application related to performance, so that the application stores at least one instruction set to display purchase information of the food;

Refrigerator.
The method of claim 1,

It further includes a speaker;

The memory, the processor,

When the first voice is recognized and the food list is displayed on the display, a voice indicating that the list is displayed is output through the speaker, and when the second voice is recognized and the purchase information of the food is displayed on the display, Storing at least one instruction set to output a voice through the speaker indicating that purchase information of the food is displayed;

Refrigerator.
A storage room for storing food;

A temperature sensor for sensing a temperature inside the storage compartment;

Cooling unit for supplying cold air to the storage compartment;

A microphone for receiving a voice;

A display for displaying information;

At least one processor electrically connected to the temperature sensing unit, the microphone, and the display; And

A memory electrically connected to the at least one processor;

The memory, the processor,

If a voice is input through the microphone, the command indicated by the input voice is performed for the application displayed on the display, and if the command indicated by the input voice cannot be performed for the application, applications related to the command. Storing at least one instruction set to execute the command for another application according to a predetermined priority for

Refrigerator.
The method of claim 13,

It further includes a speaker;

The memory, the processor,

Storing at least one instruction set to output a voice through the speaker indicating that the command cannot be performed for the application if the command indicated by the input voice cannot be performed for the application;

Refrigerator.
The method of claim 13,

It further includes a speaker;

The memory, the processor,

If there is no priority, storing at least one instruction set to output a voice through the speaker requesting confirmation as to whether to perform the command for another application,

Refrigerator.
The method of claim 15,

The memory, the processor,

When an answer to the confirmation request is input, storing at least one instruction set to perform a command indicated by the voice for an application indicated by the input answer;

Refrigerator.
The method of claim 13,

The memory, the processor,

If the command indicated by the input voice cannot be performed for the application, the application to execute the command is determined according to a predetermined priority for applications related to the command, and the command is performed for the determined application. Storing at least one instruction set to:

Refrigerator.
The method of claim 13,

The memory, the processor,

If the voice input through the microphone does not include a target to perform a command indicated by the voice, storing at least one instruction set to perform a command indicated by the input voice for an application displayed on the display;

Refrigerator.
A storage room for storing food;

A temperature sensor for sensing a temperature inside the storage compartment;

Cooling unit for supplying cold air to the storage compartment;

A microphone for receiving a voice;

A display for displaying information;

A speaker for outputting voice; And

At least one processor electrically connected to the temperature sensing unit, the microphone, the speaker, and the display; And

A memory electrically connected to the at least one processor;

When a voice is input through the microphone, when user identification is required to perform a command indicated by the voice, the user may display a user registered in advance on the display and output a voice requesting to select from the displayed users through the speaker. Storing at least one set instruction,

Refrigerator.
The method of claim 19,

The memory, the processor,

When a voice for selecting at least one user from among the displayed users is input, executes a command indicated by the voice with respect to the selected user, and stores at least one instruction set to display a result on the display;

Refrigerator.
The method of claim 19,

The memory, the processor,

If user identification is required in the execution of the command indicated by the voice, storing at least one instruction set to display a pre-registered user and a mark for identifying each user on the display;

Refrigerator.
The method of claim 21,

The memory, the processor,

When a voice for selecting at least one of the signs is input, storing at least one instruction configured to perform a command indicated by the voice in relation to the user represented by the sign and display the result on the display;

Refrigerator.
The method of claim 19,

The memory, the processor,

When a voice is input through the microphone, when the voice includes a representation indicating a user, at least one set to perform a command indicated by the voice in relation to a user included in the voice and to display a result on the display To store instructions of

Refrigerator.
The method of claim 19,

The memory, the processor,

At least one instruction configured to perform a command indicated by the voice in relation to a user represented by voice data matching the input voice among prestored voice data and to display a result on the display when a voice is input through the microphone To listen to,

Refrigerator.
In the information display method of the refrigerator,

Receiving a first voice including a food name through a microphone;

Displaying a food list on a display in front of the refrigerator, the food list including food information corresponding to the food name and an identification mark for identifying the food information, based on the recognition of the first voice;

Receiving a second voice indicating the identification mark through the microphone; And

Displaying purchase information of at least one food item corresponding to the identification mark on the display, based on the recognition of the second voice;

Information display method of the refrigerator comprising a.