KR102377971B1 - Artificial intelligence refrigerator including display apparatus - Google Patents

Abstract

An artificial intelligence-based refrigerator according to an embodiment of the present invention is installed on a door and includes a display device including a touch panel and a display, a camera for acquiring an image including food stored therein, and displaying the acquired image. A display controller for controlling a display, a microphone for acquiring a voice, a voice recognition module for recognizing the acquired voice and outputting a recognition result, and acquiring coordinates of a touch input received through the touch panel, and obtaining the coordinates and the voice and a processor for recognizing the food corresponding to the coordinates based on the recognition result of the and performing a function on the recognized food.

Description

Artificial intelligence refrigerator with display device

The present invention relates to a refrigerator, and more particularly, to an artificial intelligence refrigerator in which a display device is provided on a door.

A refrigerator is a device that can keep food fresh for a certain period of time by cooling a freezer or refrigerating chamber to a specific temperature while repeating a freezing or refrigerating cycle. In general, a refrigerator includes a body forming a storage space and a door for opening or closing the storage space. Stored items such as food are stored in the storage space, and the user can open the door to store the stored items or retrieve the stored items.

Recently, with the wide spread of refrigerators and the development of technologies, various requirements for additional functions of refrigerators are increasing. Recently, a refrigerator having a display device and a camera, or a refrigerator having a door made of a transparent material such as glass, has been developed. Such a refrigerator may acquire images of stored objects using a camera, output the acquired images through a display device, or check the inside of the refrigerator from the outside through a transparent door. Accordingly, the user can conveniently check the stored items inside the refrigerator without opening the door of the refrigerator. In addition, there are increasing attempts to provide additional functions such as registration, management, and purchase of food in the refrigerator by using such a refrigerator.

Meanwhile, interest in artificial intelligence (AI) technology is increasing recently. Among various fields of the artificial intelligence technology, machine learning exists as a technology for realizing human learning ability on a computer.

Conventional machine learning has focused on statistical classification, regression, and cluster models. In particular, in supervised learning of classification and regression models, the characteristics of the training data and a learning model that distinguishes new data based on these characteristics were defined in advance. On the other hand, in deep learning, which has recently increased in interest with the advent of the concept of big data, a computer uses a large amount of data to find and determine characteristics by itself.

Recently, deep learning frameworks provided as open sources have emerged in relation to such deep learning, and accordingly, for effective learning and recognition, in addition to deep learning algorithms, learning processes, learning methods, and extraction and selection of data used for learning are related. Technology is becoming more and more important. In addition, research to use machine learning for various products or services is increasing.

1. Publication No. 10-2015-0026412 (published on March 11, 2015)

An object of the present invention is to provide an artificial intelligence refrigerator capable of accurately recognizing food stored therein.

Another object of the present invention is to provide an artificial intelligence refrigerator capable of improving the convenience of use of functions for food stored in the refrigerator.

An artificial intelligence-based refrigerator according to an embodiment of the present invention is installed on a door and includes a display device including a touch panel and a display, a camera for acquiring an image including food stored therein, and displaying the acquired image. A display controller for controlling a display, a microphone for acquiring a voice, a voice recognition module for recognizing the acquired voice and outputting a recognition result, and acquiring coordinates of a touch input received through the touch panel, and obtaining the coordinates and the voice and a processor for recognizing the food corresponding to the coordinates based on the recognition result of the and performing a function on the recognized food.

The voice recognition module includes a voice recognizer having a recognition model for recognizing each of at least one partial voice included in the voice based on a frequency characteristic of the voice, wherein each of the at least one partial voice is included in the image. It may be identification information of the included food, or a command related to the function for the food.

The speech recognizer may update the recognition model using a frequency characteristic of each of the at least one partial speech and a recognition result of each of the at least one partial speech.

According to an embodiment, when the recognition result includes a command related to the registration of the food, the processor generates food management information including the obtained coordinates and identification information of the food, and generates the food management information can be stored in memory.

According to an embodiment, the display controller may control the display to display a tag including at least a portion of the food management information at a position corresponding to the coordinates.

According to an embodiment, when the recognition result includes a command related to the purchase of the food, the processor obtains food management information corresponding to the coordinates from a memory, and based on the command, is obtained from the recognition result The food management information may be updated to include at least one of an order food, an order quantity, and an order timer.

The processor may access the food sales service providing server through a communication unit based on a command related to the purchase of the food, and transmit the purchase information of the food to the sales service providing server.

According to an embodiment, the processor obtains food management information corresponding to the obtained coordinate information from a memory, compares the identification information included in the obtained food management information with the identification information included in the recognition result, and compares Result When the identification information included in the obtained food management information matches the identification information included in the recognition result, the function for the recognized food may be performed.

The refrigerator may further include an image processing processor configured to extract a food image including the coordinates from the acquired image based on the acquired coordinates.

According to an embodiment, the display is implemented as a transparent display, and the processor receives, through the touch panel, a touch input corresponding to a position through which any one of the foods is transmitted through the transparent display. and transform the coordinates of the received touch input into corresponding coordinates on the acquired image.

The processor, based on the converted coordinates and the recognition result of the voice, may recognize the food corresponding to the converted coordinates in the image, and perform a function on the recognized food.

According to an embodiment of the present invention, the refrigerator may accurately recognize food identification information or commands from the user's voice through an artificial intelligence-based voice recognition module. Also, the refrigerator may more accurately recognize food included in an image acquired by the camera or food seen through a transparent display, based on the coordinates of the touch input and identification information of the food.

In addition, since the user can simplify the procedure for executing various functions, such as registration or purchase of food, with a single touch input and voice input, the user's convenience can be increased and the utility of functions provided by the refrigerator can be increased. .

1 is a front view of a refrigerator having a display device according to an embodiment of the present invention.
2 is a block diagram related to a control configuration of a refrigerator according to an embodiment of the present invention.
3 and 4 are diagrams for explaining deep learning as an example of artificial intelligence applied to a voice recognition module of a wearable device according to an embodiment of the present invention.
5 is a diagram for explaining a recognition operation and a learning operation of the voice recognition module.
6 is a flowchart illustrating an operation of a refrigerator according to an embodiment of the present invention.
7 is an exemplary diagram for explaining an operation of the refrigerator registering a specific food in a food management database in relation to the operation of the refrigerator shown in FIG. 6 .
FIG. 8 is a flowchart illustrating an operation in which an image processing processor of a refrigerator extracts an image of food in relation to the embodiment of FIG. 7 .
9 is an exemplary diagram for explaining an operation of the refrigerator providing a purchase function of a specific food in relation to the operation of the refrigerator shown in FIG. 6 .
10 is a flowchart for explaining an embodiment in which the refrigerator recognizes food based on a user's touch input when the display device is implemented as a transparent display so that food inside the refrigerator is transparent to the outside.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings. The accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed in the present specification is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It should be understood to include water or substitutes.

1 is a front view of a refrigerator having a display device according to an embodiment of the present invention.

The display device 100 according to the present invention is installed on the front side of the door 11 of the refrigerator 10 and is formed to be transparent. 1 shows that the display apparatus 100 is formed in only one door among the plurality of doors, but according to an embodiment, the display apparatus 100 may be formed in each of two or more doors among the plurality of doors.

When the display device 100 is transparently formed as described above, the contents 20 such as food stored in the refrigerator 10 can be checked from the outside of the refrigerator 10 and through the display device 100 .

In addition, when the transparent display is applied to the refrigerator door as described above, information on the contents 20 such as food stored in the refrigerator 10 can be checked from the outside. In detail, when the user looks at the contents 20 , such as food stored in the refrigerator 10 , information on the corresponding contents 20 may be displayed to overlap the contents 20 . Of course, information on the corresponding content 20 is stored after being input in advance by the user. Accordingly, the user may be provided with information such as the type, use, and expiration date of the content 20 in real time.

Meanwhile, the camera 110 may be disposed inside the refrigerator 10 . For example, the camera 110 may be disposed on the side facing the inside of the refrigerator 10 among both surfaces of the door 11 to acquire an image of the inside of the refrigerator 10 . However, according to an embodiment, at least one camera 110 may be disposed at various positions in the refrigerator 10 .

The refrigerator 10 may store the acquired image or display it through the display apparatus 100 .

2 is a block diagram related to a control configuration of a refrigerator according to an embodiment of the present invention.

Referring to FIG. 2 , the refrigerator 10 may include the display device 100 and the camera 110 shown in FIG. 1 . In addition, the refrigerator 10 may include a microphone 120 , a speaker 130 , a memory 140 , a communication unit 150 , and a control unit 160 . The components shown in FIG. 2 are not essential in implementing the refrigerator 10 , and the refrigerator 10 may include more or fewer components than the components shown in FIG. 2 .

The display apparatus 100 may include a touch panel 101 that receives a touch input from a user, and a display 102 that displays graphics or text. The touch panel 101 and the display 102 may be integrally formed, or may be formed as separate stacked components.

The processor 161 included in the controller 160 detects a type of a touch input (eg, short-touch, long-touch, drag and drop, etc.) based on a touch input received through the touch panel 101 . And, it is possible to obtain coordinate information of the touch input.

The display controller 162 included in the controller 160 may control the operation of the display 102 based on a signal or data provided from the processor 161 . The display 102 may output graphics or text under the control of the display controller 162 .

The camera 110 may be disposed inside the refrigerator 10 as described above with reference to FIG. 1 . When the door 11 is opened by the user, the camera controller 163 included in the controller 160 may control the camera 110 to acquire at least one image until the door 11 is closed. At least one image acquired by the camera 110 may be stored in the image DB 144 of the memory 140 . According to an embodiment, at least one image acquired by the camera 110 may be transmitted to a server (not shown) or a terminal through the communication unit 150 .

According to an embodiment, the camera controller 163 may control the camera 110 to acquire an image when the opening angle of the door 11 reaches a preset opening angle. The preset opening angle may be an angle at which the camera 110 can obtain an image including the regions inside the refrigerator 10 maximally.

According to an embodiment, the camera controller 163 controls the camera 110 to acquire an image inside the refrigerator 10 based on a specific control command received through an input unit such as the touch panel 101 or the microphone 120 . You can also control it.

Meanwhile, the image processing processor 164 of the controller 160 may extract a food image from an image acquired by the camera 110 through various image processing techniques. Alternatively, the image processing processor 164 may detect an image change as food is put into or out of the refrigerator 10 from images acquired before and after the door 11 is opened. The processor 161 or the image processing processor 164 may detect the stored or shipped food based on the detected image change.

The microphone 120 may obtain a voice from the user and provide the acquired voice to the processor 161 . For example, the voice may be related to a function that the user wants to receive in relation to a specific food.

The voice may include at least one partial voice. The partial voice may be identification information (eg, food name) of the specific food or a command related to the function. That is, the voice may include at least one of identification information and a command of the specific food.

The speaker 130 may output information related to the operation of the refrigerator 10 in the form of sound or voice. According to an embodiment, the refrigerator 10 may output information related to the operation of the refrigerator 10 through the display device 100 .

The memory 140 may store control data or algorithms for the operation of components included in the refrigerator 10 . Meanwhile, the memory 140 includes a food management DB 142 for storing various information related to foods stored in the refrigerator 10 and an image DB 144 for storing images acquired by the camera 110 . can do. According to an embodiment, the food management DB 142 and the image DB 144 may exist in a server connected to the refrigerator 10 and the communication unit 150 through the communication unit 150 .

The communication unit 150 may include at least one communication module that connects the refrigerator 10 to a user's terminal or server. For example, the communication unit 150 may include a wireless Internet module such as Wi-Fi or a short-range wireless communication module such as Bluetooth.

The controller 160 may control the overall operation of the refrigerator 10 . The controller 160 may control at least one component included in the refrigerator 10 to provide a user with a function supported by the refrigerator 10 .

The controller 160 may include at least one CPU, an application processor (AP), a controller, a microcomputer, an integrated circuit, and the like. For example, as shown in FIG. 2 , the controller 160 may include a processor 161 , a display controller 162 , a camera controller 163 , an image processing processor 164 , and a voice recognition module 165 . However, the present invention is not limited thereto.

On the other hand, the control unit 160 (or processor 161) according to an embodiment of the present invention, by inputting the voice obtained from the microphone 230 to the artificial intelligence-based voice recognition module 165, the voice can be recognized there is. The voice recognition module 165 may recognize at least one partial voice included in the voice and output a recognition result including the recognized at least one partial voice.

As described above, the partial voice may be identification information (eg, food name) of the specific food or a command related to the function. That is, the voice may include at least one of identification information and a command of the specific food.

For example, the voice recognition module 165 includes a voice recognizer having an artificial neural network (ANN) pre-trained by machine learning, and recognizes the input voice to include the identification information or characteristic information. recognition result can be output.

The processor 161 may perform a function related to food management based on a recognition result of the voice recognition module 165 and a touch input obtained through the touch panel 101 .

Hereinafter, deep learning will be described in more detail as an example of an artificial intelligence technology applied to the voice recognition module 165 with reference to FIGS. 3 to 5 .

3 and 4 are diagrams for explaining deep learning as an example of artificial intelligence applied to a voice recognition module of a wearable device according to an embodiment of the present invention.

Artificial intelligence is a field of computer science and information technology that studies how computers can do the thinking, learning, and self-development that human intelligence can do. Machine learning, one of the research fields of artificial intelligence, can mean a system that makes predictions based on empirical data and improves its own performance through learning. Deep learning technology, which is a type of machine learning, learns by going down to a deep level in multiple stages based on data.

Deep learning can represent a set of machine learning algorithms that extract core data from multiple data as the level increases.

The deep learning structure may include an artificial neural network (ANN), for example, the deep learning structure is a deep neural network such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a deep belief network (DBN). (deep neural network) can be configured.

Referring to FIG. 3 , the artificial neural network may include an input layer, a hidden layer, and an output layer. Each layer includes a plurality of nodes, and each layer is connected to the next layer. Nodes between adjacent layers may be connected to each other with weights.

Referring to FIG. 4 , the computing device (machine) may form a feature map by discovering a certain pattern from the input data 410 . For example, the computing device (machine) may extract the low-level feature 420 , the middle-level feature 430 , and the high-level feature 440 , recognize the object, and output 450 the result.

The artificial neural network can be abstracted with higher-level features as it goes to the next layer.

3 and 4 , each node may operate based on an activation model, and an output value corresponding to an input value may be determined according to the activation model.

An output value of an arbitrary node, for example, the low-level feature 420 may be input to a next layer connected to the node, for example, a node of the middle-level feature 430 . A node of a next layer, for example, a node of the intermediate-level feature 430 may receive values output from a plurality of nodes of the lower-level feature 420 .

In this case, the input value of each node may be a value in which a weight is applied to the output value of the node of the previous layer. The weight may mean the strength of a connection between nodes.

In addition, the deep learning process can be viewed as a process of finding an appropriate weight.

Meanwhile, an output value of an arbitrary node, for example, the middle-level feature 430 may be input to a next layer connected to the corresponding node, for example, a node of the higher-level feature 440 . A node of a next layer, for example, a node of the upper-level feature 440 may receive values output from a plurality of nodes of the intermediate-level feature 430 .

The artificial neural network may extract feature information corresponding to each level by using a learned layer corresponding to each level. The artificial neural network can recognize a predetermined target by sequentially abstracting and utilizing the highest-level feature information.

For example, looking at the face recognition process by deep learning, the computer divides the bright and dark pixels from the input image according to the brightness of the pixels, and distinguishes simple shapes such as borders and edges, things can be distinguished. Finally, the computer can figure out the shape that defines the human face.

The deep learning structure according to the present invention may use various well-known structures. For example, the deep learning structure according to the present invention may be a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Belief Network (DBN), or the like.

RNN (Recurrent Neural Network) is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes with time. .

DBN (Deep Belief Network) is a deep learning structure composed of multi-layered Restricted Boltzman Machine (RBM), a deep learning technique. By repeating Restricted Boltzman Machine (RBM) learning, when a certain number of layers is reached, a Deep Belief Network (DBN) having the corresponding number of layers can be configured.

CNN (Convolutional Neural Network) is a model that simulates human brain function based on the assumption that when a person recognizes an object, it extracts the basic features of the object, then performs complex calculations in the brain and recognizes the object based on the result. am.

On the other hand, learning of the artificial neural network can be accomplished by adjusting the weight of the connection line between nodes (and adjusting the bias value if necessary) so that a desired output is obtained for a given input. In addition, the artificial neural network may continuously update a weight value by learning. In addition, a method such as back propagation may be used for learning the artificial neural network.

Meanwhile, data for learning the artificial neural network may be stored in the memory 140 or the server. Also, according to an embodiment, weights and biases constituting an artificial neural network structure may be stored in the memory 140 . Alternatively, according to an embodiment, the weights and biases constituting the artificial neural network structure may be stored in the embedded memory of the voice recognition module 165 .

Meanwhile, whenever a voice acquired from the microphone 120 is recognized, the voice recognition module 165 may perform a learning process of the voice recognizer using the acquired voice and the recognition result. As the learning process is performed, an artificial neural network structure such as weights may be updated. To this end, the refrigerator 10 may include a learning module. The learning module may be implemented within the voice recognition module 165 or may be implemented separately.

5 is a diagram for explaining a recognition operation and a learning operation of the voice recognition module.

Referring to FIG. 5 , when a user's voice is input through the microphone 120 , the voice recognizer of the voice recognition module 165 may recognize the input voice and output a recognition result. According to an embodiment, the voice recognizer may output one recognition result as the final recognition result, but by repeating or continuously performing the recognition operation a plurality of times and outputting the final recognition result based on the plurality of times recognition results, the input voice It is possible to further improve the recognition accuracy for

On the other hand, the voice recognizer collects the received voice to build a database, and uses feature points extracted from each partial voice included in the acquired voice and the recognition result of each partial voice to learn the voice recognizer. . As described above in FIG. 2 , the partial voice may correspond to food identification information (eg, food name) or a command related to a function provided by the refrigerator 10 .

For example, the voice recognizer may extract a frequency characteristic of each partial voice included in the collected voice. The voice recognizer may extract frequency characteristics of each of the partial voices using a known algorithm such as a mel-frequency cepstral coefficient (MFCC). The voice recognizer may perform learning and recognition operations of the voice recognizer based on frequency characteristics of each of the partial voices.

Specifically, the voice recognizer may extract the frequency characteristic of each of the sections by dividing the input voice by predetermined sections and analyzing a frequency spectrum for each of the divided sections. The speech recognizer may recognize the voice of each section by inputting the frequency characteristic (frequency feature vector) of each section to the artificial neural network. The voice recognizer may classify partial voices included in the acquired voice based on the recognition result, and output a recognition result of each of the partial voices. Each of the partial voices may include at least one section.

The voice recognizer may learn the voice recognizer based on frequency characteristics of each of the partial voices included in the acquired voice and a recognition result of each of the partial voices.

Since such a speech recognizer recognizes a speech having a frequency pattern that changes over time, the artificial neural network may be configured of a recurrent neural network (RNN), a convolutional neural network (CNN), or a combination thereof.

Through learning of the voice recognizer, parameters (weight and bias) of the artificial neural network included in the voice recognizer may be updated. The database may be stored in the memory 140 of the refrigerator 10 or the embedded memory of the voice recognition module 165 , or in the memory of a server (not shown) connected to the refrigerator 10 .

Hereinafter, an operation in which the refrigerator 10 according to an embodiment of the present invention provides various functions related to food management through artificial intelligence-based voice recognition will be described with reference to FIGS. 6 to 10 .

6 is a flowchart illustrating an operation of a refrigerator according to an embodiment of the present invention.

Referring to FIG. 6 , when the door 11 is opened by the user ( S100 ), the refrigerator 10 may acquire an image of the interior of the refrigerator 10 using the camera 110 ( S110 ). For example, the refrigerator 10 may acquire at least one image using the camera 110 until the door 11 is closed ( S120 ).

When it is detected that the door 11 is opened, the processor 161 of the controller 160 may transmit a control signal to the camera controller 163 to acquire an image of the interior of the refrigerator 10 using the camera 110 . For example, the refrigerator 10 may be provided with a detection sensor that detects the opening/closing or opening angle of the door 11 , and the processor 161 detects the opening of the door 11 using the detection sensor. can

The camera controller 163 may control the camera 110 to acquire an image inside the refrigerator 10 based on the received control signal. The image inside the refrigerator 10 may include a food image of each of at least one food accommodated in a storage compartment of the refrigerator 10 .

The camera 110 may acquire at least one image while the door 11 is opened. The processor 161 or the image processing processor 164 may store the acquired at least one image in the image DB 144 of the memory 140 . Alternatively, the processor 161 or the image processing processor 164 may store only one image among at least one acquired image in the image DB 144 . Any one image stored in the image DB 144 may be an image obtained when the opening angle of the door 11 is the maximum or may be an image obtained last among the at least one image, but is not limited thereto. .

According to an embodiment, the camera 110 may acquire an image of the interior of the refrigerator 10 when the opening angle of the door 11 corresponds to a preset opening angle. In this case, the processor 161 detects the opening angle of the door 11 using the detection sensor, and when the detected opening angle corresponds to a preset opening angle, the processor 161 may transmit a control signal to the camera controller 163. .

When the closing of the door 11 is sensed, the processor 161 may transmit a control signal to the camera controller 163 to end the operation of the camera 110 .

The refrigerator 10 may receive a user's touch input through the display apparatus 100 and may receive a user's voice through the microphone 120 (S130).

The processor 161 may transmit a control signal to the display controller 162 to display the image obtained in step S110 through the display apparatus 100 . For example, the processor 161 may transmit the control signal after the door 11 is closed or transmit the control signal when receiving a request for displaying an image from the user through the touch panel 101 or the microphone 120 . .

The display controller 162 may control the display 102 to display the image in response to the received control signal.

According to an embodiment, the processor 161 may control the light inside the refrigerator 10 so that the inside of the refrigerator 10 is visible from the outside instead of displaying the image, so that the user can check the inside of the refrigerator 10 . . A related embodiment will be described in more detail later with reference to FIG. 10 .

Based on the image displayed through the display 102 , the user may input a command related to a specific food included in the image. For example, a user may touch a position where the specific food is displayed among a plurality of foods included in the image, and a voice including the command or a voice including identification information (food name, etc.) of the specific food and a command can ignite.

The processor 161 may receive a user's touch input through the touch panel 101 and may receive the uttered voice through the microphone 120 .

The touch input and the voice may be simultaneously received or sequentially received.

The refrigerator 10 may recognize the received voice through the AI-based voice recognition module 165 (S140).

The processor 161 may input the received voice to the voice recognition module 165 . According to an embodiment, the processor 161 may process the received voice (filtering, amplification, noise removal, etc.) and input the processed voice to the voice recognition module 165 .

As described above, the voice recognition module 165 may include a voice recognizer to which an artificial neural network based on machine learning, which is a field of artificial intelligence, is applied.

The voice recognition module 165 recognizes at least one partial voice (food identification information and/or command) included in the input voice through the voice recognizer, and outputs a recognition result including the recognized information can do.

The refrigerator 10 may recognize food based on the coordinates of the touch input and food identification information in the recognized voice (S150). The refrigerator 10 may process the command by performing an operation related to the command in the recognized voice with respect to the recognized food ( S160 ).

The processor 161 may obtain the coordinates of the touch input received in step S130 . The processor 161 may recognize the selected food from among the foods included in the image from the identification information of the food included in the recognition result of the voice recognition module 165 and the coordinates of the touch input.

According to an embodiment, when food information corresponding to the coordinates of the touch input is stored in the food management DB 142 , the processor 161 may include identification information of food included in the recognition result of the voice and the touch input. From the food information corresponding to the coordinates of , it may be confirmed whether the food selected by the touch input matches the food included in the voice. As a result of the check, when the food selected by the touch input matches the food included in the voice, the processor 161 may recognize the selected food. On the other hand, if they do not match, the processor 161 may request the user to re-enter.

According to another embodiment, when identification information of food is not included in the recognition result of the voice recognition module 165 , the processor 161 acquires food information corresponding to the coordinates of the touch input from the food management DB 142 . It can also recognize food.

The processor 161 may perform an operation related to a command included in the recognition result of the voice recognition module 165 on the recognized food.

For example, the command may be related to various functions such as registration, purchase, recipe provision, and other management of recognized food. The processor 161 may provide any one function for the recognized food according to a command included in the recognition result.

That is, according to an embodiment of the present invention, the refrigerator 10 may accurately recognize food identification information and commands from the user's voice through the AI-based voice recognition module 165 . Also, the refrigerator 10 may more accurately recognize the food included in the image acquired by the camera 110 based on the coordinates of the touch input and the identification information of the food.

7 is an exemplary diagram for explaining an operation of the refrigerator registering a specific food in a food management database in relation to the operation of the refrigerator shown in FIG. 6 . FIG. 8 is a flowchart illustrating an operation in which an image processing processor of a refrigerator extracts an image of food in relation to the embodiment of FIG. 7 .

Referring to FIG. 7 , the refrigerator 10 may display an image 700 of the interior of the refrigerator 10 acquired by the camera 110 through the display device 100 . The image 700 may include a food image of each of at least one food stored in the refrigerator 10 .

The user USER may touch 702 a position corresponding to any one food image 701 among at least one food image included in the image 700 . Also, the user USER may utter a voice 703 including a registration command for registering the one food image 701 in the food management DB 142 . For example, when the food to be registered is 'AA milk', the user may utter a voice 703 such as "Hi LG, register AA milk". The voice 703 may include 'AA milk' as food identification information (food name) and 'registration' as a command.

The processor 161 may receive a touch input corresponding to the touch 702 through the touch panel 101 and obtain coordinates of the received touch input.

In addition, the processor 161 may receive the voice 703 through the microphone 120 and input the received voice 703 to the voice recognition module 165 . The voice recognition module 165 may recognize the identification information (food name) and commands included in the voice 703 through the above-described artificial intelligence-based voice recognizer.

The processor 161 may generate the food management information 704 for the food (AA milk) based on the recognition result of the voice 703 and the coordinates of the touch input. The food management information 704 may include information such as coordinate values of the food, identification information (food name), unique information (eg, unique user identifier (UUID), etc.), registration date, and the like.

That is, the processor 161 may recognize that the food (AA milk) exists at a position corresponding to the coordinate value in the image 700 . Accordingly, it is possible to identify the food included in the image without recognizing the food from the image through the image processing technique.

The processor 161 may complete the registration of the food by storing the generated food management information 704 in the food management DB 142 of the memory 140 . When the food registration is completed, the processor 161 may notify the user that the food has been registered through the speaker 130 or the display 102 .

In addition, the processor 161 may control the display controller 162 and the display 102 to display the tag 705 including information related to the food (AA milk) at a position corresponding to the coordinate value. The displayed tag 705 may include at least a portion of the generated food management information 704 .

That is, according to the embodiment of FIG. 7 , the refrigerator 10 may accurately recognize food to be registered based on a touch input and a voice input. In addition, since the user can simplify the procedure for registering food with one touch input and voice input, the user's convenience can also be increased.

Meanwhile, the refrigerator 10 may extract the food image 701 of the food from the image 700 based on the coordinate value of the touch input.

In this regard, referring to FIG. 8 , the refrigerator 10 may receive the above-described touch input and a voice including a food registration command ( S200 ).

The refrigerator 10 may extract a food image including the coordinates of the received touch input through an image processing technique (S210).

The processor 161 may transmit a control signal including the coordinates of the touch input to the image processing processor 164 . The image processing processor 164 may extract the food image 701 including the coordinates of the touch input from the image 700 in response to the received control signal.

For example, the image processing processor 164 uses a known contour (or edge) extraction technique to extract the contour of the partial image including the coordinates in the image 700, and based on the extracted contour, the food image ( 701) can be extracted. Alternatively, the image processing processor 164 may extract the food image 701 using various known image processing techniques.

The processor 161 generates the food management information 704 including the food image 701 extracted by the image processing processor 164 , and stores the generated food management information 704 in the food management DB 142 . can be saved

Meanwhile, the food image 701 may be stored in a separate database. In this case, the refrigerator 10 or the server may build an artificial intelligence-based image recognizer similar to the above-described voice recognizer using food images accumulated and stored in the database. Accordingly, the refrigerator 10 may accurately recognize each of the various foods included in the image acquired through the camera 110 later through the image recognizer.

9 is an exemplary view for explaining an operation of the refrigerator providing a purchase function of a specific food in relation to the operation of the refrigerator shown in FIG. 6 .

Referring to FIG. 9 , the refrigerator 10 may display an image 900 of the interior of the refrigerator 10 acquired by the camera 110 through the display device 100 . The image 900 may include a food image of each of at least one food stored in the refrigerator 10 .

The user USER may touch 902 a position corresponding to any one food image 901 among at least one food image included in the image 900 . Also, the user USER may utter a voice 903 including a purchase command for requesting purchase of food corresponding to any one of the food images 901 . For example, when the food to be registered is 'AA milk', the user may utter a voice 903 such as "Hi LG, buy 1000 ml of AA milk in 3 days". The voice 903 may include 'AA milk 1000ml' as food identification information (food name), and 'purchase after 3 days' as a command.

The processor 161 may receive a touch input corresponding to the touch 902 through the touch panel 101 and obtain coordinates of the received touch input. The processor 161 may obtain food management information corresponding to the coordinates from the food management DB 142 .

In addition, the processor 161 may receive the voice 903 through the microphone 120 and input the received voice 903 to the voice recognition module 165 . The voice recognition module 165 may recognize the identification information (food name) and commands included in the voice 903 through the above-described artificial intelligence-based voice recognizer.

The processor 161 may check whether the food (AA milk) matches or not based on the recognition result of the voice 903 and the food management information corresponding to the coordinates of the touch input.

That is, when the identification information (food name) included in the recognition result matches the food name included in the food management information, the processor 161 may recognize the food (AA milk). On the other hand, when the identification information (food name) included in the recognition result and the food name included in the food management information do not match, the processor 161 may request the user to re-enter.

When the food is recognized, the processor 161 may process a purchase command included in the recognition result.

For example, the processor 161 may update the food management information 904 to include at least one of an order timer, an order food, an order capacity, and an order quantity, based on the purchase instruction.

The processor 161 may store the updated food management information 904 in the food management DB 142 of the memory 140 . The processor 161 may notify that the user's command has been processed through the speaker 130 or the display 102 .

In addition, the processor 161 controls the display controller 162 and the display 102 to display the tag 905 including at least a part of the updated management information of the food (AA milk) at a position corresponding to the coordinate value. You can also control it.

Although not shown, the processor 161 may perform a function of purchasing AA milk after 3 days based on the updated food management information 904 . The processor 161 may perform the purchase function by connecting to a server providing a food sales service through the communication unit 150 and transmitting purchase information for purchasing AA milk to the server. For example, the purchase information may include food information, address information, payment information, and the like.

That is, according to the embodiment of FIG. 9 , the refrigerator 10 may accurately recognize the food that the user wants to purchase based on the touch input and the voice input. In addition, since the user can simplify the procedure for purchasing food with one touch input and voice input, the user's convenience can also be increased.

Although not shown, the refrigerator 10 may provide additional functions other than the above-described food registration function and purchase function in a similar manner.

For example, the refrigerator 10 may provide recipe information about the recognized food based on the touch input and the voice input. Similarly to FIG. 9 , the processor 161 accesses the recipe providing server through the communication unit 150 , obtains recipe information related to the recognized food from the server, and displays the obtained recipe information through the display 102 . can be provided to users.

As another example, the refrigerator 10 may perform a function of deleting food management information from the food management DB 142 based on the touch input and the voice input. For example, an image displayed on display 102

10 is a flowchart for describing an embodiment in which the refrigerator recognizes food based on a user's touch input when the display device is implemented as a transparent display so that food inside the refrigerator is transparent to the outside.

The display apparatus 100 of the refrigerator 10 according to an embodiment of the present invention may be implemented as a transparent display. In this case, the user can visually check the foods in the refrigerator 10 through the display 102 .

For example, when a light (illumination) corresponding to the storage compartment of the refrigerator 10 is turned on, the interior of the storage compartment becomes bright, and as a result, the interior can be visualized through the display 102 . On the other hand, when the light corresponding to the display 102 is turned on and the light corresponding to the storage compartment is turned off, visualization of the inside of the storage compartment may be impossible by the screen displayed on the display 102 . On the other hand, when the light corresponding to the storage compartment and the light corresponding to the display 102 are respectively turned on, the user may check a screen displayed overlapping the foods in the storage compartment.

Based on this, referring to FIG. 10 , the refrigerator 10 may acquire an image including foods inside the refrigerator ( S300 ). Step S300 is substantially the same as steps S100 to S120 of FIG. 6 , and a description thereof will be omitted.

The refrigerator 10 may control the light so that the food inside the refrigerator is transmitted through the display 102 (S310).

The processor 161 may turn on a light corresponding to the storage compartment, so that the food stored in the storage compartment is transmitted through the display 102 to the outside. As the light is turned on, the user can visually check the food through the display 102 .

The refrigerator 10 may receive a touch input of coordinates corresponding to any one of the projected foods (S320).

Similar to step S130 of FIG. 6 , the user may touch a position through which any one of the foods is seen through the display 102 . In addition, the user may utter a voice for receiving the function for the food.

The processor 161 may receive a user's touch input through the touch panel 101 and obtain coordinates of the touch input based on the received touch input.

In addition, the processor 161 may receive the voice through the microphone 120 and recognize the voice through the voice recognition module 165 .

The refrigerator 10 may convert the obtained coordinates of the touch input into the coordinates of the image obtained in step S300 (S330), and recognize food included in the image based on the converted coordinates (S340).

For example, the position of the food projected to the outside through the display 102 and the position of the food included in the image acquired by the camera 110 may be different from each other. In this case, the processor 161 needs to recognize which food corresponds to the food corresponding to the coordinates of the touch input among the foods included in the image acquired by the camera 110 .

Accordingly, the processor 161 converts the coordinates of the touch input into coordinates in the image, and recognizes where the food selected by the user through the touch input exists in the image.

Meanwhile, when foods in the refrigerator are seen through the display 102 , coordinates of a touch input for the same food may be different depending on the user's eye position.

The refrigerator 10 may further include a camera disposed outside the door or outside the main body to detect the user's eye position. The processor 161 may convert the coordinates of the touch input into reference coordinates based on the eye position detected using the camera, and convert the converted reference coordinates into coordinates in the image.

The processor 161 may accurately recognize food based on the converted coordinates and the recognition result of the voice recognition module 165, and may provide a function for the recognized food.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments.

The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (11)

a display device installed on the door and including a touch panel and a display;
A camera for acquiring an image including the food stored therein;
a display controller controlling the display to display the acquired image;
a microphone for acquiring voice;
a memory for storing food management information;
a voice recognition module for recognizing the acquired voice and outputting a recognition result; and
Obtaining the coordinates of the touch input received through the touch panel,
A processor for recognizing a specific food selected by a user from among the foods included in the image based on the obtained coordinates and the recognition result of the voice and performing a function on the recognized food,
The processor is
Obtaining food management information of food corresponding to the obtained coordinates from the memory,
Comparing the identification information included in the obtained food management information with the identification information included in the recognition result of the voice,
As a result of comparison, if the identification information included in the acquired food management information matches the identification information included in the recognition result, the specific food is recognized and a function for the recognized food is performed
Refrigerator. According to claim 1,
The voice recognition module,
A speech recognizer having a recognition model for recognizing each of the at least one partial speech included in the speech based on the frequency characteristic of the speech,
Each of the at least one partial voice is identification information of food included in the image or a command related to the function of the food refrigerator. 3. The method of claim 2,
The voice recognizer,
The refrigerator is configured to update the recognition model by using a frequency characteristic of each of the at least one partial voice and a recognition result of each of the at least one partial voice. 3. The method of claim 2,
When the recognition result includes a command related to the registration of the food, the processor,
To generate food management information including the obtained coordinates and identification information of the food,
A refrigerator that stores the generated food management information in memory. 5. The method of claim 4,
The display controller is
A refrigerator for controlling the display to display a tag including at least a portion of the food management information at a position corresponding to the coordinates. 3. The method of claim 2,
If the recognition result includes a command related to the purchase of the food, the processor,
Obtaining food management information corresponding to the coordinates from the memory,
Refrigerator for updating the food management information to include at least one of an order food obtained from the recognition result, an order quantity, and an order timer, based on the command. 7. The method of claim 6,
The processor is
Based on the command, access to the food sales service providing server through the communication unit,
A refrigerator for transmitting the purchase information of the food to the sales service providing server. delete According to claim 1,
Refrigerator further comprising an image processing processor for extracting a food image including the coordinates from the acquired image based on the acquired coordinates. According to claim 1,
The display is implemented as a transparent display,
The processor is
Receives a touch input corresponding to a position through which any one of the foods seen to the outside through the transparent display is transmitted through the touch panel,
A refrigerator that converts the coordinates of the received touch input into corresponding coordinates on the acquired image. 11. The method of claim 10,
The processor is
A refrigerator for recognizing food corresponding to the converted coordinates in the image based on the converted coordinates and a result of recognition of the voice, and performing a function on the recognized food.

Images