WO2019160251A1

WO2019160251A1 - Refrigerator and method for processing information using artificial intelligence according to input pattern

Info

Publication number: WO2019160251A1
Application number: PCT/KR2019/000926
Authority: WO
Inventors: 박윤식; 김상윤; 황성목
Original assignee: 엘지전자 주식회사
Priority date: 2018-02-13
Filing date: 2019-01-22
Publication date: 2019-08-22
Also published as: KR20190097903A; KR102018454B1

Abstract

The present invention relates to a refrigerator and a method for processing information using artificial intelligence according to an input pattern. The present invention comprises a refrigerator to which an interface unit is coupled, wherein the interface unit outputs a capture image captured by a camera unit, receives a touch input which selects and identifies a first area in the captured image by using one or more lines, and displays the received touch input; and a method for displaying same.

Description

Refrigerator and method for processing information using artificial intelligence according to input pattern

The present invention relates to a refrigerator and a method for processing information according to an input pattern.

Refrigerators, which are devices for maintaining or cooling various types of storage at low temperatures, accommodate various types and shapes of products. These products have different characteristics in terms of material, manufacture and use, so different management methods are required to manage them. However, in order to manage the various products, the user must record the information about the product one by one, but it is not performed properly due to the cumbersome procedure.

1 is a diagram of a technique for controlling a setting in a conventional refrigerator. The technique shown in FIG. 1 is a technique for displaying a refrigerator management screen including a refrigerator map when a preset condition is satisfied with the contents of the application number 10-2015-0184542. The external display of the refrigerator is displayed as the information on the expiration date of the products placed in the refrigerator is indicated at 1. However, inputting such information is a method in which a user uploads shopping information, inputs product information through voice, or inputs information to a tag added to a captured image.

Uploading shopping information by the user may be inaccurate in the one-to-one correspondence between the refrigerator and the product, and inputting information by voice may also result in missing or mismatched product information. In addition, in a state where the products are superimposed in the refrigerator and instructed, inputting a tag to the image in the refrigerator may also lower the accuracy when a procedure for distinguishing individual products in the captured image is not presented. That is, the prior art of FIG. 1 shows the photographed image of the food material in the refrigerator as it is or provides a simple management menu, but the photographed image in the refrigerator and UX (User eXperience) for outputting the same are absent.

Therefore, it is necessary to provide a function of controlling the food material in various ways and intuitively and conveniently managing the food material according to various types of inputs for selecting the image.

In the present specification, to solve the above-described problem, a method and apparatus for selecting an image of a product stored in the refrigerator to receive information about the product from the outside or to set the information about the product to manage the product in the refrigerator To provide.

In addition, the present disclosure is to provide a method and apparatus for diversifying the operation performed on the product according to the pattern by diversifying the pattern of the touch input input to the image of the product.

In addition, the present specification is to provide a method and apparatus that can manage the product in the refrigerator in any other space or outside of the refrigerator by receiving or setting information about the product in the refrigerator in the external communication device.

In addition, the present specification is to provide a method and apparatus for using the product in the refrigerator by receiving information on the optimal use of the product from the server when the user selects the product using the interface outside the refrigerator to optimally use the product .

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention, which are not mentioned above, can be understood by the following description, and more clearly by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

According to an embodiment of the present invention, a refrigerator coupled to an interface unit for outputting a captured image captured by a camera unit, receiving a touch input divided by one or more lines selecting a first region of the captured image, and displaying the same, and displaying the same. It includes.

According to an embodiment of the present invention, a refrigerator for outputting the product information to the interface unit in a different manner according to a pattern in which the form of the touch input line is matched using two or more pattern information for differentiating product information. How to display it.

According to one embodiment of the invention includes a communication unit for transmitting the captured image to the communication device, and a communication device for outputting the captured image to receive and output information about the product from the cloud server corresponding to the touch input thereto.

According to an embodiment of the present invention, the communication unit receives the optimal use temperature information of the product in the refrigerator selected from the captured image from the cloud server, and the control unit controls the temperature of the storage space in which the product is received so as to reach the optimum use temperature information and And a refrigerator for monitoring and outputting the usable state of the product to the interface unit when the temperature of the product reaches the optimum use temperature, and a method of displaying the same.

When the present invention is applied to the product stored in the refrigerator using the interface disposed on the door or an external communication device to select the image of the product to receive information about the product or to set the information necessary for the management of the product in the refrigerator Can manage.

In addition, when applying the present invention can diversify the pattern of the touch input input to the image of the product to diversify the operation performed on the product according to the pattern to provide a convenient and intuitive interface.

In addition, when the present invention is applied, an external communication device may receive or set information on a product in the refrigerator, thereby managing the product in the refrigerator anywhere.

The effects of the present invention are not limited to the above effects, and those skilled in the art can easily derive various effects of the present invention from the configuration of the present invention.

1 is a diagram of a technique for controlling a setting in a conventional refrigerator.

2 is a view showing a refrigerator for identifying a stored product to which an embodiment of the present invention is applied.

3 is a view showing the components for providing information about the product inside the refrigerator according to an embodiment of the present invention.

4 is a view showing the configuration of a cloud server according to the present invention.

5 is a diagram illustrating a process of operating the components described with reference to FIGS. 3 and 4 of FIG. 3 according to an embodiment of the present invention.

FIG. 6 is a diagram of outputting information of a product through a touch and capture gesture of an image in a refrigerator according to one embodiment of the present invention; FIG.

7 is a diagram illustrating a process of confirming information by using an image in a refrigerator in a communication device according to an embodiment of the present invention.

FIG. 8 is a diagram illustrating information received by a communication server through a cloud server about a product in a refrigerator according to an embodiment of the present invention.

9 is a diagram illustrating a table in which a user draws a pattern and an operation or information corresponding thereto according to an embodiment of the present invention.

FIG. 10 is a diagram illustrating a process of a refrigerator processing information by using the pattern illustrated in FIG. 9.

FIG. 11 is a diagram illustrating an operation process of a refrigerator when a user performs a touch input with a specific pattern on a product set by a pattern according to an embodiment of the present invention.

FIG. 12 is a diagram illustrating various patterns of products in a refrigerator according to one embodiment of the present invention; FIG.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention.

In order to clearly describe the present invention, parts irrelevant to the description are omitted, and like reference numerals designate like elements throughout the specification. In addition, some embodiments of the invention will be described in detail with reference to exemplary drawings. In adding reference numerals to components of each drawing, the same components may have the same reference numerals as much as possible even though they are shown in different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description may be omitted.

In describing the components of the present invention, terms such as first, second, A, B, (a), and (b) can be used. These terms are only to distinguish the components from other components, and the terms are not limited in nature, order, order, or number of the components. If a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected to or connected to that other component, but between components It is to be understood that the elements may be "interposed" or each component may be "connected", "coupled" or "connected" through other components.

In addition, in the implementation of the present invention may be described by subdividing the components for convenience of description, these components may be implemented in one device or module, or one component is a plurality of devices or modules It can also be implemented separately.

In the present specification, a device for refrigerating or freezing storage items will be described based on a refrigerator, which includes various refrigeration devices including a refrigerator for storing ordinary food, a kimchi refrigerator, a beverage refrigerator, a home refrigerator, a commercial refrigerator, and a freezer only. It includes all the devices mainly for refrigeration. In addition, the present invention also applies to a device for refrigerating non-food storage, such as a cosmetic refrigerator, and also includes a refrigeration device installed in a mobile station that is not fixed, for example, a large refrigeration trailer.

The present specification provides a function to enable simple and efficient management of various food materials through providing a convenient and intuitive UX on the display screen which can be taken in the refrigerator with a camera and can be touched. That is, by adding additional information to the images of the photographed products, and displaying a menu, a menu can be applied to manage the specific products.

In particular, the type of processing information about the product may be set differently according to gestures such as touch and capture of different patterns performed by the user with respect to the displayed image. In this case, when the user notifies the food material to be managed, at the same time, the product is recognized and automatically classified so that the food material can be easily managed / utilized.

The user can perform the touch and capture gestures according to a predefined pattern in order to obtain the necessary information about the product, thereby identifying the necessary information by dividing the products intuitively and conveniently. In addition, since the management method for the products in the refrigerator is automatically set, it is possible to quickly and easily manage the food materials in the refrigerator. In addition, it is possible to mark separately for food ingredients for cooking to enable automatic sorting. The word inside the refrigerator refers to the inside of a device called a refrigerator, and refers to a freezer compartment, a fresh room, a refrigerator compartment, and the like.

In the present specification, the camera unit photographs the products in the refrigerator and is disposed on an outer surface of the refrigerator to output an image captured by an interface unit that combines a display function of outputting an image and a function of receiving a touch input. The interface unit generates and stores information selected according to the characteristics of the trajectory of the selected product according to the trajectory of the touch input received from the user. This operation may be performed in a separate communication device connected through communication with the refrigerator.

In the present specification, information is generated and stored differently according to the characteristics of the trajectory of the selected product according to the trajectory of the touch input for checking the product from the outside and selecting the product from the outside through the transparent interface unit without photographing the product. . In this case, a separate photographing may not be necessary when receiving a touch input, but an image captured in the refrigerator may be used in the process of extracting or storing product information.

The images extracted through the user's touch notation and capture are translated into the language of the letters on the product through the vision and text recognition functions performed by the product and the cloud server. It can be output in connection with various information such as the case, and the user can display additional additional information and store it in the storage unit in the refrigerator.

2 is a view showing a refrigerator for identifying a stored product to which an embodiment of the present invention is applied. 10 is an exterior of the refrigerator 100 in a closed state, and 20 is an exterior of the refrigerator 100 in an open state. The space in which any one door 21 of the plurality of

doors

21, 31, 41, and 51 constituting the refrigerator 100 manages opening and closing may be divided into a plurality of divided

storage spaces

23 and 24. Each of the

storage spaces

23 and 24 may be temperature controlled independently. Of course, the temperature may be equally controlled for the spaces opened and closed by one door. And each storage space can be divided into a shelf again inside.

The camera unit 120 may be disposed in the door or the side or the rear of the internal storage space of the refrigerator 100. This is to identify products stacked at various angles.

The refrigerator 100 includes an interface unit 110. The interface unit 110 provides a display function of displaying information or displaying the inside of the refrigerator 100 in an image or transparently and receiving a user's touch input from the outside. When the interface unit 110 provides a transparent display function, the interface unit 110 is disposed on the door of the refrigerator and allows the inside of the refrigerator to be checked from the outside.

The interface unit 110 may be disposed at the front of the specific door 31 or may be disposed at the side of the refrigerator 100. The interface unit 110 may include a transparent display panel that can check the inside of the refrigerator without the camera unit 120 being photographed. In addition, the display 110 may include a display panel that displays an image captured by the camera unit 120 in the refrigerator. The interface unit 110 may be coupled to a touch panel that receives an external touch input.

The interface unit 110 provides a function of displaying products in the storage space as an image or displaying a description of each product. The interface unit may be disposed in a door of the refrigerator or a side of the refrigerator, and outputs a captured image captured by the camera unit 120 and selects a touch input that is divided into one rational line to select a specific region (first region) among the captured images. Can be received.

Even when the interface unit 110 provides a transparent display function that shows the inside of the refrigerator without any image processing, the user selects a specific region (first region) on the interface unit 110 for a specific product visible from the outside. A touch input divided into one or more lines may be received.

In this case, the specific area means a user's area of a product, and the user draws a shape having a closed curve pattern such as a circle or a square so that the user includes the periphery of the product or a large part of the product, or the center of the product. Checking or marking notation like X is touch input. Of course, the interface unit 110 may output an image such as a check or an X along the touch input trajectory so that the user can check the result of the touch input.

The camera unit 120 photographs a product stored in the storage space. The storage space may be photographed and output from the interface unit 110. Alternatively, the image photographed by the communication unit 160 may be transmitted to an external communication device.

The image extractor 130 extracts an image of a product corresponding to the first region determined by a touch input, such as selecting a specific region by the interface unit 110. Even when the interface unit 110 receives a touch input in a closed curve pattern over a part of the product or receives a touch input such as a check / ex table, the image extractor 130 extracts the entire image of the product.

Meanwhile, when a touch input is received according to the transparent display function, the image extractor 130 checks which product the touch input corresponds to and determines the product corresponding to the first area from the image captured by the camera unit 120. You can extract the image. For example, when the camera unit 120 converts and applies location information on which a touch input is generated on an image captured by the camera unit 120, the user may select a product to be selected by the touch input.

The temperature sensing unit 140 senses the temperature of the product in the storage space. The temperature sensing unit 140 may directly sense the temperature of a specific product. Alternatively, the temperature of the product may be estimated by sensing the temperature change in the storage space where the specific product is placed.

The communication unit 160 transmits the extracted image to the cloud server to receive information about the product. The information on the product may be information on the recognition of the characters included in the product image, information on which product group the product belongs to, or management information on the product (expiration date, key management, storage temperature, etc.). In this case, the information on the product may be different according to the above-described pattern of the touch input. Alternatively, the information about the product may be stored differently according to the pattern of the touch input in the storage 190.

The controller 150 outputs the information previously received by the communication unit 160 to the interface unit 110. That is, the controller 150 stores the extracted image of the product and the received information and outputs the information and the image to the interface unit 110. Thereafter, additional information additionally input by the user in the interface unit 110 may also be stored in the storage unit 190.

The storage unit 190 stores an image of the product and information about the product. In particular, information on the management or the expiration date of the product may be stored, and the controller 150 searches the information stored in the storage 190 at regular intervals, so that the expiration date is imminent or needs management, or a specific time. An alarm may be provided to the user by using information set as being cooked in the kitchen.

4 is a view showing the configuration of a cloud server according to the present invention. The communication unit 26 of the cloud server 200 receives an image from a plurality of refrigerators. The image recognition unit 230 recognizes a character from a label or description included in the received product image, or recognizes the character from the image.

In addition, the image recognition unit 230 may calculate information on the shape or color of the product image. Thereafter, the recognized characters and the calculated information search the information in the searcher 220 to generate information on the corresponding product.

The search unit 220 may include a kind of database, and may classify and store product images transmitted from the plurality of refrigerators and information recognized therein. In addition, the search unit 220 may store images and information about various products that can be worn in the refrigerator by connecting to an external Internet. The controller 250 controls the aforementioned components.

The camera unit 120 of the refrigerator 100 photographs the inside of the refrigerator (S401). In this case, one camera unit 120 may be disposed in the storage space or a plurality of camera units 120. In the case of multiple arrangements, these images may be combined or displayed sequentially.

The interface unit 110 outputs the photographed image, receives an input of a specific pattern based on a touch and a gesture from a user, and marks the input pattern on an image of the product, and the image extracting unit 130 is marked. The product image is extracted (S402). The image of the product may be an image of the whole product or a part of the product.

In the marking process, when a user touches a shape such as a circle or a rectangle with a touch-drag gesture for a specific product, the interface unit 110 displays an outline such as a circle or a rectangle around the product according to the trajectory of the touch input position. Means to display to be identified. Alternatively, when the user touches a check or X, the interface 110 may display a check or X on a product.

In this process, although the interface is slightly different from the trajectory of the position input by the control of the controller 150, the interface unit 110 may output the adjusted ratio of the outline so that the shape of the circle or the quadrangle is more accurately.

The image of the product selected by the user by performing a touch input is extracted and transmitted to the cloud server by the communication unit 160 (S403). The cloud server 200 recognizes information and an image of what mark (circle, rectangle, check, X mark, etc.) the user has made (S404). Recognition of the image may be the cloud server 200 checks the information on the product based on the text recognition, color recognition, the pattern of the marking performed by the user.

The search unit 220 of the cloud server 200 searches for information on a corresponding product, information on a classification in which the corresponding product is included, or information related to the corresponding product in an embedded database using the above-described information (S405). . Information related to the product may be cooking methods, storage temperature, and intake temperature.

In one embodiment, when the product is a beverage, the beverage may be the most appropriate temperature information for drinking. The information retrieved from the cloud server 200 is transmitted to the refrigerator 100 (S406). The refrigerator 100 receiving the information outputs the received information to the interface unit 110.

The process of FIG. 5 captures food in the refrigerator, extracts a marker and a necessary image through touch and gesture on the captured image, and transmits the image to the cloud to classify and related information for food management through marker / text / food recognition. After searching for, it shows a process of outputting the product to pass the information to the refrigerator showing.

The interface unit 110 of the refrigerator 100 photographs and displays an image therein. Alternatively, in another embodiment, the interface unit 110 may enable the user to check the product inside by activating the transparent display function.

The user makes a touch input on the image of the captured product 301 as indicated at 302. When the user draws a rectangular pattern on the interface unit 110 by a touch input, the image extractor 130 extracts an image of the product 301 on which the pattern is drawn. In addition, the communication unit 160 transmits the extracted image to the cloud server 200 (S310).

The image recognizing unit 230 and the retrieving unit 220 of the cloud server 200 recognizes and translates a character in the transmitted image to recognize which product is the product. Accordingly, the cloud server 200 is a variety of information about the product, for example, classification of the product, information necessary to manage the product, the main information of the product (appropriate temperature, shelf life, etc.), the use case of the product (cooking information) The search is sent again to the refrigerator 100 (S320).

The interface unit 110 of the refrigerator 100 displays the received information. In this case, the content may be displayed as 303 adjacent to the corresponding product 301 so as to easily check the contents.

In FIG. 6, if the internal product is identified by the transparent display function, the user may draw a pattern as shown at 302 at a position corresponding to the internal product. In this case, the interface unit 110 determines which product in the refrigerator the input pattern corresponds to, and the image extractor 130 may transmit an image of a product of a location indicated by the pattern to the cloud server 200.

6 may be implemented using a communication device such as a mobile phone. An image showing the interior of the refrigerator in FIG. 6 may be transmitted to a communication device. In the communication device, the inside of the refrigerator is displayed as a first image, and a user may perform a touch input of a predetermined pattern on a second image (partial image of the first image) of a specific product among the first images using the communication device. In this case, information about the product may be confirmed by transmitting and receiving information between the communication device and the cloud server.

7 is a diagram illustrating a process of confirming information by using an image in a refrigerator in a communication device according to an embodiment of the present invention. The communication device 500 communicates with the refrigerator 100 and the cloud server 200 and has a special purpose application for transmitting and receiving information.

The communication unit 160 of the refrigerator 100 transmits the captured image captured by the camera unit 120 to the communication device 500. The communication device 500 outputs a captured image and receives a touch input that is divided into one or more lines that select a specific region of the captured image. The communication device 500 may extract an image of a product corresponding to the touch input area, transmit the image of the product to the cloud server, receive information about the product, and output the extracted product image and information.

The refrigerator 100 photographs products in the refrigerator (S411) and transmits the same to the communication device 500 (S412). The display unit of the communication device 500 displays an image, and when a user draws a predetermined pattern with a touch and gesture on the screen 510 of the communication device 500 and marks the image, the application of the communication device 500 displays the image. Extract it and transmit it to the cloud server 200 (S414).

The cloud server performs steps S415 and S416 in the same manner as the processes S404 and S405 of FIG. 5, and transmits the retrieved information to the communication device 500. The communication device 500 displays the retrieved information (S418) and the user can check detailed information of the product disposed in the refrigerator remotely.

The image in the refrigerator illustrated in FIG. 6 is displayed on the screen 510 of the smartphone, which is the communication device 500. When the user draws a line on the product indicated by 301 in the same pattern as 302, the communication device 500 transmits an image 301 of the product on which the line is drawn to the cloud server 200, and information is output as shown in 303.

In the lower part of the smartphone, a link (2nd Box, 3rd Box) for checking an image of another storage space in addition to the image of the storage space (1st Box) currently displayed is displayed. When the user clicks each link, the camera unit disposed in another storage space of the refrigerator may capture an image in the refrigerator and transmit the image to the communication device 500.

6 to 8, by vision recognition of the image extracted through the capture gesture with respect to the captured image in the refrigerator, the recognized image is transmitted to the cloud server 200 to recognize food or text and translate, The user can check information on food use methods or management methods for foods that are difficult to identify in a foreign language. In addition, the expiration date of the food or immediate cooking to set the classification for management can be monitored and the related content can be displayed through the refrigerator 100 or the communication device 500.

9 is a diagram illustrating a table in which a user draws a pattern and an operation or information corresponding thereto according to an embodiment of the present invention. Such a table may be stored in the storage 190 of the refrigerator 100. The refrigerator 100 may request the cloud server for necessary information corresponding to the input pattern.

The table 195 stores the types of patterns that a user can draw on the image of the product. Patterns that can be drawn by the user are arranged in the "Pattern" item. Squares, circles, checks, and Xs are listed.

"Action" suggests what to do when the pattern is entered. That is, the controller 150 stores category information on a task to be performed by the controller 150 in response to the pattern. "GET_INFORMATION" means receiving information from the cloud server 200 and interworking the service with the received information. "SET_CATEGORY" automatically classifies products and links necessary services.

"Parameter" lists the kind of information to be requested or the kind of category to be set. That is, it includes parameter information necessary to perform a task. For example, for rectangular patterns, we request text recognition (TEXT_RECOGNITION), translation (TRANSLATION), information on proper temperature or shelf life or necessary management methods (RELATED_DATA), and information on how to use this product (USAGE_INFORMATION). It is stated.

"Parameter" information may be transmitted to the cloud server 200 by the communication unit. That is, the communication unit 160 transmits the parameter information of the pattern corresponding to the extracted image and the touch input to the cloud server 200, and what kind of information is the information that the cloud server 200 should provide for the corresponding product. You can check.

For the circle pattern, it is specified to request a recipe using the product (RECIPE).

For the check pattern, it means automatic sorting of foods requiring expiration date control (EXPIRATION_SETTING). The X pattern is classified into ingredients (READY_FOR_COOK) to be cooked.

The "Sample_Pattern" item saves the pattern entered by the user. This means a sample pattern previously input corresponding to the pattern. The user cannot draw a square / circle / check / ex pattern exactly as the pattern shown in "Pattern" and draws a shape similar to this stored pattern. Therefore, in order to accurately recognize the patterns input by the user, the patterns input by the user may be stored and compared.

Even if a user inputs a pattern as "Sample_pattern", the refrigerator 100 or the communication device 500 may correct the output and output the pattern in the pattern "Pattern".

For example, when the shape of the touch input line is matched with a first pattern (for example, a quadrangle) defined in a table stored in the storage 190 to classify the product, the controller 150 may apply a line around the product. The ratio of the contour of one pattern may be adjusted to control the output from the interface unit 110.

The storage unit 190 stores two or more pattern information for differentiating product information. Differentiating the information of the product means differentiating the requested information or the applied information according to the pattern displayed on the product. If the pattern input by the user matches the first pattern, the refrigerator requests text recognition and displays the result. If the pattern input by the user matches the second pattern, the refrigerator requests a recipe and displays the result. If the user input pattern matches the third pattern, the refrigerator sets expiration date management. In this way, various patterns input by the user may be applied to differentiate the information in managing or requesting product information. The matching process between the input pattern and the first pattern / second pattern / third pattern includes detailed functions such as matching an image of patterns or matching a direction of a line in the image.

In one embodiment, the table 195 of FIG.

The interface unit 110 generates location information of one or more lines touched on the image captured in the refrigerator or on the transparent display (S421). Position information may be the position information of the start and end points of the lines or points in the middle. The input location information may constitute one closed curve. Alternatively, the inputted position information may form one line having a closed curve but partially open. The location information may be x-y coordinates of the interface unit 110 for each point.

The controller 150 searches for a pattern that matches the shape of the touch input line using the location information (S422). The pattern may be searched for using a "Pattern" item or a past sampled "Sample_Pattern" as shown in FIG. 9. If there is no found pattern (S423) because the user touched or touched incorrectly, the existing image is maintained (S424). Alternatively, the touch input points may not be displayed in the transparent display state.

On the other hand, if there is a found pattern, the controller displays the pattern on the corresponding product (S425). In this case, when a touch input is made on the image, the image extractor 130 may extract the product image from the corresponding image. When a touch is input on the transparent display, the controller 150 determines a product disposed behind the transparent display at the touch input position as a corresponding product.

Thereafter, the controller 150 extracts "Action" and "Parameter" corresponding to the pattern (S426). Information corresponding to the pattern identified in the table 195 of FIG. 9 may be extracted.

The controller 150 processes corresponding to the extracted "Action" and "Parameter" (S427). The processing may vary according to the pattern, and the processed information is stored in the storage unit (S428).

S427 corresponds to "Action" and "Parameter" of FIG. 9 earlier, i) output information of the product to the interface unit 110 in different ways, or ii) store differently set or received information in the storage unit, or iii Or requesting different information from a cloud server. Outputting the information of the product differently means receiving and storing the information of the product from the cloud server 200 and outputting it in another manner.

For example, when a rectangular pattern is input to a specific product, if text recognition and related information are already stored in the storage 190 for the corresponding product, the controller 150 displays the text recognized information and related information in the interface unit ( Output to 110). Meanwhile, when a circle is input to a specific product, the controller 150 outputs the stored recipe of the corresponding product to the interface unit 110.

In addition, when the X pattern and the check pattern are set, different categories may be set and stored in the storage 190 corresponding to the X pattern and the check pattern. Alternatively, since the information received about the product from the cloud server 200 may be different according to the square / circular pattern, the controller 150 may store differently received information in the storage 190.

In addition, the type of information requested to the cloud server 200 may be different according to the pattern, and the controller 150 may transmit a message requesting different information to the cloud server 200 according to the pattern.

The image extractor 130 may extract an image as much as the size of the pattern corresponding to the pattern input by the user, or expand or reduce the image of the product to be extracted according to the type of work applied by the pattern than the pattern. have.

For example, a pattern is set to a specific area, which is described in an embodiment in which this area corresponds only to some areas of the product. If the pattern is a pattern requesting a recipe, an image of the entire product may be needed, and thus the image extractor 130 may extract an image from the entire region of the product.

In addition, when the pattern is set to a specific area, and the area includes not only the first product but also includes a part of another adjacent second product, the image extractor 130 may increase the accuracy of the extraction of the first product image. The image can only be extracted for the first product.

In addition, when the pattern is a pattern (for example, a rectangle) for requesting interpretation or content retrieval of a trademark or text, the image extracting unit 130 extracts an image in which the trademark or text is placed instead of the image of the entire product. You can extract the image by limiting it.

The user may touch input a preset pattern (star, triangle, etc.) for a specific product in the interface unit 110. Such a pattern is called a comfort pattern. That is, the interface unit receives a comfort pattern for a specific product (S431).

The image extractor 130 extracts an image of the product (S432). The controller 150 transmits a message for requesting optimal use temperature information of the product to the cloud server 200 together with the extracted image (S433).

The optimal use temperature information means the most suitable temperature for using or drinking the product. In the case of a beverage, an optimum use temperature may be set for each type of beverage.

The cloud server 200 recognizes an image of a product included in the received message and recognizes a text in the image to classify the corresponding product to generate optimal use temperature information of the product (S434). The optimal use temperature information may be built in the searcher 220 based on the category of the product or the detailed information of the product.

In particular, the producers of the product provides the cloud server 200 with the image and the name of the produced product and the optimum use temperature information, the cloud server 200 in the search unit 220 to retrieve the image of the product transmitted from the refrigerator 100 You can search for to generate the optimum temperature information.

The controller receives the optimal use temperature information of the product from the cloud server (S435). The controller 150 may adjust the temperature in the storage space so that the temperature of the product reaches the optimal use temperature information.

The controller 150 receives the optimal refrigeration temperature information and the optimal use temperature information of the product from the cloud server (S435).

Thereafter, the controller 150 controls the temperature of the storage space by reflecting the optimal refrigeration temperature information (S436). In one embodiment, rapid refrigeration may be performed on the storage space.

Thereafter, the controller 150 controls the temperature sensing unit 140 to monitor the temperature of the product used to check whether the temperature of the product reaches the optimum use temperature (S436). If so, the controller 150 controls the interface unit 110 to indicate that the product is in a usable state (S437).

The user keeps new ingredients, beverages, fruits, etc. in the refrigerator 100 and inputs a comfortable pattern in the image displayed on the interface unit 110. Alternatively, the user inputs a comfortable pattern at the interface unit 110 in front of the food, beverage or fruit which is identified in the refrigerator by the transparent display function.

According to the sequence shown in FIG. 11, the refrigerator 100 operates to receive the optimum use temperature corresponding to the product from the cloud server 200 and to quickly reach the optimum use temperature of the product. Then, when the temperature is reached, the user may use the product by displaying a specific pattern or outputting a message through voice or on the interface unit 110.

FIG. 12 is a diagram illustrating various patterns of products in a refrigerator according to one embodiment of the present invention; FIG. The first shelf in the refrigerator 100 has a rectangular pattern as shown in 311 and a star pattern as shown in 312. The square pattern is an embodiment of receiving information related to a product through the cloud server 200. Star pattern 312 is a comfortable pattern that controls the operation of the refrigerator 100 to drink the milk received at the optimum use temperature.

The X-pattern 313 and the check pattern 314 are disposed on the second shelf, and the third shelf is a state in which the original pattern 315 and the X-pattern 316 are displayed. The type of information each pattern sets for the corresponding product or the type of information requested through the cloud server 200 has been described above.

The pattern is for a user to indicate a specific product or to obtain information by extracting a specific area of the product. Therefore, the controller 150 and the image extractor 130 may extract a partial image or extract an image of the entire product in which the pattern is disposed, according to the type of work or the type of necessary information. Alternatively, if the information is not included in the pattern but the information requested by the pattern is in another part of the image of the product, the corresponding part may be extracted.

Depending on the type of pattern input by the user, the purpose of use is different so that text recognition / translation, expiration date management, food classification, cooking material classification, shopping necessity food classification, etc. can be automatically classified at the same time as the gesture. For example, when inputting a batch check pattern for products requiring management of an expiration date in the refrigerator, the controller 150 may apply a process of managing an expiration date for the corresponding products.

When inputting a batch X pattern for products required for cooking, the controller 150 may continuously output from the interface unit 110 that the products are required for cooking. Alternatively, if the product is still in the refrigerator, a message indicating that the product is not carried out through the interface unit 110 is output, so that the user does not miss a particular material and cooks it.

When the function implemented by the refrigerator in FIG. 12 includes only the X pattern and the check pattern, the communication unit 100 may be excluded from the configuration of FIG. 3. That is, when the interface unit 110 receives a touch input divided into one or more lines selecting the first area of the captured image, the controller 150 checks a product corresponding to the first area and displays a pattern (eg, a touch input). For example, the work to be applied to the product may be set according to an X pattern or a check pattern, where the identification of the product may use only the image of the product and the location of the product (the specific position in the shelf). Saves the job information set for this.

Here, the touch input divided into one or more lines for selecting the first area includes an embodiment in which the first area is selected with two lines in the check pattern. In addition, an embodiment of selecting a first region as one circle in an ellipse pattern is included. In addition, the embodiment includes selecting the first region by two crossing lines in the X pattern. That is, the touch input herein includes a touch input composed of one or more lines to indicate that the first area is selected.

One embodiment of the job information is to set the type of the user can be notified about a specific product through the interface unit 110, such as setting the expiration date, the setting of the cooking material, or a schedule alarm Yes.

The above-described pattern and corresponding information, tasks, and the like may vary according to implementation, and the square / circle / x / check pattern described herein is only an example, and the present invention is not limited thereto.

Although all components constituting the embodiments of the present invention have been described as being combined or operating in combination, the present invention is not necessarily limited to these embodiments, and all of the components are within the scope of the present invention. It can also be combined to operate selectively. In addition, although all of the components may be implemented in one independent hardware, each or all of the components may be selectively combined to perform some or all functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. Codes and code segments constituting the computer program may be easily inferred by those skilled in the art. Such a computer program may be stored in a computer readable storage medium and read and executed by a computer, thereby implementing embodiments of the present invention. The storage medium of the computer program includes a storage medium including a magnetic recording medium, an optical recording medium and a semiconductor recording element. In addition, the computer program for implementing an embodiment of the present invention includes a program module transmitted in real time through an external device.

In the above description, the embodiments of the present invention have been described, but various changes and modifications can be made at the level of ordinary skill. Therefore, it will be understood that such changes and modifications are included within the scope of the present invention without departing from the scope of the present invention.

-부호의 설명-Explanation of sign

100: refrigerator 110: interface unit

120: camera unit 130: image extraction unit

140: temperature sensing unit 150: control unit

160: communication unit 190: storage unit

Claims

One or more separate storage spaces;

A camera unit for photographing a product stored in the storage space;

An interface unit disposed on a door of the refrigerator, the interface unit configured to output a captured image captured by the camera unit and receive a touch input separated by one or more lines for selecting a first area of the captured image;

An image extracting unit extracting an image of a product corresponding to the first region;

A communication unit for transmitting the extracted image to a cloud server and receiving information on the product;

A control unit which stores the extracted image of the product and the information and outputs the information to the interface unit;

And a storage unit which stores an image of the product and information about the product, wherein the refrigerator processes the information according to an input pattern.
The method of claim 1,

The storage unit stores two or more pattern information for differentiating the information of the product,

The information of the product is output to the interface unit in a different way according to the pattern of the touch input line matching, or the information set or received differently according to the pattern of the touch input line matching And the communication unit requests different information from the cloud server according to a pattern stored in a storage unit or the shape of the touch input line is matched.
The method of claim 2,

The storage unit processes information according to an input pattern, storing category information on the pattern, a task to be performed by the controller corresponding to the pattern, the task and necessary parameter information, and a sample pattern input corresponding to the pattern. Refrigerator.
The method of claim 3,

And the communicator transmits parameter information of the extracted image and the pattern corresponding to the touch input to the cloud server.
The method of claim 2,

The controller is configured to adjust the ratio of the contour of the first pattern around the product to the interface unit when the shape of the touch input line is matched with the first pattern stored in the storage unit to distinguish the product. A refrigerator that processes information in accordance with an input pattern to be output.
The method of claim 1,

Further comprising a temperature sensing unit for sensing the temperature of the product in the storage space,

The communication unit receives the optimal use temperature information for the product from the cloud server,

The controller controls the temperature of the storage space so that the product reaches the optimum use temperature information,

The controller controls the temperature sensing unit to monitor the temperature of the product, and when the temperature of the product reaches the optimum use temperature, the controller controls the interface unit to display an available state of the product. Refrigerator processing information according to.
The method of claim 1,

The communication unit transmits the captured image to a communication device,

The communication device outputs the captured image and receives a touch input divided by one or more lines selecting the second area of the captured image, and then extracts an image of a product corresponding to the second area to the cloud server. A refrigerator for processing the information according to an input pattern after receiving the information about the product, and outputs the image and the information of the extracted product.
One or more separate storage spaces;

A camera unit for photographing a product stored in the storage space;

An interface unit disposed on the door of the refrigerator and providing a transparent display function to check the inside of the refrigerator from the outside, and receiving a touch input separated by one or more lines selecting the first area from the outside.

An image extracting unit extracting an image of a product corresponding to the first region from the image photographed by the camera unit;

A communication unit for transmitting the extracted image to a cloud server and receiving information on the product;

A control unit which stores the extracted image of the product and the information and outputs the information to the interface unit; And

And a storage unit which stores an image of the product and information about the product, wherein the refrigerator processes the information according to an input pattern.
The method of claim 8,

The storage unit stores two or more pattern information for differentiating the information of the product,

The information of the product is output to the interface unit in a different way according to the pattern of the touch input line matching, or the information set or received differently according to the pattern of the touch input line matching And the communication unit requests different information from the cloud server according to a pattern stored in a storage unit or the shape of the touch input line is matched.
The method of claim 8,

Further comprising a temperature sensing unit for sensing the temperature of the product in the storage space,

The communication unit receives the optimal use temperature information for the product from the cloud server,

The controller controls the temperature of the storage space so that the product reaches the optimum use temperature information,

The controller controls the temperature sensing unit to monitor the temperature of the product, and when the temperature of the product reaches the optimum use temperature, the controller controls the interface unit to display an available state of the product. Refrigerator processing information according to.
One or more separate storage spaces;

A camera unit for photographing a product stored in the storage space;

An interface unit disposed on a door of the refrigerator, the interface unit configured to output a captured image captured by the camera unit and receive a touch input separated by one or more lines for selecting a first area of the captured image;

A controller configured to identify a product corresponding to the first area and to set a job to be applied to the product according to a pattern of the touch input; And

And a storage unit for storing job information set for the product, wherein the refrigerator processes the information according to an input pattern.
Photographing a product stored in a storage space of a camera unit disposed in the refrigerator;

An interface unit disposed on a door of the refrigerator to output a captured image captured by the camera unit, and receive a touch input for dividing the captured image into one or more lines for selecting a first region of the captured image;

Extracting an image of a product corresponding to the first area by the image extractor of the refrigerator;

A communication unit of the refrigerator transmitting the extracted image to a cloud server and receiving information on the product; And

And outputting the extracted image of the product and the information to the interface unit by a control unit of the refrigerator.
The method of claim 12,

The storage unit of the refrigerator stores the pattern, category information on a task to be performed by the controller corresponding to the pattern, the task and necessary parameter information, and a sample pattern input corresponding to the pattern.

The communication unit further comprises the step of transmitting parameter information of the pattern corresponding to the extracted image and the touch input to the cloud server, the information processing method according to the input pattern.
The method of claim 12,

The temperature sensing unit of the refrigerator senses the temperature of the product in the storage space,

The communication unit of the refrigerator receiving the information on the product

Receiving, by the communication unit, an optimal use temperature information of the product from the cloud server,

The control unit of the refrigerator outputting the extracted image of the product and the information to the interface unit

Controlling, by the control unit, a temperature of the storage space such that the product reaches the optimum use temperature information; And

And controlling the temperature sensing unit to monitor the temperature of the product, and when the temperature of the product reaches the optimum use temperature, controlling the interface unit to display a usable state of the product. To process the information according to the input pattern.