KR20180090724A - Method for controlling refrigerator and refrigerator controller - Google Patents

Abstract

According to an embodiment, a refrigerator controller comprises: an image acquisition unit receiving a current frame photographing an internal condition of a refrigerator; an area detection unit calculating a pixel value with respect to each pixel of the current frame, and comparing the pixel value with another pixel value with respect to a corresponding pixel of a previous frame to detect a change area; a divided image generating unit generating a divided image changing a pixel value of each pixel of the current frame to a first value or a second value based on the change area; a box extraction unit extracting a bounding box having a pixel set with the first value contained in the divided image; and a determination unit determining a change in a condition of the refrigerator by comparing the bounding box extracted through the box extraction unit with at least one pre-stored bounding box.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a refrigerator control method and a refrigerator control method,

An embodiment of the present invention relates to a refrigerator control method and a refrigerator control apparatus.

2. Description of the Related Art Generally, a refrigerator is a device for supplying cold air generated by driving a refrigeration cycle, and is a device for storing food in a low temperature state. In the case of a conventional refrigerator, only the function of storing food in a low temperature state can be performed. However, in recent years, there is an increasing need for additional functions in addition to the food storage function.

The refrigerator is a device for storing and storing a certain object. In order to check the inside of the refrigerator, the refrigerator door must be opened and checked. In addition, when the user tries to buy goods from markets or marts, he or she can not buy the same food or buy necessary food if the amount and type of the food stored in the refrigerator can not be grasped.

Japanese Patent Publication No. 3450907 and Japanese Patent Laid-Open No. 2004-183987 disclose techniques for photographing the inside of a refrigerator by installing a camera on a door. Further, according to Japanese Patent Application Laid-Open No. 2001-294308, a technique is disclosed in which a camera is installed in a room, a drawer, and a door, respectively.

An object of the present invention is to obtain a current frame photographed inside a refrigerator and to determine a state change occurring in the refrigerator through comparison between previous frames photographed previously.

It is also an object of the present invention to extract a bounding box from a current frame based on a change area in which a change occurs and to identify an object included in the bounding box using a pre-trained database.

It is also an object of the present invention to determine whether a state change is a downward trend or a load based on a difference between pixel values of a corresponding pixel between a current frame and a previous frame.

An apparatus for controlling a refrigerator according to embodiments of the present invention includes: an image obtaining unit that receives a current frame of an internal situation of a refrigerator; An area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame; A half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; A box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image; And a determination unit for determining a state change of the refrigerator by comparing the bounding box extracted through the box extracting unit with at least one pre-stored bounding box.

Wherein the determination unit compares the extracted identification value of the bounding box and the identification values of the previously stored bounding boxes to calculate a non-unidirectional figure, and determines that the state change is a load if the unidirectional angle is greater than or equal to a predetermined first threshold value, If the non-inference degree is less than the first threshold value, it is possible to determine the change in status as a decrease.

The area detecting unit may calculate a difference value which is a difference between pixel values of a corresponding pixel of a previous frame stored in a current frame and a buffer, and may set a pixel having a difference value exceeding a predetermined second threshold value as a change area.

The binary image generating unit may generate a binary image in which a pixel value of a pixel included in the change area is changed to a first value and a pixel value of a pixel not included in the change area is changed to a second value.

The judging unit may determine the name of the loaded entity according to the type and color of the entity included in the bounding box using the pre-trained database when the state change is judged to be loaded through the judging unit.

After determining the name of the entity, the determination unit may generate a notification message including the name of the loaded entity and the loading date and time through the determination unit, and provide the generated notification message to the user terminal.

A method of controlling a refrigerator according to embodiments of the present invention includes: receiving a current frame in which a controller captures an internal state of a refrigerator; Detecting the change area by the controller calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value for a corresponding pixel of the previous frame; The controller generating a dichromatic image that changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; The control device extracting a bounding box containing a pixel set to a first value contained in the binary image; And comparing the extracted bounding box with one or more pre-stored bounding boxes to determine a state change of the refrigerator.

Wherein the step of determining the state change comprises: comparing the identification value of the extracted bounding box and the identification values of the previously stored bounding boxes to calculate a non-unidirectional diagram; if the non-inductive diagram is greater than or equal to a predetermined first threshold value, And if the non-inference degree is less than the first threshold value, it is possible to determine the change in state as a decrease.

Wherein the step of detecting the change area comprises: calculating a difference value, which is a difference between pixel values of a current frame and a corresponding pixel of a previous frame stored in the buffer; and calculating a pixel having a difference value exceeding a predetermined second threshold value as a change area Can be set.

The generating of the binary image may generate a binary image in which a pixel value of a pixel included in the change area is changed to a first value and a pixel value of a pixel not included in the change area is changed to a second value.

The step of determining the state change may determine the type of the entity included in the bounding box and the name of the loaded entity based on the color using the pre-trained database when the state change is determined to be the load.

After determining the name of the entity, the determining step may generate a notification message including the name of the entity loaded through the determination unit and the loading date and time, and provide the notification message to the user terminal.

A computer program according to an embodiment of the present invention may be stored in a medium using a computer to execute any one of the control methods of the refrigerator according to the embodiment of the present invention.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium for recording a computer program for executing the method are further provided.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and the detailed description of the invention.

According to the present invention, it is possible to acquire a current frame photographed inside a refrigerator, and to determine a state change occurring in the refrigerator through comparison between previously photographed previous frames.

Further, according to the present invention, a bounding box can be extracted from a current frame based on a change area in which a change has occurred, and an object included in the bounding box can be specified using a pre-trained database.

Further, the present invention can determine whether the state change is a downward change or a load based on a difference between pixel values of a corresponding pixel between a current frame and a previous frame.

1 is a view illustrating a refrigerator control apparatus according to an embodiment of the present invention.
2 to 6 are flowcharts of a method of controlling a refrigerator according to embodiments of the present invention.
7 is a view illustrating an example of a refrigerator including a refrigerator control apparatus according to embodiments of the present invention.
8 is a view for explaining a refrigerator control system according to embodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention and methods of achieving them will be apparent with reference to the embodiments described in detail below with reference to the drawings. However, the present invention is not limited to the embodiments described below, but may be implemented in various forms.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout the drawings, and a duplicate description thereof will be omitted .

In the following embodiments, the terms first, second, and the like are used for the purpose of distinguishing one element from another element, not the limitative meaning.

In the following examples, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise.

In the following embodiments, terms such as inclusive or possessed mean that a feature or element described in the specification is present, and does not exclude the possibility that one or more other features or components are added in advance.

If certain embodiments are otherwise feasible, the particular process sequence may be performed differently from the sequence described. For example, two processes that are described in succession may be performed substantially concurrently, and may be performed in the reverse order of the order described.

In the following embodiments, the term "circuit" refers to any circuitry, circuitry, and / or circuitry, including, for example, hardwired circuitry, programmable circuitry, state machine circuitry, and / or firmware that stores instructions executed by a programmable circuit, either alone or in any combination . The application may be implemented as code or instructions that may be executed on a programmable circuit, such as a host processor or other programmable circuit. A module, as used in any of the embodiments herein, may be implemented as a circuit. The circuitry may be implemented as an integrated circuit, such as an integrated circuit chip.

In the following embodiments, when a component is referred to as "comprising ", it means that it can include other components as well, without excluding other components unless specifically stated otherwise. Also, the terms " part, "" module," and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

1 is a view illustrating a refrigerator control apparatus 110 according to an embodiment of the present invention.

1, a refrigerator control apparatus 110 according to an embodiment of the present invention includes an image obtaining unit 111, a buffer 112, an area detecting unit 113, an image generating unit 114, a box extracting unit 115 ), And a determination unit 116. [

The refrigerator control apparatus 110 according to embodiments of the present invention can acquire images from one or more electrically connected imaging apparatuses. At this time, the refrigerator control device 110 may control to acquire an image from the imaging device when a predetermined signal is sensed from an electrically connected sensor. For example, when the opening and closing of the door is detected, the refrigerator control device 110 can acquire an image from the imaging device. The refrigerator control device 110 acquires an image periodically and senses a state change only when an opening and closing of the door is detected rather than detecting a state change.

The image acquiring unit 111 may acquire a current frame from one or more electrically connected devices. The acquired current frame may be passed to the buffer 112.

The buffer 112 may record the acquired image. When a new frame is received, the buffer 112 may delete the previously stored frame and store the new frame. For example, the buffer 112 may delete the pre-stored frame and store a new frame only when it is determined that the internal state of the refrigerator is changed. The buffer 112 may have as many storage areas as the number of connected imaging devices. For example, the controller 110 of a refrigerator connected to three imaging devices may have a storage area that is three times the size of one frame capacity. The buffer 112 can be managed separately for each imaging device. For example, the first address may be connected to the first imaging device, the second address may be connected to the second imaging device, and the third address may be connected to the third imaging device.

In addition, when the door is opened or closed, the refrigerator control device 100 can sense images through the three imaging devices and store the images in the buffers classified according to the imaging devices. The refrigerator control device 100 directly accesses the addresses connected to the imaging devices and reads the data stored in the buffer.

The area detecting unit 113 can detect a change area by comparing the current frame and a previous frame ex_frame stored in the buffer. The area detecting unit 113 may calculate a difference value between the pixel value in the current frame and the pixel value in the previous frame and detect a region having a difference value exceeding a predetermined threshold value as a change area. The area detecting unit 113 may calculate the difference between the pixel value of each pixel in the current frame and the pixel value of the corresponding pixel in the previous frame as a difference value. Here, the pixel value may be a value corresponding to an average of color values of each pixel. The color values may be R, G, B values. Here, the previous frame is a frame stored in the buffer, and may be a frame photographed in a state where nothing is loaded in the refrigerator. Further, the previous frame may be a frame photographed at an arbitrary point in time prior to the present point in time.

The half image generating unit 114 can generate a half image bw_frame that can be clearly distinguished from the change area. The half image generating unit 114 may generate the half image based on the pixel value of the corresponding pixel between the current frame and the previous frame. The half image generating unit 114 may generate a half image based on the door opening / closing. For example, when an event occurs in which a door of a refrigerator is opened, a half image can be generated based on difference values of pixel values by comparing with pixel values of a previous frame. The half image generating unit 114 can generate a half image by changing the pixel value of each pixel of the current frame. The pixel value of the pixel included in the change area among the pixels within the current frame may be changed to the first value and the pixel value of the pixel not included in the change area may be changed to the second value. At this time, the first value and the second value may be different values. For example, the first value may be #FFFFFF and the second value may be # 000000. At this time, the half-tone image generating unit 114 may generate a half-tone image based on the difference image obtained by subtracting the pixel value of each pixel of the current frame from the pixel value of the corresponding pixel of the previous frame. When the pixel value of each pixel in the difference image exceeds a predetermined second threshold value, the binary image generating unit 114 sets the pixel value of the corresponding pixel to the first value, and if not, Can be set to the second value.

When obtaining images from a plurality of image capture devices, the refrigerator control device 110 can detect a change area using the obtained current frames and previous frames for the corresponding image capture device, and perform a process of generating a half image have.

The box extracting unit 115 may extract a bounding box including a pixel set to a first value included in the binary image. The bounding box has the shape of a rectangle, and the identification value corresponding to the bounding box includes the coordinates of the starting point and the lengths (x, y, width, height) of the box. The box extracting unit 115 may first extract a two-dimensional bounding box from the two-dimensional image. The box extracting unit 115 can convert the three-dimensional bounding box into a three-dimensional box using the corresponding current frames and two images. The box extracting unit 115 may generate a three-dimensional bounding box using the half images obtained based on the current frames photographed at the same point from different angles. The identification value of the two-dimensional bounding box can be created on the basis of the x-y, y-z, and z-x planes according to the position where the current frame as the source is obtained. For example, the identification value of the bounding box obtained on the left side is set based on the y-z plane, and the identification value of the bounding box obtained on the back side can be set based on the x-y plane.

Suppose that the first to third half images are acquired for the first point. The first to third half images can be obtained from frames photographed at different angles. The box extracting unit 115 may utilize at least three two-dimensional rounding boxes to obtain a three-dimensional rounding box. The box extracting unit 115 generates a three-dimensional coordinate system, and arranges each two-dimensional rounding box along the x-axis, the y-axis, and the z-axis of the coordinate system. Based on the identification values of the bounding boxes of the respective half images, the coordinates (the coordinates with the smallest z component coordinates) protruding to the foreground are selected to be the front coordinates. Based on the identification values of the bounding boxes of the respective half images, the coordinates at the rear most (coordinates with the largest z component coordinates) are determined as the rear coordinates. Based on the identification values of the bounding boxes of the respective half images, the uppermost coordinate is used as the upper coordinate (the coordinate with the largest y component coordinate). Based on the identification values of the bounding boxes of the respective half images, the coordinates at the lowermost position (coordinates with the smallest y component coordinates) are used as the lower coordinates. Based on the identification values of the bounding boxes of the respective half images, the leftmost coordinate (the coordinate with the largest x component coordinate) is used as the left coordinate. Based on the identification values of the bounding boxes of the respective half images, the coordinates located at the rightmost position (coordinates with the smallest x component coordinates) are used as the right side coordinates. The rectangular parallelepiped having six coordinate points, which are the front coordinate, the rear coordinate, the bottom coordinate, the bottom coordinate, the sitting coordinate, and the rear coordinate extracted through the above process, can be extracted as a three-dimensional bounding box.

If there is an inconsistency region between two-dimensional bounding boxes, that is, a point extracted by the bounding box through the first dichromatic image may not be included in the bounding box from the second dichromatic image. In the embodiment of the present invention, extracting the three-dimensional bounding box is for the case where a mismatch area occurs. The box extracting unit 115 can extract a three-dimensional bounding box by extracting six coordinate points satisfying the corresponding condition from the identification values of the bounding box extracted from the three half images. According to embodiments of the present invention, a more accurate change area can be detected by extracting a three-dimensional bounding box.

The bounding box extracted through the box extracting unit 115, especially the three-dimensional bounding box, may be stored in the storage medium 118. The storage medium 118 may store the bounding box in correspondence with the objects included in the loading list.

The determination unit 116 can determine whether the state change of the extracted bounding box is stacked or not by comparing the bounding box extracted through the box extracting unit 115 with the bounding box of the stored storage item have. The determination unit 116 may repeatedly compare the extracted bounding box with the bounding box of all the load items because the determination unit 116 may be a change detected due to movement of the entity. According to the embodiments of the present invention, in connection with the movement of the object, the determination unit 116 may sense the load of the object and the loading of the object.

At this time, the determination unit 116 may compare the identification values (reference coordinates, width, height, height) of the extracted bounding box with the identification values of the bounding box of the loading item to calculate the non-union diagram. The determination unit 116 determines that the state change of the bounding box is lower than the predetermined threshold, and if the non-preferred state exceeds the third threshold, the state change of the bounding box is the load Can be judged.

If it is determined in a lower way through the determination unit 116, the processing unit 117 deletes data on the object included in the bounding box from the storage medium 118. The processing unit 117 can delete the corresponding object from the object list of the refrigerator because the object to be unloaded is no longer present in the refrigerator. The data related to the unloaded object, the 3D bounding box of the unloaded object, the loaded date and time of the unloaded object, and the like are deleted from the object list.

If it is determined through loading the determination unit 116, the processing unit 117 may determine the object included in the bounding box using a pre-trained database. If the decision fails, that is, the entity can not be identified through a pre-trained database, the entity can be determined to be a generic name. Here, the pre-trained database is used to search for an appropriate name for the object based on the boundary, color, etc. of the object. The processing unit 117 may use a form of an entity to determine a name group, a set and the like related to the entity based on the color of the entity, and to confirm the correct name in the name group and the set. The processing unit 117 may analyze the current frame included in the bounding box including the entity and determine the name of the corresponding entity. When the name of the loaded object is determined, the processing unit 117 may calculate and provide the load date and time of recommended use, etc. of the object through the display unit of the user's terminal or the refrigerator. At this time, the recommended use limit date is determined on the basis of the standard shelf life and the expiration date of the individual, but the freshness of the individual can be further determined. The recommended restricting days vary by individual, and can vary depending on the particular condition of the individual (freshness, size, etc.). For example, in the case of unripe fruit, the recommended shelf-life may be set to be longer than the recommended shelf-life of the ripe fruit. For seeds, the recommended use limit for vegetables is set to be shorter than the recommended use limit for fresh vegetables. Recommended use limit days are tabulated by individual and can be tabulated by the state of the entity.

The freshness of an entity may be determined using the current frame of the bounding box. The freshness of the object can be determined using the shape or color of the object in the current frame. For example, if the shape of an object or the boundary of an object is not smooth and uneven, the freshness of the object is judged to be low. In such a case, the recommended use restriction date of the object may be determined to be earlier than the determined recommended use restriction date. The standard expiration date of the object, and the expiration date as long as it is proportional to the expiration date. Since the smoothness of the boundary of the object represents the state of the moisture of the object, it can be used as a criterion for calculating the freshness of the object.

The processing unit 117 may delete the unloaded object from the object list of the refrigerator and add the loaded object to the object list of the refrigerator. When the recommended use restriction date of one or more objects included in the object list comes or is imminent, a notification message prompting the use of the object can be generated and transmitted to the user's terminal or the refrigerator. A notification message that prompts use may be provided with the recipe utilizing the food.

The controller and the control method of the refrigerator according to the embodiments of the present invention can determine the name of the entity using the pre-trained database. To identify a loaded or unloaded entity, the change area containing the entity may be extracted as described above, but is not limited thereto. The controller and the control method of the refrigerator according to the embodiments of the present invention detect a user's motion and detect a change area including an object. A moving object is extracted using the three imaging devices installed in the refrigerator, a change area for a moving object is detected, and a name of the object included in the change area is determined using a pre-trained algorithm.

2 to 6 are flowcharts of a method of controlling a refrigerator according to embodiments of the present invention.

Referring to FIG. 2, the method for controlling a refrigerator includes a step S100 of obtaining a current frame, a step S200 of detecting a change area, a step S300 of generating a half image, a step S400 of creating a box area, , And determining a state change (S500).

In S100, the controller of the refrigerator can acquire the current frame from one or more electrically connected imaging devices.

In S200, the controller of the refrigerator can detect the change area by comparing the current frame and the previous frame stored in the buffer. The controller of the refrigerator can calculate the difference value between the pixel value of each pixel in the current frame and the pixel value of the corresponding pixel in the previous frame. The controller of the refrigerator can detect the region having the difference value exceeding the first threshold value as the change region. Here, the pixel value may be a value corresponding to an average of color values of each pixel. The color values may be R, G, B values. Here, the previous frame is a frame stored in the buffer, and is a frame photographed at an arbitrary point in time prior to the present point in time. The previous frame may be a frame photographed at the time when nothing is loaded in the refrigerator. In addition, the previous frame may be a frame photographed at a time when a load or load is determined.

In S300, the controller of the refrigerator can generate a half image by changing the pixel value of each pixel of the current frame. The pixel value of the pixel included in the change area among the pixels within the current frame may be changed to the first value and the pixel value of the pixel not included in the change area may be changed to the second value. At this time, the first value and the second value may be different values. For example, the first value may be #FFFFFF and the second value may be # 000000. At this time, the controller of the refrigerator can generate the half image based on the difference image obtained by subtracting the pixel value of each pixel of the current frame and the pixel value of the corresponding pixel of the previous frame. The controller of the refrigerator sets the pixel value of the corresponding pixel to the first value when the pixel value of each pixel in the difference image exceeds the predetermined second threshold value and otherwise sets the pixel value of the corresponding pixel to the second value Value.

In S400, the controller of the refrigerator may extract a bounding box including pixels set to a first value included in the binary image. The bounding box has the shape of a rectangle, and the identification value corresponding to the bounding box includes the coordinates of the starting point and the lengths (x, y, width, height) of the box. The controller of the refrigerator can first extract a two-dimensional bounding box from the half-image. The controller of the refrigerator may convert the three-dimensional bounding box using corresponding current frames, half images. The controller of the refrigerator can generate a three-dimensional bounding box using the half images obtained based on the current frames photographed from the same point at different angles. The process of creating the three-dimensional bounding box is the same as the description of FIG.

In S500, the controller of the refrigerator compares the bounding box extracted through the box extracting unit 115 with the bounding box of the stored storage item to determine whether the state change of the extracted bounding box is stacked or not . At this time, the controller of the refrigerator can calculate the non-guide figure by comparing the identification values (reference coordinates, width, height, height) of the extracted bounding box with the identification values of the bounding box of the load item. The controller of the refrigerator determines that the state change of the bounding box is lower when the non-inference degree is lower than the predetermined third threshold value, and when the non-inference degree exceeds the third threshold value, .

Next, the step of detecting the change area will be described in detail. Referring to FIG. 3, the step of detecting a change area S200 includes calculating (S210) a difference value between pixel values of corresponding pixels of a current frame and a previous frame, comparing the difference value of the corresponding pixels with a predetermined first threshold value (S220), and detecting a change area including the pixel (S230).

In S210, the refrigerator control device can detect the change area by comparing the current frame and the previous frame stored in the buffer. The refrigerator control device can calculate the difference value between the pixel value in the current frame and the pixel value in the previous frame. The refrigerator control device can calculate the difference value between the pixel value of each pixel in the current frame and the pixel value of the corresponding pixel of the previous frame. In S220, the refrigerator control device may determine whether the difference value of each pixel exceeds a first threshold value. In S230, the refrigerator control device may detect a change area including a pixel whose difference value exceeds a first threshold value.

Next, the step of determining the state change (S500) will be described in detail. Referring to FIG. 4, the step S500 of determining a state change includes a step S510 of calculating an unillustrated figure, a step S520 of determining whether the unillustrated figure is less than a third threshold value, (S530), and processing (S540) by loading.

In S510, the refrigerator control unit may calculate the non-union diagram by comparing the bounding box extracted through the box extracting unit with the bounding box of the stored or extracted loading item. The refrigerator control device can calculate the non-derivation by comparing the identification values (reference coordinates, width, height, height) of the extracted bounding box with the identification values of the bounding box of the loading item. For example, the difference value between the identification value of the extracted bounding box and the identification value of the bounding box of the load item can be calculated, and the non-inference degree can be calculated in proportion to the difference value. The convergence of 0 to the non - inference is similar between the bounding boxes.

In step S520, the refrigerator control device determines whether the relative angle between the bounding boxes is less than the third threshold value. In other words, it can be said that the non-inference is low, and below the third threshold is similar between bounding boxes. In step S530, if the relative angle between the bounding boxes is less than the third threshold value, the refrigerator control device can determine the state of the bounding box as a lower bound. In S540, when the relative angle between the bounding boxes is equal to or greater than the third threshold value, the refrigerator control device can determine the state of the bounding box to be loaded. In S530 and S540, a further process of confirming the load or loading can be additionally performed. The refrigerator control apparatus according to the embodiments of the present invention controls the refrigerator control device according to embodiments of the present invention based on the average of the difference value obtained by dividing the sum of the average values of the differences of the color values between the current frame and the corresponding pixels in the previous frame, It can be judged whether or not it is loaded. If the difference value obtained by dividing the sum of the average values of the differences of the color values between the current frame and the corresponding pixel in the previous frame by the number of pixels is less than the predetermined fourth threshold value, the state change is determined to be lower; otherwise, . ≪ / RTI >

FIGS. 5 and 6 are diagrams for explaining a processing procedure when a state change is detected.

In S610, when the state change of the load is detected, the control device of the refrigerator can delete the unloaded object from the object list. In addition, a bounding box for a dropped object stored in the storage medium 118 can be searched and deleted.

In S620, the controller of the refrigerator can provide a message about the unloaded object. A notification message such as 'OO has been received' may be output through the output unit of the user terminal or the refrigerator. The notification message may include an icon that can be modified if it is determined that the unloading event has been detected incorrectly. When an input to which an icon that can be modified is selected is received, it is possible to control so that the unloaded object is not deleted from the object list and the bounding box of the corresponding object is also not deleted.

As shown in FIG. 6, in step S710, when the state change of the load is detected, the controller of the refrigerator can retrieve the loaded object using a pre-trained database. Color, and size of the object based on the current frame and the bounding box, and retrieve the name of the object according to the shape, color, and size of the object. Since the form of the food can be modified in various ways, the control device of the refrigerator can primarily determine a group or group of names including individuals based on color. In order to determine the exact name within the name group, set, the type, size, etc. of the entity may be utilized.

In S720, the control device of the refrigerator can use the database to determine the recommended use restriction date of the object based on the name of the retrieved object. At this time, the recommended use limit date can be adjusted not only by the name of the individual but also by the state (freshness, ripe, etc.) of the individual. The controller of the refrigerator can judge the current time point as the load date and time of the object. The recommended use limit date can be calculated based on the date and time of loading.

In S730, the control device of the refrigerator may generate a message including the name of the entity, the loading date, and the recommended use restriction date. The name of the entity, the date and time of loading, and the recommended use restriction date can be stored in the object list. In addition, the bounding box for an entity may be associated with a location in the object list and stored in a storage medium.

According to the present embodiment, the message may include information on the degree of corruption and degree of corruption of each entity included in the load list. That is, it can be determined using the recommended use day of each entity. A user who does not want to be notified can stop receiving the message by selecting the stop notification icon at the bottom of the message.

The controller of the refrigerator according to the present embodiment can generate and provide a loading list according to a request received from a user terminal connected to the network. The user will be able to identify the foods in the refrigerator while in an external location.

7 is a diagram illustrating an example of a refrigerator 100 including a refrigerator control apparatus according to embodiments of the present invention.

The controller 110 of the refrigerator may be electrically connected to the sensor unit 120, the image sensing device 130, the input unit 140, and the output unit 150. At least one of the sensor unit 120, the image capturing apparatus 130, the input unit 140, and the output unit 150 may be included in the refrigerator or may be electrically connected to the refrigerator.

The sensor unit 120 senses whether the door of the refrigerator is open or closed or whether the object is close to the door. When it is detected by the sensor unit 200 whether the door of the refrigerator is open or closed, the controller of the refrigerator can perform a process of determining a change in state occurring in the refrigerator. When the sensor unit 120 senses a user approaching the refrigerator, the controller of the refrigerator can control the state of the refrigerator, that is, the loading list to be outputted through the output unit 150. [

The image pickup device 130 is mounted inside the refrigerator and performs shooting of an internal image. Camera, but the embodiment is not limited thereto. The imaging device 130 is mounted at a plurality of points, i.e., left, right, and rear, in order to check changes in the state and state of the inside of the refrigerator. The current frame obtained from each imaging device 130 can be managed for each imaging device. The imaging device 130 can photograph each of the divided areas of the refrigerator. A plurality of divided zones included in the refrigerator

The input unit 140 may include a key pad, a dome switch, a touch pad (a contact type capacitance type, a pressure resistive type, an infrared detection type, a surface ultrasonic wave An electromagnetism measuring method, an electromagnetism measuring method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like.

The output unit 150 may be a liquid crystal display, a thin film transistor-liquid crystal display (LCD), or the like, as a means for providing a message related to the status change of the refrigerator and the loading list, ), An organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

8 is a view for explaining a refrigerator control system according to embodiments of the present invention.

The refrigerator control system according to embodiments of the present invention may include a refrigerator 100 including a refrigerator control device, a user terminal 200, and a communication network 300.

The refrigerator (100) including the refrigerator control device

The user terminal 200 may perform a function of providing a load list or status message received through the communication network 300. [ Messages may be provided via the output included in the user terminal 200. [ Means a communication terminal that can use a web service in a wired / wireless communication environment. Here, the user terminal 200 may be a user's personal computer or a user's portable terminal.

In more detail, the user terminal 200 may be a computer (e.g., a desktop, a laptop, a tablet, etc.), a media computing platform (e.g., a cable, a satellite set top box, a digital video recorder), a handheld computing device E. G., A PDA, an email client, etc.), any form of cellular telephone, or any other type of computing or communication platform, but the invention is not so limited.

Meanwhile, the communication network 300 connects the user terminal 200 and the refrigerator 100. That is, the communication network 300 refers to a communication network that provides a connection path so that the user terminal 200 can connect to the refrigerator control device and then send and receive data. The communication network 300 may be a wired network such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, But the scope of the present invention is not limited thereto.

The embodiments of the present invention described above can be embodied in the form of a computer program that can be executed on various components on a computer, and the computer program can be recorded on a computer-readable medium. At this time, the medium may be a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floptical disk, , A RAM, a flash memory, and the like, which are specifically configured to store and execute program instructions.

Meanwhile, the computer program may be designed and configured specifically for the present invention or may be known and used by those skilled in the computer software field. Examples of computer programs may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like.

The specific acts described in the present invention are, by way of example, not intended to limit the scope of the invention in any way. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of such systems may be omitted. Also, the connections or connecting members of the lines between the components shown in the figures are illustrative of functional connections and / or physical or circuit connections, which may be replaced or additionally provided by a variety of functional connections, physical Connection, or circuit connections. Also, unless explicitly mentioned, such as " essential ", " importantly ", etc., it may not be a necessary component for application of the present invention.

The use of the terms " above " and similar indication words in the specification of the present invention (particularly in the claims) may refer to both singular and plural. In addition, in the present invention, when a range is described, it includes the invention to which the individual values belonging to the above range are applied (unless there is contradiction thereto), and each individual value constituting the above range is described in the detailed description of the invention The same. Finally, the steps may be performed in any suitable order, unless explicitly stated or contrary to the description of the steps constituting the method according to the invention. The present invention is not necessarily limited to the order of description of the above steps. The use of all examples or exemplary language (e.g., etc.) in this invention is for the purpose of describing the present invention only in detail and is not to be limited by the scope of the claims, It is not. It will also be appreciated by those skilled in the art that various modifications, combinations, and alterations may be made depending on design criteria and factors within the scope of the appended claims or equivalents thereof.

Claims (1)

An image obtaining unit for receiving a current frame of the internal state of the refrigerator;
An area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame;
A half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area;
A box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image;
And a determination unit for determining a state change of the refrigerator by comparing the bounding box extracted through the box extracting unit with at least one pre-stored bounding box.

Images