KR20190004162A - Method for controlling refrigerator and refrigerator controller - Google Patents

Abstract

An embodiment discloses a refrigerator controller. The refrigerator controller includes an image obtaining unit for receiving a current frame of the internal state of the refrigerator; an area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame; a half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; a box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image; a determining unit for determining a state change of the refrigerator by comparing the bounding box extracted through the box extracting unit with at least one pre-stored bounding box; and a processing unit for generating images having sizes that are visually recognizably adjusted, while corresponding to the bounding box, when a state change is judged to be a load through the determining unit, generating a map in which the generated images are arranged in the first order, comparing object database constructed with the bounding box and images corresponding to the bounding box as a reference and the map, and determining a name of the loaded entity.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a refrigerator control method, a refrigerator control method, and a computer program for executing the method.

An embodiment of the present invention relates to a refrigerator control method, a refrigerator control apparatus, and a computer program for executing the method.

2. Description of the Related Art Generally, a refrigerator is a device for supplying cold air generated by driving a refrigeration cycle, and is a device for storing food in a low temperature state. In the case of a conventional refrigerator, only the function of storing food in a low temperature state can be performed. However, in recent years, there is an increasing need for additional functions in addition to the food storage function.

The refrigerator is a device for storing and storing a certain object. In order to check the inside of the refrigerator, the refrigerator door must be opened and checked. In addition, when the user tries to buy goods from markets or marts, he or she can not buy the same food or buy necessary food if the amount and type of the food stored in the refrigerator can not be grasped.

Japanese Patent Publication No. 3450907 and Japanese Patent Laid-Open No. 2004-183987 disclose techniques for photographing the inside of a refrigerator by installing a camera on a door. Further, according to Japanese Patent Application Laid-Open No. 2001-294308, a technique is disclosed in which a camera is installed in a room, a drawer, and a door, respectively.

An object of the present invention is to obtain a current frame photographed inside a refrigerator and to determine a state change occurring in the refrigerator through comparison between previous frames photographed previously.

It is also an object of the present invention to extract a bounding box from a current frame based on a change area in which a change occurs and to identify an object included in the bounding box using a pre-trained database.

It is also an object of the present invention to determine whether a state change is a downward trend or a load based on a difference between pixel values of a corresponding pixel between a current frame and a previous frame.

An apparatus for controlling a refrigerator according to embodiments of the present invention includes: an image obtaining unit that receives a current frame of an internal situation of a refrigerator; An area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame; A half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; A box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image; A determining unit for determining a state change of the refrigerator by comparing the bounding box extracted through the box extracting unit with at least one pre-stored bounding box; When the state change is judged to be load through the determination unit, generates images in which the sizes are visually recognized, corresponding to the bounding box, generates a map in which the generated images are arranged in the first order, And a processing unit for determining the name of the loaded entity by comparing the object database constructed with the images corresponding to the bounding box and the bounding box received, and the map.

Wherein the determination unit compares the extracted identification value of the bounding box and the identification values of the previously stored bounding boxes to calculate a non-unidirectional figure, and determines that the state change is a load if the unidirectional angle is greater than or equal to a predetermined first threshold value, If the non-inference degree is less than the first threshold value, it is possible to determine the change in status as a decrease.

The area detecting unit may calculate a difference value which is a difference between pixel values of a corresponding pixel of a previous frame stored in a current frame and a buffer, and may set a pixel having a difference value exceeding a predetermined second threshold value as a change area.

The binary image generating unit may generate a binary image in which a pixel value of a pixel included in the change area is changed to a first value and a pixel value of a pixel not included in the change area is changed to a second value.

The judging unit may determine the name of the loaded entity according to the type and color of the entity included in the bounding box using the pre-trained database when the state change is judged to be loaded through the judging unit.

After determining the name of the entity, the determination unit may generate a notification message including the name of the loaded entity and the loading date and time through the determination unit, and provide the generated notification message to the user terminal.

A method of controlling a refrigerator according to embodiments of the present invention includes: receiving a current frame in which a controller captures an internal state of a refrigerator; Detecting the change area by the controller calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value for a corresponding pixel of the previous frame; The controller generating a dichromatic image that changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; The control device extracting a bounding box containing a pixel set to a first value contained in the binary image; And comparing the extracted bounding box with one or more pre-stored bounding boxes to determine a state change of the refrigerator.

Wherein the step of determining the state change comprises: comparing the identification value of the extracted bounding box and the identification values of the previously stored bounding boxes to calculate a non-unidirectional diagram; if the non-inductive diagram is greater than or equal to a predetermined first threshold value, And if the non-inference degree is less than the first threshold value, it is possible to determine the change in state as a decrease.

Wherein the step of detecting the change area comprises: calculating a difference value, which is a difference between pixel values of a current frame and a corresponding pixel of a previous frame stored in the buffer; and calculating a pixel having a difference value exceeding a predetermined second threshold value as a change area Can be set.

The generating of the binary image may generate a binary image in which a pixel value of a pixel included in the change area is changed to a first value and a pixel value of a pixel not included in the change area is changed to a second value.

The step of determining the state change may determine the type of the entity included in the bounding box and the name of the loaded entity based on the color using the pre-trained database when the state change is determined to be the load.

After determining the name of the entity, the determining step may generate a notification message including the name of the entity loaded through the determination unit and the loading date and time, and provide the notification message to the user terminal.

An apparatus for controlling a refrigerator according to embodiments of the present invention includes: an image obtaining unit that receives a current frame of an internal situation of a refrigerator; An area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame; A half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; A box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image; A determining unit for determining a state change of the refrigerator by comparing the bounding box extracted through the box extracting unit with at least one pre-stored bounding box; Wherein when the state change is judged to be a load through the determination unit, the control unit generates at least one image by adjusting the size of the bounding box to correspond to the bounding box, forms a map in which the generated images are arranged in a first order, And a processor for inferring the name of the loaded entity by comparing the map with the object database constructed with the images.

Wherein the processing unit converts the images included in the map using a predetermined algorithm to generate a transformation map and extracts a second image having a size corresponding to the first image from the first image and the object data of the transformation map By comparison, it can be characterized that the name of the entity is determined.

After determining the name of the entity, the determination unit may generate a notification message including the name of the loaded entity and the loading date and time through the determination unit, and provide the generated notification message to the user terminal.

A method of controlling a refrigerator according to embodiments of the present invention includes: receiving a current frame in which a controller captures an internal state of a refrigerator; Detecting the change area by the controller calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value for a corresponding pixel of the previous frame; The controller generating a dichromatic image that changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; The control device extracting a bounding box containing a pixel set to a first value contained in the binary image; Comparing the extracted bounding box with one or more pre-stored bounding boxes to determine a state change of the refrigerator; Wherein when the state change is determined to be a load, the size of the bounding box is adjusted to generate at least one image corresponding to the bounding box, a map in which the generated images are arranged in a first order, And deducing the name of the loaded entity by comparing the map with the object database constructed from the received images.

Wherein the step of inferring the name of the entity comprises the steps of generating a transformation map by transforming the images included in the map using a preset algorithm, And comparing the first image having the first image with the second image having the second image.

In the refrigerator control method according to embodiments of the present invention, when a plurality of names of the entities are obtained, the controller compares the maps and the sub images of the corresponding images extracted from the entity database, And determining the name of the sub-image as the name of the sub-image.

The method of controlling a refrigerator according to embodiments of the present invention may include generating a notification message including a name of a loaded entity and a loading date and time through the determination unit and providing the notification message to a user terminal after determining a name of the entity, .

A computer program according to an embodiment of the present invention may be stored in a medium using a computer to execute any one of the control methods of the refrigerator according to the embodiment of the present invention.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium for recording a computer program for executing the method are further provided.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and the detailed description of the invention.

According to the present invention, it is possible to acquire a current frame photographed inside a refrigerator, and to determine a state change occurring in the refrigerator through comparison between previously photographed previous frames.

Further, according to the present invention, a bounding box can be extracted from a current frame based on a change area in which a change has occurred, and an object included in the bounding box can be specified using a pre-trained database.

Further, the present invention can determine whether the state change is a downward change or a load based on a difference between pixel values of a corresponding pixel between a current frame and a previous frame.

1 is a view showing a refrigerator control apparatus according to an embodiment of the present invention.
2 is a view showing a structure of a processing section.
3 to 8 are flowcharts of a method of controlling a refrigerator according to embodiments of the present invention.
9 is a view illustrating an example of a refrigerator including a refrigerator control apparatus according to embodiments of the present invention.
10 is a view for explaining a refrigerator control system according to embodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention and methods of achieving them will be apparent with reference to the embodiments described in detail below with reference to the drawings. However, the present invention is not limited to the embodiments described below, but may be implemented in various forms.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout the drawings, and a duplicate description thereof will be omitted .

In the following embodiments, the terms first, second, and the like are used for the purpose of distinguishing one element from another element, not the limitative meaning.

In the following examples, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise.

In the following embodiments, terms such as inclusive or possessed mean that a feature or element described in the specification is present, and does not exclude the possibility that one or more other features or components are added in advance.

If certain embodiments are otherwise feasible, the particular process sequence may be performed differently from the sequence described. For example, two processes that are described in succession may be performed substantially concurrently, and may be performed in the reverse order of the order described.

In the following embodiments, the term "circuit" refers to any circuitry, circuitry, and / or circuitry, including, for example, hardwired circuitry, programmable circuitry, state machine circuitry, and / or firmware that stores instructions executed by a programmable circuit, either alone or in any combination . The application may be implemented as code or instructions that may be executed on a programmable circuit, such as a host processor or other programmable circuit. A module, as used in any of the embodiments herein, may be implemented as a circuit. The circuitry may be implemented as an integrated circuit, such as an integrated circuit chip.

In the following embodiments, when a component is referred to as "comprising ", it means that it can include other components as well, without excluding other components unless specifically stated otherwise. Also, the terms " part, "" module," and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

1 is a view illustrating a refrigerator control apparatus 110 according to an embodiment of the present invention.

1, a refrigerator control apparatus 110 according to an embodiment of the present invention includes an image obtaining unit 111, a buffer 112, an area detecting unit 113, an image generating unit 114, a box extracting unit 115 A determination unit 116, and a processing unit 117. [0033] FIG.

The refrigerator control apparatus 110 according to embodiments of the present invention can acquire images from one or more electrically connected imaging apparatuses. At this time, the refrigerator control device 110 may control to acquire an image from the imaging device when a predetermined signal is sensed from an electrically connected sensor. For example, when the opening and closing of the door is detected, the refrigerator control device 110 can acquire an image from the imaging device. The refrigerator control device 110 acquires an image periodically and senses a state change only when an opening and closing of the door is detected rather than detecting a state change. The refrigerator control device 110 can acquire images from three or more imaging devices. The three imaging devices can shoot the same point at an angle of 90 degrees each. The imaging devices take images of planes that are not coplanar but orthogonal to each other.

The image acquiring unit 111 may acquire a current frame from one or more electrically connected devices. The acquired current frame may be passed to the buffer 112.

The buffer 112 may record the obtained frame, image. When a new frame is received, the buffer 112 may delete the previously stored frame and store the new frame. For example, the buffer 112 may delete the pre-stored frame and store a new frame only when it is determined that the internal state of the refrigerator is changed. The buffer 112 may have as many storage areas as the number of connected imaging devices. For example, the controller 110 of a refrigerator connected to three imaging devices may have a storage area that is three times the size of one frame capacity. The buffer 112 can be managed separately for each imaging device. For example, the first address may be connected to the first imaging device, the second address may be connected to the second imaging device, and the third address may be connected to the third imaging device.

The area detecting unit 113 can detect a change area by comparing the current frame and a previous frame ex_frame stored in the buffer. The area detecting unit 113 may calculate a difference value between the pixel value in the current frame and the pixel value in the previous frame and detect a region having a difference value exceeding a predetermined threshold value as a change area. The area detecting unit 113 may calculate the difference between the pixel value of each pixel in the current frame and the pixel value of the corresponding pixel in the previous frame as a difference value. Here, the pixel value may be a value corresponding to an average of color values of each pixel. The color values may be R, G, B values. Here, the previous frame is a frame stored in the buffer, and may be a frame photographed in a state where nothing is loaded in the refrigerator. Further, the previous frame may be a frame photographed at an arbitrary point in time prior to the present point in time.

The half image generating unit 114 can generate a half image bw_frame that can be clearly distinguished from the change area. The half image generating unit 114 can generate a half image by changing the pixel value of each pixel of the current frame. The pixel value of the pixel included in the change area among the pixels within the current frame may be changed to the first value and the pixel value of the pixel not included in the change area may be changed to the second value. At this time, the first value and the second value may be different values. For example, the first value may be #FFFFFF and the second value may be # 000000. At this time, the half-tone image generating unit 114 may generate a half-tone image based on the difference image obtained by subtracting the pixel value of each pixel of the current frame from the pixel value of the corresponding pixel of the previous frame. When the pixel value of each pixel in the difference image exceeds a predetermined second threshold value, the binary image generating unit 114 sets the pixel value of the corresponding pixel to the first value, and if not, Can be set to the second value.

In an alternative embodiment, the half-image generating unit 114 may further perform a process of removing the noise of the half-image. A rectangle corresponding to one or more regions set to a second value in the two images can be extracted. At this time, the rectangle may be rotated at a predetermined angle with respect to the horizontal axis or the vertical axis of the image. The half-image generating unit 114 searches for and removes one or more rectangles whose lengths are shorter than the critical length of one of the one or more rectangles, and includes remaining rectangles, that is, rectangles whose length and width are longer than the critical length And converts the outline of the object into a block polygon (convex) corresponding to the object based on the outline of the object. Finally, the half-image generating unit 114 can generate a half-image complemented by filling the object, that is, the inside of the convex polygon, with white.

When obtaining images from a plurality of image capture devices, the refrigerator control device 110 can detect a change area using the obtained current frames and previous frames for the corresponding image capture device, and perform a process of generating a half image have.

The box extracting unit 115 may extract a bounding box including a pixel set to a first value included in the binary image. The bounding box has the shape of a rectangle, and the identification value corresponding to the bounding box includes the coordinates of the starting point and the lengths (x, y, width, height) of the box. The box extracting unit 115 may first extract a two-dimensional bounding box from the two-dimensional image. The box extracting unit 115 can convert the three-dimensional bounding box into a three-dimensional box using the corresponding current frames and two images. The box extracting unit 115 may generate a three-dimensional bounding box using the half images obtained based on the current frames photographed at the same point from different angles. The identification value of the two-dimensional bounding box can be created on the basis of the x-y, y-z, and z-x planes according to the position where the current frame as the source is obtained. For example, the identification value of the bounding box obtained on the left side is set based on the y-z plane, and the identification value of the bounding box obtained on the back side can be set based on the x-y plane.

Suppose that the first to third half images are acquired for the first point. The first to third half images can be obtained from frames photographed at different angles. The box extracting unit 115 may utilize at least three two-dimensional rounding boxes to obtain a three-dimensional rounding box. The box extracting unit 115 generates a three-dimensional coordinate system, and arranges each two-dimensional rounding box along the x-axis, the y-axis, and the z-axis of the coordinate system. In setting three axes, an arbitrary direction of the space is set as an x-axis, a y-axis in an orthogonal relationship on the same plane as the x-axis is set, and an orthogonal relationship Axis can be set. A coordinate value (the coordinate with the smallest z component coordinate) protruding most frontward is selected based on the identification values of the bounding boxes of the respective half images, and a plane having the front coordinate value as the z component is determined as the first plane. The coordinate value (the coordinate with the largest z component coordinate) located at the rear side is determined based on the identification values of the bounding boxes of the respective half images, and the plane having the rear coordinate value as the z component is determined as the second plane. Based on the identification values of the bounding boxes of the respective half images, the uppermost coordinate value is used as the upper coordinate value (the coordinate with the largest y component coordinate). And the plane having the upper coordinate value as the y component is determined as the third plane. Based on the identification values of the bounding boxes of the respective half images, the lowest coordinate value (coordinate with the smallest y component coordinate) is used as the lower coordinate value. And the plane having the y coordinate of the lower coordinate value is determined as the fourth plane. Based on the identification values of the bounding boxes of the respective half images, the leftmost coordinate value (the coordinate with the largest x component coordinate) is used as the sitting surface coordinate value. And the plane having the x-component of the seat surface coordinate value is determined as the fifth plane. Based on the identification values of the bounding boxes of the respective half images, the coordinate value located at the rightmost position (the coordinate with the smallest x component coordinate) is used as the right side coordinate value. And the plane having the x coordinate of the right side coordinate value is determined as the sixth plane. The rectangular parallelepiped made up of the first to sixth planes extracted through the above process can be extracted into a three-dimensional bounding box.

If there is an inconsistency region between two-dimensional bounding boxes, that is, a point extracted by the bounding box through the first dichromatic image may not be included in the bounding box from the second dichromatic image. In the embodiment of the present invention, extracting the three-dimensional bounding box is for the case where a mismatch area occurs. The box extracting unit 115 can extract a three-dimensional bounding box by extracting six coordinate points satisfying the corresponding condition from the identification values of the bounding box extracted from the three half images. According to the embodiments of the present invention, it is possible to retrieve the name of the loaded or unloaded object more accurately by extracting the three-dimensional bounding box. The bounding box extracted through the box extracting unit 115, especially the three-dimensional bounding box, may be stored in the storage medium 118. The storage medium 118 may store the bounding box in correspondence with the objects included in the loading list.

The determination unit 116 deduces the state change of the extracted bounding box by comparing the bounding box extracted through the box extracting unit 115 with the bounding box of one or more previously stored load items. A change in the state of the inferred bounding box can determine whether to load or not. The determination unit 116 may repeatedly compare the extracted bounding box with the bounding box of all the load items because the determination unit 116 may be a change detected due to movement of the entity. According to embodiments of the present invention, in connection with the movement of the object, the determination unit 116 may sense the load of the object or the loading of the object.

At this time, the determination unit 116 may compute the non-dissimilarity by comparing the identification values (reference coordinates, width, height, height) of the extracted bounding box with the identification values of the bounding boxes of the stored loading items. When the non-inference degree is less than the predetermined third threshold value, it is determined that the state change of the bounding box is down. If the non-inference state exceeds the third threshold value, the state change of the bounding box may be determined as the load.

If the change of the state of the bounding box is determined to be lower through the determination unit 116, the processing unit 117 deletes the data of the lowered object included in the bounding box on the storage medium 118. Objects that are unloaded from the refrigerator object list can be deleted. The data related to the unloaded object, the 3D bounding box of the unloaded object, the loaded date and time of the unloaded object, and the like are deleted from the object list.

If it is determined through the determination unit 116 that the data is to be loaded, the processing unit 117 may further perform a process of verifying whether or not the data is secondarily loaded. The processing unit 117 may compare the previously stored bounding box with the position corresponding to the bounding box to calculate the difference of the color values of the corresponding pixels for each color. When the average of the differences of the color values (Red, green, and blue values) calculated by the processing unit 117 is equal to or greater than the fourth threshold value, the state change of the bounding box is confirmed as a load. Here, the pre-stored bounding box may be a bounding box included in the photographed image when the corresponding position is empty, from the image photographed at the previous point corresponding to the extracted bounding box. When the average of the difference of the calculated color values is less than the fourth threshold value, the state change of the bounding box is conformed to the lower limit.

The processing unit 117 can determine the object included in the bounding box using a pre-trained database. If the entity fails to be determined, that is, the entity included in the bounding box can not be searched or identified through the pre-trained database, the entity's name can be determined to be a generic name of the entity most similar to the entity. Here, the pre-trained database searches for an appropriate name for the object based on the boundaries of the bounding box containing the object, color, and the like. For example, the processing unit 117 may determine a naming group, set, or the like associated with an entity based on features of the bounding box that includes the entity and identify the exact name within the naming group, In order to do so, the form or characteristic of the entity may be utilized. The entity may be specified only by some features, not by the overall features of the entity. When the name of the loaded object is retrieved, the processing unit 117 can calculate and provide the load date and time of recommended use, etc. of the object through the display unit of the user's terminal or the refrigerator. At this time, the recommended use limit date is determined based on the standard expiration date of the individual, the expiration date, etc., but the freshness or maturity of the individual can be further considered. The recommended restricting days vary from individual to individual, and can vary depending on the specific status of the individual (freshness, maturity, size, etc.). For example, in the case of fruits whose maturity is less than the predetermined baseline maturity, the recommended use limit day may be set to be longer than the recommended use limit day of the reference maturity fruit. In the case of vegetables whose freshness is above the predetermined reference freshness, the recommended use limit date may be set to be shorter than the recommended use limit date of the freshness reference freshness. Recommended Use Limit Days are tabulated by entity and can be tabulated by entity state (freshness or maturity).

The processing unit 117 can determine the name of the entity based on the data stored in the entity database. The processing unit 117 corresponds to a bounding box (hereinafter referred to as an " entity area ") that includes an entity, and can convert the bounding box to another type or format. The processing unit 117 may convert the object region by adopting or removing at least one of color information, boundary information, and brightness information representing the object region. The processing unit 117 may change the type or format of the entity region so as to express at least one of color information, boundary information, and brightness information constituting the entity region. The processing unit 117 may convert the entity region so as to leave only the information necessary for determining the name of the entity. For example, the processing unit 117 may remove the color information of the object and convert the object region so that only the boundary information of the object is expressed. In another embodiment, the processing unit 117 may respectively calculate the difference of pixel values between each pixel and neighboring pixels located adjacent to the periphery of each pixel, and determine the direction of each pixel based on the calculated difference .

Accordingly, the processing unit 117 can simplify various information constituting the object region so that the computer can understand it. According to the embodiment of the present invention, unnecessary information can be removed in the process of simplification or can be changed to different information, respectively.

Here, the object database can be constructed by learning about the received image and the object information included in the image. The present invention can utilize an object database for searching for the name of a food. The object database manages one or more images belonging to the same object individually. The object database can be constructed through the process of repeatedly receiving images and learning objects included in each image. The object database can be designed in consideration of a somewhat large number of objects, for example, more than 10,000 objects, in order to recognize a single object. The object database may include a plurality of images belonging to the 'cat', and data obtained by sequencing the transformed images based on each image and each image individually in a preset reference or order.

The processing unit 117 converts the object region into various sizes, and compares the converted object region with the image stored in the object database having the corresponding size. Specifically, the processing unit 117 may compare an image belonging to the smallest first size among the converted entity regions with an image stored in the object database. As the size of the image of the first size becomes smaller, individual characteristics of the object can be diluted. The processing unit 117 may convert the bounding box based on the size and resolution of the bounding box including the object.

In addition, the processing unit 117 may generate a map in which the recognized entity regions are arranged in a predetermined order to search for the current name. The maps may be generated in magnitude order. The processing unit 117 may compare the map and the object database by size. The processing unit 117 compares the 1-1 second image of the first size arranged in the map and the 1-2 image corresponding to the first size of the object database, The second-1 image and the second-2 image of the second size, which are one step higher than the first size, are compared with each other. The 2-1 image can be called a child image of the 1-1 image. The image 2-2 may be called a child image of the image 1-2. This comparison process repeats itself by continuously decreasing or increasing the size. The comparison process is repeated until the number of names of the images included in the object database corresponding to the received image becomes 1, that is, the singular value.

Among various information (color information, boundary information, brightness information, etc.) included in the image, the processing unit 117 can utilize only a part of the information and search for similar images. For example, the processing unit 117 can search for similar images based only on the boundary information of the image.

The freshness of an entity may be determined using the current frame of the bounding box. The freshness of the object can be determined using the shape or color of the object in the current frame. For example, when the shape of an object, that is, the boundary of the object is not smooth and uneven, the freshness of the object is judged to be low. In such a case, the recommended use limit day of the object may be determined to be earlier than the determined recommended use limit day. The standard expiration date of the object, and the expiration date as long as it is proportional to the expiration date.

The processing unit 117 may delete the unloaded object from the object list of the refrigerator and add the loaded object to the object list of the refrigerator. When the recommended use restriction date of one or more objects included in the object list comes or is imminent, a notification message prompting the use of the object can be generated and transmitted to the user's terminal or the refrigerator. A notification message that prompts use may be provided with the recipe utilizing the food.

According to an embodiment of the present invention, an apparatus for controlling a refrigerator includes: an image obtaining unit that receives a current frame of an internal situation of a refrigerator; An area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame; A half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area; And a box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image. The determining unit may receive a name corresponding to the object and the entity included in the extracted bounding box and compare the bounding box extracted through the box extracting unit with one or more pre-stored bounding boxes to determine a state change of the refrigerator part; A mapping unit configured to map the transformed images from the bounding box in a predetermined order and to compare the object database and the map on a size-by-size basis, And a processing unit for determining the processing unit.

Fig. 2 is a diagram for explaining the operation of the processing unit 117. Fig.

The processing unit 117 may include a converting unit 1171 and a comparing unit 1172. [ The processing unit 117 may perform a function for searching for the name of the received entity. In addition, the processing unit 117 may perform a function of learning an image including the received entity and the entity. The processing unit 117 includes a conversion unit 1171 for converting the format of the bounding box and a comparison unit 1172 for comparing the image stored in the object database with the bounding box containing the object so as to dilute the characteristic of the object . The processing unit 117 generates one or more images corresponding to the bounding box by adjusting the size of the bounding box, and forms a map in which the generated images are arranged in a first order The name of the loaded object can be deduced by comparing the map of the object database constructed with the cumulatively received images and the map.

The converting unit 1171 generates a map in which the converted images are arranged in a predetermined order based on the bounding box including the object. The conversion unit 1171 may further perform a conversion process so that the aspect ratio of the bounding box (entity region) including the entity can be converted to a predetermined ratio. The conversion unit 1171 may downsize the size or resolution of the entity region to a predetermined ratio (e.g., 0.5, 0.7). The conversion unit 1171 converts the image of the object region into a first ratio of a predetermined ratio to the original size of the image, and converts the image of the first region into the first ratio by applying a predetermined ratio to the first ratio. Lt; / RTI > The ratio of adjusting the size or resolution of the image can be calculated using a predetermined ratio.

The converting unit 1171 may extract at least one of color information, boundary information, and brightness information from the individual region. The conversion unit 1171 can generate an image represented by the extracted information. In addition, the converting unit 1171 may extract at least one of color information, boundary information, and brightness information from a plurality of size-adjusted entity regions. At this time, the images composed of the extracted information can be generated as transformation maps arranged in order of magnitude.

The conversion unit 1171 converts the image so that the characteristics of the object contained in the images included in the map are revealed. The conversion unit 1171 may convert the image so as to include the directionality of the boundary line included in the image. A transformation map containing transformed images may be generated.

The comparator 1172 searches the object database D1 for an image (hereinafter, referred to as a first image) similar to the n-th object area of the generated map or transformation map. The comparator 1172 searches for an image (hereinafter referred to as a second image) similar to the n-th object region among the images having the size corresponding to the size of the n-th object region. The comparing unit 1172 determines the name of the searched image as the first name. The comparator 1172 may search similar images based on the relationship values of each pixel of the image. For example, the comparator 1172 may search for similar images based on the difference value of the feature values between each pixel and the surrounding pixels of each pixel. At this time, the comparator 1172 may search the similar image for the image including the difference values arranged similarly to the matrix in which the difference values of the feature values among the pixels included in the first image are arranged.

The name of the entity linked to the searched second image becomes the name of the entity included in the entity region. If the names of objects connected to the second image are plural, the above process is repeated until the name of the searched image becomes one.

For example, if the second image is searched and the names of the entities connected to the second image are plural, the comparing unit 1172 selects, among the one or more fourth images having a size larger than the second image, Search for an image similar to the one located at -1. The search process is repeated until the name of the searched image becomes one. N-1, n-2, ..., and so on until the name of the entity is specified. Similar images are retrieved from the object database based on the image located at the first position. The name of the searched image is determined as the name of the entity. At this time, the sizes of the two images to be compared should be the same. The comparing unit 1172 can increase the size of the compared image.

Finally, when an image corresponding to the object is not found, the comparing unit 1172 may add the map generated from the object regions to the object database D1. At this time, the map or transformation map generated from the object is added to the image corresponding to the object area in the object database D1. The name of the object of the added map can be set by the user.

After the name of the entity is determined, the map generated from the bounding box containing the entity, or the transformation map, may be added to the object database as a corresponding location. The corresponding location is a location determined by the name of the entity.

3 to 7 are flowcharts of a method of controlling a refrigerator according to embodiments of the present invention.

Referring to FIG. 3, a method of controlling a refrigerator includes a step S100 of obtaining a current frame, a step S200 of detecting a change area, a step S300 of generating a half image, a step S400 of creating a box area, , And determining a state change (S500).

In S100, the controller of the refrigerator can acquire the current frame from one or more electrically connected imaging devices.

In S200, the controller of the refrigerator can detect the change area by comparing the current frame and the previous frame stored in the buffer. The controller of the refrigerator can calculate the difference value between the pixel value of each pixel in the current frame and the pixel value of the corresponding pixel in the previous frame. The controller of the refrigerator can detect the region having the difference value exceeding the first threshold value as the change region. Here, the pixel value may be a value corresponding to an average of color values of each pixel. The color values may be R, G, B values. Here, the previous frame is a frame stored in the buffer, and is a frame photographed at an arbitrary point in time prior to the present point in time. The previous frame may be a frame photographed at the time when nothing is loaded in the refrigerator. In addition, the previous frame may be a frame photographed at a time when a load or load is determined.

In S300, the controller of the refrigerator can generate a half image by changing the pixel value of each pixel of the current frame. The pixel value of the pixel included in the change area among the pixels within the current frame may be changed to the first value and the pixel value of the pixel not included in the change area may be changed to the second value. At this time, the first value and the second value may be different values. For example, the first value may be #FFFFFF and the second value may be # 000000. At this time, the controller of the refrigerator can generate the half image based on the difference image obtained by subtracting the pixel value of each pixel of the current frame and the pixel value of the corresponding pixel of the previous frame. The controller of the refrigerator sets the pixel value of the corresponding pixel to the first value when the pixel value of each pixel in the difference image exceeds the predetermined second threshold value and otherwise sets the pixel value of the corresponding pixel to the second value Value.

In S400, the controller of the refrigerator may extract a bounding box including pixels set to a first value included in the binary image. The bounding box has the shape of a rectangle, and the identification value corresponding to the bounding box includes the coordinates of the starting point and the lengths (x, y, width, height) of the box. The controller of the refrigerator can first extract a two-dimensional bounding box from the half-image. The controller of the refrigerator may convert the three-dimensional bounding box using corresponding current frames, half images. The controller of the refrigerator can generate a three-dimensional bounding box using the half images obtained based on the current frames photographed from the same point at different angles. The process of creating the three-dimensional bounding box is the same as the description of FIG.

In S500, the controller of the refrigerator compares the bounding box extracted through the box extracting unit 115 with the bounding box of the stored storage item to determine whether the state change of the extracted bounding box is stacked or not . At this time, the controller of the refrigerator can calculate the non-guide figure by comparing the identification values (reference coordinates, width, height, height) of the extracted bounding box with the identification values of the bounding box of the load item. The controller of the refrigerator determines that the state change of the bounding box is lower when the non-inference degree is lower than the predetermined third threshold value, and when the non-inference degree exceeds the third threshold value, .

Next, the step of detecting the change area will be described in detail. Referring to FIG. 3, the step of detecting a change area S200 includes calculating (S210) a difference value between pixel values of corresponding pixels of a current frame and a previous frame, comparing the difference value of the corresponding pixels with a predetermined first threshold value (S220), and detecting a change area including the pixel (S230).

In S210, the refrigerator control device can detect the change area by comparing the current frame and the previous frame stored in the buffer. The refrigerator control device can calculate the difference value between the pixel value in the current frame and the pixel value in the previous frame. The refrigerator control device can calculate the difference value between the pixel value of each pixel in the current frame and the pixel value of the corresponding pixel of the previous frame. In S220, the refrigerator control device may determine whether the difference value of each pixel exceeds a first threshold value. In S230, the refrigerator control device may detect a change area including a pixel whose difference value exceeds a first threshold value.

Next, the step of determining the state change (S500) will be described in detail. Referring to FIG. 4, the step S500 of determining a state change includes a step S510 of calculating an unillustrated figure, a step S520 of determining whether the unillustrated figure is less than a third threshold value, (S530), and processing (S540) by loading.

In S510, the refrigerator control unit may calculate the non-union diagram by comparing the bounding box extracted through the box extracting unit with the bounding box of the stored or extracted loading item. The refrigerator control device can calculate the non-derivation by comparing the identification values (reference coordinates, width, height, height) of the extracted bounding box with the identification values of the bounding box of the loading item. For example, the difference value between the identification value of the extracted bounding box and the identification value of the bounding box of the load item can be calculated, and the non-inference degree can be calculated in proportion to the difference value. The convergence of 0 to the non - inference is similar between the bounding boxes.

In step S520, the refrigerator control device determines whether the relative angle between the bounding boxes is less than the third threshold value. In other words, it can be said that the non-inference is low, and below the third threshold is similar between bounding boxes. In step S530, if the relative angle between the bounding boxes is less than the third threshold value, the refrigerator control device can determine the state of the bounding box as a lower bound. In S540, when the relative angle between the bounding boxes is equal to or greater than the third threshold value, the refrigerator control device can determine the state of the bounding box to be loaded. In S530 and S540, a further process of confirming the load or loading can be additionally performed. The refrigerator control apparatus according to the embodiments of the present invention controls the refrigerator control device according to embodiments of the present invention based on the average of the difference value obtained by dividing the sum of the average values of the differences of the color values between the current frame and the corresponding pixels in the previous frame, It can be judged whether or not it is loaded. If the difference value obtained by dividing the sum of the average values of the differences of the color values between the current frame and the corresponding pixel in the previous frame by the number of pixels is less than the predetermined fourth threshold value, the state change is determined to be lower; otherwise, . ≪ / RTI >

FIGS. 6 and 7 are diagrams for explaining a processing procedure when a state change is detected.

In S610, when the state change of the load is detected, the control device of the refrigerator can delete the unloaded object from the object list. Further, the control device of the refrigerator can search for and delete the bounding box for the stored object stored in the storage medium 118.

In S620, the controller of the refrigerator can provide a message about the unloaded object. A notification message such as 'OO has been received' may be output through the output unit of the user terminal or the refrigerator. The notification message may include an icon that can be modified if it is determined that the unloading event has been detected incorrectly. When an input to which an icon that can be modified is selected is received, it is possible to control so that the unloaded object is not deleted from the object list and the bounding box of the corresponding object is also not deleted.

As shown in FIG. 7, in S710, when the state change of the loading is detected, the controller of the refrigerator can retrieve the loaded object using the pre-trained database. Color, and size of the object based on the current frame and the bounding box, and retrieve the name of the object according to the shape, color, and size of the object. Since the form of the food can be modified in various ways, the control device of the refrigerator can primarily determine a group or group of names including individuals based on color. In order to determine the exact name within the name group, set, the type, size, etc. of the entity may be utilized.

In S720, the control device of the refrigerator can use the database to determine the recommended use restriction date of the object based on the name of the retrieved object. At this time, the recommended use limit date can be adjusted not only by the name of the individual but also by the state (freshness, ripe, etc.) of the individual. The controller of the refrigerator can judge the current time point as the load date and time of the object. The recommended use limit date can be calculated based on the date and time of loading.

In S730, the control device of the refrigerator may generate a message including the name of the entity, the loading date, and the recommended use restriction date. The name of the entity, the date and time of loading, and the recommended use restriction date can be stored in the object list. In addition, the bounding box for an entity may be associated with a location in the object list and stored in a storage medium.

According to the present embodiment, the message may include information on the degree of corruption and degree of corruption of each entity included in the load list. That is, it can be determined using the recommended use day of each entity. A user who does not want to be notified can stop receiving the message by selecting the stop notification icon at the bottom of the message.

The controller of the refrigerator according to the present embodiment can generate and provide a loading list according to a request received from a user terminal connected to the network. The user will be able to identify the foods in the refrigerator while in an external location.

FIG. 8 is a flowchart detailing an operation of searching for an entity (S710).

The object searching step S710 includes receiving a bounding box S711, downsizing a bounding box S712, creating a map connecting a plurality of downsized images S713, A step S714 of searching the object database for an image similar to the image located at the n-th position of the transformation map or map (S715), a step S715 for determining whether the first name is plural (S716), and determining the name of the entity included in the bounding box as the first name (S718).

In S711, the refrigerator control device receives a bounding box containing the loaded object. In S712, the refrigerator control device downsizes the bounding box including the object until the size of the bounding box containing the object is 2x2. In S713, the refrigerator control device generates a map in which a plurality of images generated by downsizing the bounding box are arranged in order of magnitude. If the width and height of the bounding box are not one-to-one, then it may be necessary to square the bounding box proactively.

In S714, the refrigerator control device converts each image included in the map, and then generates a conversion map composed of the converted images. The refrigerator control device can convert at least one of color information, boundary information, and brightness information representing each image. For example, a refrigerator control device can convert an image to represent boundary information of an object.

In S715, the refrigerator control device can search an object database for an image similar to the n-th image located in the conversion map. The refrigerator control device searches an image similar to the n-th image among the images having the same size of the object database. The refrigerator control device determines the name of the searched image as the first name. The refrigerator control device can search similar images based on the relationship values of each pixel of the image. For example, the refrigerator control device may search for similar images based on the difference value of the feature values between each pixel and the surrounding pixels of each pixel. At this time, the refrigerator control device can search for images including difference values arranged in a matrix similar to the array of difference values of the feature values between respective pixels included in the first image, by a similar image.

In S716, the refrigerator control device can determine whether the first name is plural.

In S717, the refrigerator control device compares the image located at the (n-1) th position of the conversion map with the child image of the image located in the object database when the first name is plural. The refrigerator control device determines the name of the child image as the first name. In S718, the refrigerator control device outputs the first name when the first name is a single number.

In another embodiment, when the name of the entity included in the bounding box is not specified, the refrigerator control device can determine the generic name of the image corresponding to the entity as the name of the entity.

9 is a diagram illustrating an example of a refrigerator 100 including a refrigerator control apparatus according to embodiments of the present invention.

The controller 110 of the refrigerator may be electrically connected to the sensor unit 120, the image sensing device 130, the input unit 140, and the output unit 150. At least one of the sensor unit 120, the image capturing apparatus 130, the input unit 140, and the output unit 150 may be included in the refrigerator or may be electrically connected to the refrigerator.

The sensor unit 120 senses whether the door of the refrigerator is open or closed or whether the object is close to the door. When it is detected by the sensor unit 200 whether the door of the refrigerator is open or closed, the controller of the refrigerator can perform a process of determining a change in state occurring in the refrigerator. When the sensor unit 120 senses a user approaching the refrigerator, the controller of the refrigerator can control the state of the refrigerator, that is, the loading list to be outputted through the output unit 150. [

The image pickup device 130 is mounted inside the refrigerator and performs shooting of an internal image. Camera, but the embodiment is not limited thereto. The imaging device 130 is mounted at a plurality of points, i.e., left, right, and rear, in order to check changes in the state and state of the inside of the refrigerator. The current frame obtained from each imaging device 130 can be managed for each imaging device. The imaging device 130 can photograph each of the divided areas of the refrigerator. A plurality of divided zones included in the refrigerator

The input unit 140 may include a key pad, a dome switch, a touch pad (a contact type capacitance type, a pressure resistive type, an infrared detection type, a surface ultrasonic wave An electromagnetism measuring method, an electromagnetism measuring method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like.

The output unit 150 may be a liquid crystal display, a thin film transistor-liquid crystal display (LCD), or the like, as a means for providing a message related to the status change of the refrigerator and the loading list, ), An organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

10 is a view for explaining a refrigerator control system according to embodiments of the present invention.

The refrigerator control system according to embodiments of the present invention may include a refrigerator 100 including a refrigerator control device, a user terminal 200, and a communication network 300.

The user terminal 200 may perform a function of providing a load list or status message received through the communication network 300. [ Messages may be provided via the output included in the user terminal 200. [ Means a communication terminal that can use a web service in a wired / wireless communication environment. Here, the user terminal 200 may be a user's personal computer or a user's portable terminal.

In more detail, the user terminal 200 may be a computer (e.g., a desktop, a laptop, a tablet, etc.), a media computing platform (e.g., a cable, a satellite set top box, a digital video recorder), a handheld computing device E. G., A PDA, an email client, etc.), any form of cellular telephone, or any other type of computing or communication platform, but the invention is not so limited.

Meanwhile, the communication network 300 connects the user terminal 200 and the refrigerator 100. That is, the communication network 300 refers to a communication network that provides a connection path so that the user terminal 200 can access the refrigerator control device 110 and transmit / receive data. The communication network 300 may be a wired network such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, But the scope of the present invention is not limited thereto.

The embodiments of the present invention described above can be embodied in the form of a computer program that can be executed on various components on a computer, and the computer program can be recorded on a computer-readable medium. At this time, the medium may be a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floptical disk, , A RAM, a flash memory, and the like, which are specifically configured to store and execute program instructions.

Meanwhile, the computer program may be designed and configured specifically for the present invention or may be known and used by those skilled in the computer software field. Examples of computer programs may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like.

The specific acts described in the present invention are, by way of example, not intended to limit the scope of the invention in any way. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of such systems may be omitted. Also, the connections or connecting members of the lines between the components shown in the figures are illustrative of functional connections and / or physical or circuit connections, which may be replaced or additionally provided by a variety of functional connections, physical Connection, or circuit connections. Also, unless explicitly mentioned, such as " essential ", " importantly ", etc., it may not be a necessary component for application of the present invention.

The use of the terms " above " and similar indication words in the specification of the present invention (particularly in the claims) may refer to both singular and plural. In addition, in the present invention, when a range is described, it includes the invention to which the individual values belonging to the above range are applied (unless there is contradiction thereto), and each individual value constituting the above range is described in the detailed description of the invention The same. Finally, the steps may be performed in any suitable order, unless explicitly stated or contrary to the description of the steps constituting the method according to the invention. The present invention is not necessarily limited to the order of description of the above steps. The use of all examples or exemplary language (e.g., etc.) in this invention is for the purpose of describing the present invention only in detail and is not to be limited by the scope of the claims, It is not. It will also be appreciated by those skilled in the art that various modifications, combinations, and alterations may be made depending on design criteria and factors within the scope of the appended claims or equivalents thereof.

Claims (8)

An image obtaining unit for receiving a current frame of the internal state of the refrigerator;
An area detecting unit for detecting a change area by calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value of a corresponding pixel of a previous frame;
A half image generating unit for generating a half image which changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area;
A box extracting unit for extracting a bounding box including a pixel set to a first value included in the binary image;
A determining unit for determining a state change of the refrigerator by comparing the bounding box extracted through the box extracting unit with at least one pre-stored bounding box;
Wherein when the state change is judged to be a load through the determination unit, the control unit generates at least one image by adjusting the size of the bounding box to correspond to the bounding box, forms a map in which the generated images are arranged in a first order, And a processor for inferring the name of the loaded entity by comparing the map with the object database constructed with the images. The method according to claim 1,
The processing unit
By converting the images included in the map using a predetermined algorithm to generate a transformation map and comparing the first image of the transformation map and the second image having the size corresponding to the first image in the object data, Characterized in that the name of the entity is determined.
The method according to claim 1,
The determination unit
And generates a notification message including the name of the loaded object and the loading date and time through the determination unit, and provides the notification message to the user terminal after determining the name of the entity.
Receiving a current frame in which the control device has photographed the internal state of the refrigerator;
Detecting the change area by the controller calculating a pixel value for each pixel of the current frame and comparing the pixel value with a pixel value for a corresponding pixel of the previous frame;
The control device generating a dichromatic image that changes a pixel value of each pixel of the current frame to a first value or a second value based on the change area;
The control device extracting a bounding box containing a pixel set to a first value contained in the binary image;
Comparing the extracted bounding box with one or more pre-stored bounding boxes to determine a state change of the refrigerator;
Wherein when the state change is determined to be a load, the size of the bounding box is adjusted to generate at least one image corresponding to the bounding box, a map in which the generated images are arranged in a first order, And deducing the name of the loaded entity by comparing the map with the object database constructed from the received images. 5. The method of claim 4,
The step of deducing the name of the entity
By converting the images included in the map using a predetermined algorithm to generate a transformation map and comparing the first image of the transformation map and the second image having the size corresponding to the first image in the object data, Characterized in that the name of the entity is determined. 6. The method of claim 5,
Comparing the maps and the lower images of the corresponding images extracted from the object database to determine the name of the loaded object as the name of the lower image of the object database when the names of the objects are obtained in plurality The refrigerator control method. The method according to claim 6,
And generating a notification message including the name of the loaded entity and the loading date and time and providing the notification message to the user terminal after determining the name of the entity. A computer program stored in a computer-readable storage medium for executing the method of claim 4 using a computer.

Images