KR102347134B1 - Refrigerator and Controlling Method for the same - Google Patents

Abstract

The present invention relates to a refrigerator and a method for controlling the same, and more particularly, to a refrigerator capable of providing information about food stored in the refrigerator even if a user does not open a door, and a method for controlling the same.
According to an embodiment of the present invention, a storage room; a drawer movably provided in the storage compartment and provided with a marker; a camera for photographing the drawer from the outside of the drawer; And by detecting the position of the marker in the photos continuously taken through the camera, the drawer's state information including at least one of the drawer's withdrawal degree, whether the drawer is drawn out, the moving direction, the stationary state or the moving state is determined A refrigerator comprising a control unit may be provided.

Description

Refrigerator and Controlling Method for the same

The present invention relates to a refrigerator and a method for controlling the same, and more particularly, to a refrigerator capable of providing information about food stored in the refrigerator even if a user does not open a door, and a method for controlling the same.

BACKGROUND ART In general, a refrigerator is a device for supplying cold air generated by driving a refrigeration cycle, and is a device for storing food at a low temperature. In the case of the conventional refrigerator, it could simply perform a function for storing food at a low temperature. However, in recent years, the need for additional functions in addition to the food storage function is increasing.

The refrigerator is a device for storing and storing certain items, and in order to check the inside, there is no choice but to open the refrigerator door. In addition, when a user tries to buy a product at a market or a mart, if he or she does not know the amount and type of food stored in the refrigerator, there is the inconvenience of buying the same food repeatedly or not being able to buy necessary food.

Japanese Patent Publication No. 3450907 and Japanese Patent Application Laid-Open No. 2004-183987, which are prior art, disclose a technology in which a camera is installed in a door to photograph the inside of a refrigerator. In addition, Japanese Patent Laid-Open No. 2001-294308 discloses a technology in which a camera is installed in a cabinet, a drawer, and a door, respectively.

However, according to the reviewed prior art, since the photographing range of the camera installed in the refrigerator is limited, there is a problem that a plurality of cameras must be used to photograph various storage areas.

Therefore, it is necessary to reduce the number of cameras installed in the house, while increasing the range of the area photographed by the camera to improve the shooting efficiency of the camera.

In addition, according to the prior art, there were various problems in acquiring an image of the inside of the drawer. For example, a specific method for obtaining a picture taken at a desired point in time, concerns about condensation generated by the camera, and concerns for improving power consumption by the camera have not been made in detail.

Specifically, according to the prior art, when there are a plurality of storage areas including drawers in the refrigerator, there is a problem in that the specificity for providing optimal information to the user in consideration of the specificity and positional relationship of each area is lacking.

An object of the present invention is to basically solve the above-mentioned problem.

In order to solve the above problems through an embodiment of the present invention, the present invention provides a storage compartment display device for a refrigerator that provides information about food stored in the refrigerator even if the user does not open the refrigerator door, and a method for controlling the same. will provide

Another object of the present invention is to provide a refrigerator and a method for controlling the same through which a user can intuitively recognize information about food stored in each of a plurality of storage areas in a storage room through an embodiment of the present invention.

In addition, through an embodiment of the present invention, a user can intuitively recognize a positional relationship between a plurality of storage areas, and a refrigerator capable of providing stored food information substantially consistent with the user's visual point of view, and a refrigerator and the same We want to provide a control method.

In addition, through an embodiment of the present invention, the driving time of the camera and the photographing time of the camera are linked with the door opening and/or the rotation angle of the door, thereby reducing power consumption by the camera and providing food through the camera at an optimal time. An object of the present invention is to provide a refrigerator capable of acquiring storage information and a method for controlling the same.

In addition, through an embodiment of the present invention, it is intended to prevent the quality of the picture taken by the camera from being deteriorated due to dew condensation.

In addition, through an embodiment of the present invention, it is possible to provide the user with the latest information on the food stored in the refrigerator.

In addition, through an embodiment of the present invention, an administrator or a user can directly correct an area of a picture displayed to a user, thereby providing food storage information in an optimal state to the user.

Another object of the present invention is to provide a refrigerator capable of providing optimal food storage information to a user by installing a camera on a fixed body and preventing the camera from shaking, and a method for controlling the same.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof that can effectively identify and provide food storage information in the drawer while simplifying the configuration by grasping the state information of the drawer through software.

In addition, through an embodiment of the present invention, a refrigerator capable of providing the latest food storage information for each of a plurality of storage areas inside a storage room including a drawer through continuous shooting through one camera and a control method thereof are provided. want to

In addition, through an embodiment of the present invention, food stored in another storage area in which at least a portion of the inner space of the drawer overlaps up and down or another drawer inner space in which at least a portion of the inner space of the drawer overlaps up and down An object of the present invention is to provide a refrigerator capable of providing stored information using a single camera and a method for controlling the same.

In addition, through the embodiment of the present invention, it is possible to prevent the quality of the picture taken by the camera from being deteriorated due to dew condensation.

In addition, through the embodiment of the present invention, it is possible to provide a refrigerator capable of reducing the cost of the refrigerator by limiting the number of cameras installed in the refrigerator to one.

Another object of the present invention is to provide a refrigerator capable of effectively preventing an increase in power consumption due to a camera and a method for controlling the same through an embodiment of the present invention.

Another object of the present invention is to provide a refrigerator capable of minimizing the load of the control unit and the memory unit, and effectively and efficiently processing continuously taken pictures, and a method for controlling the same.

In order to achieve the above object, according to an embodiment of the present invention, a storage room; a drawer movably provided in the storage compartment and provided with a marker; a camera for photographing the drawer from the outside of the drawer; And by detecting the position of the marker in the photos continuously taken through the camera, the drawer's state information including at least one of the drawer's withdrawal degree, whether the drawer is drawn out, the moving direction, the stationary state or the moving state is determined A refrigerator comprising a control unit may be provided.

The controller may perform image processing only on a region within a preset range in which the marker may be located in a single photo.

The controller may determine a coordinate value of the marker in the photographed picture from reference coordinates for a path on which the marker is movable.

The reference coordinates may be one-dimensional coordinates parallel to the movement direction of the drawer. For example, if a coordinate value in a state in which the drawer is closed and in a state immediately before the drawer is effectively opened is 0, it may be determined that the drawer is more open as the coordinate value increases.

The control unit may compare two pictures taken continuously, and determine whether or not to move the drawer and the direction of movement according to a change in the position of the marker. For example, when the coordinate value is increased, it may be determined that the drawer is being opened, and in the opposite case, it may be determined that the drawer is being closed. And, when the coordinate value does not change, it can be determined that the drawer is in a stationary state.

In a section (first section) in which the drawer reaches a state in which the drawer is drawn out a predetermined distance, the control unit determines the position of the marker in one picture, and reaches a state in which the drawer is drawn out at the predetermined distance In the section (second section), the controller may determine a change in the position of the marker by comparing two consecutive photos.

The second section is preferably equal to or smaller than the length of the first section.

It is preferable that the time interval for capturing the camera is equal to or longer than the time required for the control unit to compare the positions of the markers located in the second section in the two consecutive photos.

It is preferable that the time interval for capturing the camera is equal to or longer than the time required for the control unit to determine the position of the marker.

The marker may be disposed on the upper surface of the front wall of the drawer.

The drawer includes a drawer disposed on the left side and a drawer disposed on the right side of the center of the storage compartment on the left and right, and the marker may be installed in each drawer.

The camera may be fixed to the upper ceiling of the drawer at the left and right central positions of the storage compartment, and the marker may be provided close to the left and right central positions of the storage compartment in each of the two drawers.

The marker may include a color different from the color of the drawer positioned at the border of the marker. In this way, the marker can be recognized more effectively.

Preferably, the marker includes a vertical side defined parallel to the movement direction of the drawer and a horizontal side defined perpendicular to the movement direction of the drawer, and the horizontal side is longer than the vertical side.

Preferably, the marker includes a first color part and a second color part having a different color from the first color part, and the first color part and the first color part are alternately arranged.

Preferably, the markers are repeatedly arranged and include at least one shape having different colors.

In order to achieve the above object, according to an embodiment of the present invention, a storage room; a drawer movably provided in the lower portion of the storage compartment and provided with a marker; a camera which is provided to be fixed to the ceiling of the storage chamber and is provided to photograph the inside of the drawer from the outside of the drawer; And by detecting the position of the marker in the photos continuously taken through the camera, the drawer's state information including at least one of the drawer's withdrawal degree, whether the drawer is drawn out, the moving direction, the stationary state or the moving state is determined A refrigerator comprising a control unit may be provided.

The controller may determine the state information of the drawer by detecting that the marker position in the continuously taken pictures is changed based on the marker position in the state in which the drawer is fully retracted.

In order to reduce the load of the control unit in the section (first section) from the drawn-in state to the drawn-out state by a predetermined distance and the section (second section) from the drawn-out state to the maximum withdrawal state (second section), The marker tracking and image processing method in the controller and whether the captured image is stored may be different.

Among the photos taken in the second section, a photo taken just before the drawer is closed may be captured and provided to the user. That is, the picture for the corresponding area may be updated using the picture taken at this time.

The second section is preferably shorter than the first section.

In the second section, the position of the marker may be detected by sequentially comparing two pictures taken continuously.

In order to achieve the above object, according to an embodiment of the present invention, the refrigerator body, formed by a fixed heat insulating wall, the storage compartment is provided with an opening; a door rotatably provided on the body to open and close the opening; a camera fixed to the storage compartment and photographing the inside of the storage compartment in order to update food information in the storage compartment; a door switch for detecting opening and closing of the door; a door sensor for detecting an opening angle of the door; And, according to the detection result of the door switch, a switching time between the standby mode of the camera and the driving mode for starting shooting is determined, and according to the detection result of the door sensor, a valid photo for updating among the captured photos A refrigerator may be provided that includes a control unit for determining a reference time point for selecting .

In order to achieve the above object, according to an embodiment of the present invention, the refrigerator body, formed by a fixed heat insulating wall, the storage compartment is provided with an opening; a door rotatably provided on the body to open and close the opening; a drawer provided in the storage compartment; Ceiling of the storage compartment to photograph an area (first area) in which food is stored in the space outside the drawer in the storage compartment and an area (second area) in which food is stored in the inner space of the drawer in the storage compartment A camera provided to be fixed to the; And through the shooting time of the photo in which the first area and the second area are taken together, the part for the first area (photo of the first area) and the part for the second area (photo of the second area) in the photo Separately, it may include a control unit for separately classifying and storing.

In order to achieve the above object, according to an embodiment of the present invention, the refrigerator body, formed by a fixed heat insulating wall, the storage compartment is provided with an opening; a door rotatably provided on the body to open and close the opening; a drawer provided in the storage compartment; Ceiling of the storage compartment to photograph an area (first area) in which food is stored in the space outside the drawer in the storage compartment and an area (second area) in which food is stored in the inner space of the drawer in the storage compartment A camera provided to be fixed to the; A part for the first area (a photo of the first area) and a part for the second area (a photo of the second area) are separated from the photo through the shooting time of the photo in which the first area and the second area are taken together. a control unit that separates and stores them individually; In addition, it is possible to provide a refrigerator including a display for displaying the photo of the first area and the photo of the second area to be distinguished from each other.

In order to achieve the above object, according to an embodiment of the present invention, the refrigerator body, formed by a fixed heat insulating wall, the storage compartment is provided with an opening; a door rotatably provided on the body to open and close the opening; a drawer provided in the storage compartment; Ceiling of the storage compartment to photograph an area (first area) in which food is stored in the space outside the drawer in the storage compartment and an area (second area) in which food is stored in the inner space of the drawer in the storage compartment A camera provided to be fixed to the; And the second area in the picture taken at a specific point in time when the drawer is drawn in after the drawer is pulled out and separated by separating the part (first area picture) of the first area from the picture taken at a specific point in time when the door is opened and closed A refrigerator may be provided, comprising a control unit that separates, separates, stores, and updates the photo of the first area and the photo of the second area by separating a part (a photo of the second area) for .

In order to achieve the above object, according to an embodiment of the present invention, the refrigerator body, formed by a fixed heat insulating wall, the storage compartment is provided with an opening; a door rotatably provided on the body to open and close the opening; a drawer provided in the storage compartment; at least one shelf provided in the storage compartment and positioned above the drawer; And an area (first area) in which food is stored by the shelf in the outer upper space of the drawer and an area (second area) in which food is stored in the inner space of the drawer are photographed together in the storage compartment, A refrigerator including a camera may be provided so as to be fixed to the ceiling of the storage compartment between the front end of the shelf and the opening.

In the above-described exemplary embodiments of the present invention, each feature may be implemented in combination with each other as long as they are not mutually exclusive or contradictory.

According to the present invention, it is not necessary for the user to open the door to obtain information about the food stored in the refrigerator, and thus, it is possible to prevent the cold air contained in the storage compartment from leaking. Accordingly, it is possible to improve the energy efficiency of the refrigerator by preventing unnecessary cold air loss.

In addition, in the present invention, since it is possible to provide the user with the latest information on the food stored in the refrigerator, the reliability of the food information provided to the user can be improved.

In addition, in the present invention, since information about food stored in various locations can be provided by one camera, only a structure for installing one camera is added, so that design can be made easily. In particular, the cost incurred due to the use of the camera can be reduced.

According to the present invention, dew is prevented from forming on the camera installed in the refrigerator, so that a picture taken by the camera can be stably provided to the user.

According to the present invention, an individual can remotely grasp the current internal situation of the refrigerator, and can receive food-related information necessary for the individual from an external business operator.

And, according to the present invention, it is possible to provide a screen in a form similar to the image captured by the camera of the inside of the drawer is actually seen when the user uses the refrigerator.

Also, according to the present invention, it is possible to provide a user with a location that cannot be viewed at once as a flat image on one screen because they overlap in spatial arrangement.

1 is a front view of a refrigerator according to the present invention.
Figure 2 is a view in which the door is opened in Figure 1;
3 is a view showing a third area disposed on the floor of the storage room;
4 is a control block diagram of an embodiment according to the present invention;
5 is a view for explaining an angle of view of a camera;
Fig. 6 is a side cross-sectional view of Fig. 5;
7 is a screen taken by the camera in the state of FIG.
8 is a view for explaining the positioning of the camera.
9 is a cross-sectional view showing the main part of the refrigerator.
10 is a view for explaining an operation method of the door sensor.
11 is a view specifically illustrating a left hinge part and a right hinge part;
12 is a screen provided to a user;
13 is a view for explaining a method of adjusting a picture taken by a camera.
14 is a perspective view showing a camera;
Fig. 15 is a view showing the main part of the camera;
Fig. 16 is a cross-sectional view of Fig. 14;
Fig. 17 is a view for explaining the arrangement of the heater;
18 is a view for explaining a picture taken in a state in which the camera is mounted in a vertical direction on the refrigerator.
19 is a view for explaining a picture taken in a state in which the camera is turned horizontally in the refrigerator;
20 is a view showing a state in which the camera housing is assembled;
21 is a front view of the first housing;
22 is a front view of the second housing;
23 is a front view showing a state in which the camera is installed in the first housing.
Fig. 24 is a side cross-sectional view of Fig. 23;
25 is a cross-sectional view showing a state in which the camera housing is installed in the inner case.
26 is a table comparing the power consumption of the camera in the present invention.
27 is a diagram comparing the supplied standby current and the driving current;
28 and 29 are diagrams for explaining a start point of the camera taking pictures and continuous shooting of the camera.
30 is a diagram illustrating an embodiment of a drawer sensing unit.
FIG. 31 is a view for explaining a method of detecting movement of a drawer in the drawer sensing unit of FIG. 30;
32 is a view showing a state in which a marker is displayed on the drawer.
Figure 33 (a) is a view showing a photograph of the state in which the left and right drawers are withdrawn.
33 (b) is a view showing a state in which the left drawer is drawn in, and the right drawer is drawn out.
34 is a flowchart illustrating a method of operating a refrigerator according to an embodiment of the present invention.
35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.
36 is a view for explaining a marker for detecting the withdrawal/input timing of the drawer according to an embodiment of the present invention.
37 is a view for explaining the shape of a marker according to an embodiment of the present invention.
38 is a flowchart illustrating a method for detecting a drawer closing operation according to an embodiment of the present invention.
39 is a flowchart illustrating an operation of the refrigerator performed when the completion of a drawer closing operation is detected according to an embodiment of the present invention.
40 is a flowchart for explaining a method of controlling an image of a specific area inside a refrigerator captured at a specific point in time according to an embodiment of the present invention.
41 is a view showing various types of markers;
42 is a view for explaining a marker position recognition method;
43 is a view showing the degree to which the drawer is opened.
Fig. 44 is a diagram for explaining movement of a marker;
45 is a control flow diagram according to an embodiment of the present invention.
Fig. 46 is a control flowchart for explaining a modification of Fig. 45;
Fig. 47 is a control flowchart for explaining another modification of Fig. 46;
Fig. 48 is a control flowchart for explaining another modification of Fig. 45;
Figure 49 is a modified example of Figure 45 differently.
50 is a control flowchart according to another embodiment of the present invention.
51 is a view for explaining a process of updating an image in a state in which two doors and two drawers are provided to open and close a storage room;
52 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
53 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
54 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
55 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
Fig. 56 is a view for explaining the operation of the heater of the camera;
57 is a view for explaining the experimental results regarding the dew condensation generated by temperature on the transparent window of the camera.
58 is a cross-sectional view of a transparent window;
59 and 60 are views schematically illustrating a structure in which a camera is installed in an inner case.
61 is a view showing another type of refrigerator to which the present invention is applicable.
62 is a screen provided to a user from the refrigerator of FIG. 61;
63 is a view for explaining a method of adjusting a picture taken by a camera in the refrigerator according to FIG. 61;

Hereinafter, a preferred embodiment of the present invention that can specifically realize the above object will be described with reference to the accompanying drawings.

1 is a front view of a refrigerator according to the present invention, and FIG. 2 is a view with the door opened in FIG. 1 . Hereinafter, it will be described with reference to FIGS. 1 and 2 .

The refrigerator according to the present invention includes a top mount type in which a freezing compartment and a refrigerating compartment, which are storage compartments in which food is stored, are partitioned up/down, and the freezing compartment is disposed above the refrigerating compartment, and a freezer compartment and a refrigerating compartment are partitioned into left/right sides. The same can be applied to side-by-side-type refrigerators.

However, in the present embodiment, for convenience of explanation, a bottom freezer type (Bottom Freezer-Type) in which the freezing compartment and the refrigerating compartment are partitioned up/down and the freezing compartment is disposed below the refrigerating compartment will be mainly described.

The main body of the refrigerator includes an outer case 10 that forms an overall appearance when viewed by a user from the outside, and an inner case 12 that forms a storage compartment 22 in which food is stored. A predetermined space may be formed between the outer case 10 and the inner case 12 to form a passage for circulating cool air. Meanwhile, a heat insulating material is filled between the outer case 10 and the inner case 12 so that the inside of the storage compartment 22 can maintain a relatively low temperature compared to the outside.

In addition, in the machine room (not shown) formed in the space between the outer case 10 and the inner case 12, a refrigerant cycle device for generating cool air by circulating the refrigerant is installed. The freshness of foods stored in the refrigerator can be maintained by keeping the inside of the refrigerator at a low temperature using the refrigerant cycle device. The refrigerant cycle device includes a compressor for compressing the refrigerant, and an evaporator (not shown) for converting a liquid refrigerant into a gaseous state to exchange heat with the outside.

The refrigerator is provided with doors 20 and 30 for opening and closing the storage compartment. At this time, each door may include a freezer compartment door 30 and the refrigerating compartment door 20 , and each door is rotatably installed on the main body of the refrigerator by a hinge at one end thereof. The freezing compartment door 30 and the refrigerating compartment door 20 may be formed in plurality. That is, as shown in FIG. 2 , the refrigerating compartment door 20 and the freezing compartment door 30 may be installed in a form that is opened around both corners of the refrigerator toward the front.

A foaming agent may be filled between the outer case 10 and the inner case 12 to insulate the space between the outside and the storage chamber 22 .

The storage chamber 22 forms a space insulated from the outside by the inner case 12 and the door 20 . When the door 20 seals the storage chamber 22 , the storage chamber 22 may form a space that is isolated from the outside and insulated. In other words, the storage chamber 22 is It can be said that the space is isolated from the outside through the insulation wall and the insulation wall by the case (10, 12).

The cold air supplied from the machine room can flow everywhere in the storage chamber 22 , so that the food stored in the storage chamber 22 can maintain a low temperature state.

The inner case 12 may constitute a barrier 60 at the bottom of the storage compartment 22 . A barrier 60 is installed at the lower end of the storage compartment 22 to distinguish the freezing compartment from the refrigerating compartment. The barrier 60 may have a predetermined thickness and may be formed in the inner case 12 . The barrier 60 may extend in a horizontal direction.

The storage compartment 22 may include a shelf 40 on which food is mounted. In this case, a plurality of shelves 40 may be provided, and food may be placed on each shelf 40 . The shelf 40 may partition the inside of the storage chamber in a horizontal direction.

A drawer 50 capable of being drawn in or drawn out is installed in the storage chamber 22 . Food and the like are accommodated and stored in the drawer 50 . Two drawers 40 may be disposed on the left and right in the storage chamber 22 . The user may open the left door of the storage compartment 22 in order to access the drawer disposed on the left side. On the other hand, in order for the user to access the drawer disposed on the right side, the right door of the storage compartment 22 may be opened.

A space in which food is stored may be formed in the barrier 60 . Since the barrier 60 is not provided on the door, but on the inner case 12, it is not moved according to the rotation of the door and is maintained in a fixed state. Accordingly, the user can store or withdraw food stably.

The storage compartment 22 is divided into a space positioned above the shelf 40 , a space formed by the drawer 50 , a space formed in the barrier 60 , etc., and a space in which food is stored It may be divided into a plurality of these.

In this case, each space is a storage space disposed inside one storage compartment 22 , and cold air supplied to the storage compartment 22 may be moved to each space. In other words, since each space is partitioned to the extent that cold air can be moved to each other, it has a different meaning from the storage room described above.

In particular, each space does not form a space insulated from each other, although a temperature difference may exist in one storage room, unlike the storage room forming a space insulated.

The cold air supplied to one storage chamber does not freely move to another storage chamber, but the cold air supplied to one storage chamber may freely move to each partitioned space installed inside one storage chamber. That is, the cold air positioned above the shelf 40 is movable into the space formed by the drawer 50 .

In addition, the present invention may include a single camera 70 for taking a picture inside the storage chamber (22). At this time, the camera 70 is provided to maintain a fixed position, it is possible to photograph the same part.

In particular, the camera 70 is installed on the upper wall of the inner case 12 to face downward, so that it is possible to take a picture of the food stored in the storage compartment 22 . The shape of the photographed picture may have a shape viewed by the user, that is, a shape viewed by the user when the refrigerator is actually used.

In particular, the camera 70 is installed in a position corresponding to the inside of the drawer when the drawer is in the fully open state, and the photographed picture can provide a screen similar to that seen by the user when the inside of the drawer is viewed from above.

In this case, the storage compartment 22 may include a first area 42 in which food is placed on the shelf 40 and a second area 52 defined by an inner space of the drawer 50 . In addition, a storage space different from the first area and the second area may include a third area 62 overlapping the second area 52 when the drawer 50 is withdrawn. This part will be described in more detail later.

The first region 42 and the second region 52 may have different heights as a whole in the storage chamber 22 . That is, the first region 42 and the second region 52 may be lowered in the order.

A picture taken by the camera 70 or various information about the refrigerator may be provided to the user through the display 14 provided on the front of the refrigerator. Also, the user may control the refrigerator through the display 14 .

The cold air supplied to the storage chamber 22 may move to the first area 42 and the second area 52 , and the cold air located in each area may move to other areas.

3 is a view showing a third area disposed on the floor of the storage room. Hereinafter, it will be described with reference to FIG. 3 .

A recessed portion is provided in the barrier 60 to store food in the corresponding portion, and this area may be referred to as the third area 62 . The third region 62 is a space formed in the barrier 60 .

In this case, the third area is a storage space disposed at a location different from that of the first area and the second area, and may provide a user with another space for storing food.

It is possible to store foods with high frequency of use, such as eggs, in the third area 62 , and for this purpose, it is possible to include a cover 68 for opening and closing the third area 62 . At this time, it is preferable that the cover 68 is made of a transparent material, so that when the camera 70 is photographing from the upper side, even if the cover 68 is covered, the food stored in the third area 62 can be photographed. .

Meanwhile, the drawer 50 is disposed above the barrier 60 . Accordingly, the user can access the third area 62 while the drawer 50 is drawn into the lower space of the shelf.

On the other hand, when the drawer 50 is withdrawn from the lower space of the shelf, the drawer 50 is disposed above the third area 62 , and the user approaches the third area 62 . Can not.

That is, when the drawer 50 is drawn out toward the user, the drawer 50 and the third area 62 overlap each other. At this time, when the user looks down on the drawer 50 , the third area 62 is not shown to the user, but the inside of the drawer 50 may be seen.

On the other hand, in a state in which the drawer 50 is not drawn out toward the user, the drawer 50 and the third area 62 are disposed so as not to overlap each other. When the user looks down on the third area 62 , the inside of the drawer 50 is not shown to the user, and only the third area 62 may be shown.

The first region 42 , the second region 52 , and the third region 62 may have different heights as a whole in the storage chamber 22 . That is, the first region 42 , the second region 52 , and the third region 62 may be lowered in this order.

The cold air supplied to the storage chamber 22 may be moved to the first area 42 , the second area 52 , and the third area 62 , and the cold air located in each area goes to another area. can also be moved.

That is, the area photographed by the camera 70 may be a plurality of storage spaces inside the storage chamber 22 . The camera 70 takes a picture of the inside of one storage chamber 22, but information about a plurality of partitioned storage spaces may be included in the inside of one picture.

At this time, the camera 70 may take a picture so that a plurality of storage spaces are included in one picture. In particular, it is possible to take a different storage space depending on the time taken for one photo.

For example, a camera disposed in a fixed form may photograph the first area and the second area or selectively select the second area and the third area according to the relationship between the time point at which a picture is taken and the position of the door or the drawer. It is possible to shoot with Details on this will be described later.

4A is a control block diagram of an embodiment according to the present invention. Hereinafter, it will be described with reference to FIG. 4A.

The present invention may include a control unit 100 that manages a single picture taken by the camera 70 by dividing it into a plurality of images. In this case, the image may mean a part of the photo processed or corrected by the controller 100 .

The controller 100 may command the camera 70 to take a picture, and may receive a picture taken by the camera 70 .

In addition, the control unit 100 may provide a portion of the divided image to the display 14 to provide the user with the latest information on the food stored in the storage compartment 22 . In this case, the display 14 may be installed on the front of the refrigerator, or it may be a separate device separated from the refrigerator. That is, it is possible for the user to receive the image related to the storage room 22 through an external communication terminal such as a mobile phone and obtain information.

In this case, the control unit 100 may divide the picture taken by the camera 70 for each area, classify it into a plurality of independent images, and provide it to the display 14 . In this case, the photo selected by the control unit 100 may include information about the recent storage room state in which the user finally accesses the storage room 22 and takes out food from the storage room or puts food into it.

An embodiment of the present invention may include a door switch 110 capable of detecting whether the door 20 opens or closes the storage compartment 22 . At this time, the door switch 110 is provided in the outer case 10 , and when the door 20 comes into contact with the outer case 10 , it is sensed that the door 20 seals the storage chamber 22 . can do. Also, when the door 20 does not contact the outer case 10 , it may be detected that the door 20 opens the storage compartment 22 .

In addition, an embodiment of the present invention may include a door sensor 120 that detects a rotation angle of the door 20 . In this case, the door sensor 120 may detect a rotation direction of the door 20 and a rotation angle of the door 20 . For example, when the door 20 is rotated by more than a certain angle, a change occurs in the door sensor 120 , and the door sensor 120 detects that the door 20 is rotated by more than a certain angle have. In addition, when the door 20 is rotated in a specific direction, it is possible to sense the rotation direction of the door 20 by detecting a change in a pulse generated according to the direction. Of course, the door sensor 120 may be modified in various forms other than the above-described method.

In addition to this, the door sensor 120 includes a light emitting unit and a light receiving unit to determine whether the light irradiated from the light emitting unit is transmitted to the light receiving unit, and thus it is also possible to detect the rotation angle of the door 20 .

In particular, the door sensor 120 may be driven only when the door switch 110 determines that the door 20 has opened the storage compartment 22 .

Meanwhile, when the door sensor 120 determines that the door 20 is rotated at a specific angle, it is also possible to take a picture with the camera 70 . In this case, even if the door switch 110 detects that the door 20 is opened, the camera 70 does not immediately start taking a picture, but takes a picture when the door sensor 120 reaches a specific angle. will be filmed

When the camera 70 starts shooting, it is also possible to continuously take pictures at predetermined time intervals. Of course, if the door sensor 120 determines that the door 20 is opened by a certain angle, the camera 70 can continue to take pictures until a command to stop taking pictures is provided, instead of a single picture. do.

The control unit 100 may include a drawer detection unit 130 that detects whether the drawer 50 is drawn out or retracted. In this case, the drawer detection unit 130 may refer to a portion in which the picture taken by the camera 70 reads an image in the control unit 100 . The drawer sensing unit 130 may sense the movement of the drawer 50 as well as the movement direction of the drawer 50 .

In particular, the drawer detection unit 130 may be implemented by software. The drawer detection unit 130 may detect the position of the drawer 50 by using information captured from a picture taken by the camera 70 .

The drawer 50 is provided with a recognition mark called a marker, and whether the drawer 50 has entered a specific section through the photograph in which the marker is taken, whether it has entered, whether it is stopped, or whether the movement direction is changed after stopping information can be found about The markers will be described in detail later.

Since the marker is marked on the drawer, it has the same movement as the drawer 50 . Accordingly, various information about the movement of the drawer 50 can be obtained by using the information of the picture taken by the camera 70 without a separate sensor for detecting the movement of the drawer.

The picture taken by the camera 70 may be stored in the storage unit 18 . In this case, the storage unit 18 may be provided in the refrigerator or may be provided in a device separate from the refrigerator. In this case, the storage unit 18 may be disposed in a refrigerator together with the display 14, or may be provided in a device other than the refrigerator together with the display 14, a server connected to the refrigerator and a network, or a terminal connected to the refrigerator and a network. have.

In this case, the storage unit 18 may not store all the photos taken by the camera 70 . For example, when a command to select a specific picture does not occur in the control unit 100, the storage unit 18 may delete a previously taken picture and store the currently taken picture (First In). , First Out). The storage unit 18 may reduce the storage capacity by storing only some of the photos taken by the camera 70 .

4B is a control block diagram of a modified form of an embodiment according to the present invention. Hereinafter, it will be described with reference to FIG. 4B.

In FIG. 4B , unlike in FIG. 4A , the drawer detection unit 130 may detect the movement of the drawer 50 separately without using information obtained from a picture taken by the camera 70 .

For example, the drawer sensing unit 130 may have the same shape as a plurality of Hall sensors. A plurality of Hall sensors may be installed in the movement path of the drawer 50 . When the drawer 50 is moved, it is possible to detect a change in the plurality of Hall sensors, respectively, to detect a position and a movement direction of the drawer 50 .

The drawer detection unit 130 detects the movement of the drawer, and when the camera 70 takes a picture, it is possible to grasp a point in time at which the inside of the drawer is included in the picture taken by the camera 70 . . That is, the drawer detection unit 130 may detect a point in time at which information about an item stored in the drawer can be expressed in the picture taken by the camera.

The drawer detection unit 130 may detect the movement of the drawer, and may detect a point in time when the user determines that the drawer is in the latest state after using the drawer. That is, the drawer detection unit 130 may detect a point in time including the latest information that the user has finished using the drawer in the picture taken by the camera. That is, it is possible to indirectly determine the completion time of the arrival and departure of goods or the end time of the update of the goods.

5 is a view for explaining an angle of view of a camera. Hereinafter, it will be described with reference to FIG. 5 .

In FIG. 5 , for convenience of explanation, the drawer 50 is withdrawn so that the camera 70 can photograph the second area 52 , and the shelf is omitted.

The camera 70 may be disposed on the upper wall of the inner case 12 to be inclined toward the rear wall of the inner case 12 .

Meanwhile, the picture taken by the camera 70 may have a horizontal angle of view along the width direction of the storage chamber 22 and a vertical angle of view along the front-rear direction.

At this time, the horizontal angle of view according to the camera 70 is provided so that the food stored in the inner space of the two drawers 50, that is, the second area 52, can be seen in the state where the two drawers 50 are drawn out. it is preferable

In addition, the range of the horizontal angle of view according to the camera 70 may include at least a part of the left and right ends of the shelf 40 . This is because it is preferable to photograph from the camera 70 to both ends of the shelf 40 in order to obtain information about the food placed on the upper side of the shelf 40 .

As the angle of view of the camera 70 is changed, the range inside the storage chamber shown in the picture is changed.

6 is a side cross-sectional view of FIG. 5 , and FIG. 7 shows a screen taken by the camera in the state of FIG. 6 . Hereinafter, it will be described with reference to FIGS. 6 to 7 .

An opening 14 is formed at the front of the inner case 12 so that a user can access the storage compartment 22 . Through the opening 14 , a user can put food into the storage compartment 22 or take out food from the storage compartment 22 . The door 20 may open and close the opening 14 .

The camera 70 may be installed on the outside of the drawer to take pictures while the drawer is moved or stopped to take pictures of the open state, the closed state, and the moving state of the drawer. That is, when the camera 70 continuously takes pictures, the state in which the position of the drawer is changed according to the shooting time can be expressed in the pictures taken by the camera 70 .

In addition, the camera 70 may provide the user with a screen including the recent arrival and departure status of the items inside the storage room 22 .

The camera 70 is preferably installed within a range l between the opening 14 and the front end of the shelf 40 . When there are a plurality of shelves 40, it is preferable to be installed in a region (l, see FIG. 8) between the front end and the opening 14, which is located at the front of the front ends. That is, it is preferable to be installed on the ceiling of the storage room corresponding to this area. Since the camera 70 has to selectively take pictures of the first area and the second area 52 or the first area and the third area 62 , it is better than the front end of the shelf 40 . It cannot be moved to the rear wall of the inner case 12 .

6 shows the vertical angle of view of the camera 70, one end (front end) of the vertical angle of view is preferably arranged so that the drawer 50 can be photographed up to the front end of the drawer 50 in an open state. do. In addition, the other end (rear end) of the vertical angle of view is preferably arranged so as to be able to photograph up to the rear end of the uppermost shelf.

Since the second area 52 by the drawer 50 is included in the range of the vertical angle of view of the camera 70 , the camera 70 allows the user to provide information on the second area 52 . can be obtained.

As shown in FIG. 7 , the camera 70 may take pictures of the first area 42 and the second area 52 as a single picture. At this time, the camera 70 may take pictures of the two drawers 50 disposed on the left and right. In addition, it can be seen that, when a plurality of shelves are provided, the first area 42 and the second area 52 divided by the plurality of shelves can be photographed together.

The camera 70 may be installed inside the storage chamber 22 to take a picture including information about the interior of the storage chamber 22 . Since the camera 70 is installed inside the storage chamber, it does not move. That is, the camera itself does not move because it is generally fixedly installed in the storage compartment of the main body, which is always fixed. Therefore, when the camera 70 takes a picture, the camera 70 is always in a stopped state.

The picture taken by the camera 70 includes the first area 42 and the second area 52 , or includes the first area 42 and the third area 62 .

When the camera 70 takes a picture including the first area 42 and the second area 52 , the user can view the food stored on the upper side of the shelf 40 and the deed through the corresponding picture. Information on food stored in the lower 50 may be acquired.

On the other hand, when the camera 70 takes a picture including the first area 42 and the third area 62 , the user uses the picture to store food stored on the upper side of the shelf 40 and Information on food stored in the barrier 60 may be obtained.

The camera 70 captures the inside of one storage chamber 22 and transmits the photographed picture, and the control unit 100 may obtain information on a plurality of storage spaces.

The camera 70 is disposed above the first area 42 , the second area 52 , and the third area 62 , and takes pictures while looking down.

8 is a view for explaining the positioning of the camera. Hereinafter, it will be described with reference to FIG. 8 .

As shown in FIG. 8 , the camera may be mounted at positions L1, L2, L3, and L4.

L1 is an external space partitioned by the outer case 10 to the outside of the storage chamber 22 . When the camera is installed at the position L1, the user uses the refrigerator while accessing the storage compartment 22 with the door open, but a problem that the camera is covered by the user may occur.

As disclosed in the published Japanese patent, when the camera is installed outside the refrigerator to photograph the inside of the storage room, the image interference by the user is too great, and in particular, the drawer is blocked and the movement of the drawer cannot be tracked. In the embodiment of the present invention, it was impossible to grasp the image of the drawer inside and the movement state of the drawer together. In addition, the L1 position may mean that the camera is located on the door. In this case, since the door is a movable configuration, the camera may be moved during shooting. Therefore, it can be said that it is very difficult to obtain a stable and clear picture.

L2 is the interior of the storage chamber 22 , and refers to the upper wall of the storage chamber 22 , that is, the ceiling. In an embodiment of the present invention, it corresponds to a location where the camera is installed, but there is no problem of being obscured by the user. In addition, since the upper wall of the storage chamber 22 corresponds to the inner case 12, it is convenient to arrange a wire for supplying electricity to the camera. And, since the inner case 12 is generally always fixed configuration, it can be stably mounted to the configuration in which the camera is fixed.

L3 is an inside of the storage chamber 22 and a side wall of the storage chamber 22 . At this time, the camera is installed so as to be inclined to take a picture from one side of the storage chamber 22 toward the other side of the storage chamber 22 . In order for the camera to be disposed in the middle based on the height of the storage chamber 22 rather than the uppermost end of the storage chamber 22 , it has to be installed on the side wall of the inner case 12 .

When the camera is installed on one wall, the camera lens is inevitably installed to face the other wall, so there is a problem in that an asymmetrical image is obtained due to the difference in the distance between the left and right sides of the interior of the storage chamber 22 .

L4 is the inside of the storage chamber 22 and means one end of the shelf 40 . In order for the camera to be disposed based on the height of the storage compartment 22 , it must be supported by a specific structure. In this case, the camera may be fixed to one end of the shelf 40 .

In this case, since the distance between the drawer 50 and the camera is shorter than that of L2, there is a problem in that a camera having a larger angle of view than that of L2 should be used. In addition, there is also a problem that the degree of distortion of the obtained picture becomes severe. In addition, various problems that may cause inconvenience to the user may occur in appearance, such as installing the electric wire supplied to the camera on the shelf 40 . And, the shelf 40 is generally movable. Accordingly, there may be a problem that the camera itself shakes when food is placed on the shelf.

The camera applied to the embodiment of the present invention has an angle of view of 120 degrees, and it is possible to provide performance of 20 fps in the VGA level.

9 is a cross-sectional view illustrating a main part of the refrigerator. Hereinafter, it will be described with reference to FIG. 9 .

A basket 21 for storing food may be provided inside the door 20 . The basket 21 may be installed on a surface where the door 20 faces the storage compartment 22 .

There may be a problem that the basket 21 is shown in the picture taken by the camera 70 . Therefore, in the embodiment of the present invention, it is possible to detect whether the door 20 is rotated by a predetermined angle θ or more through the door sensor 120 , and take a picture through the camera 70 . Of course, it is also possible to transmit information about the corresponding time point to the control unit 100 from the door sensor 120 so as to select a picture taken at a time point adjacent to the time point detected by the door sensor 120 .

For example, the predetermined angle θ may be 60 to 80 degrees. Of course, the angle may be variously changed according to the capacity of the refrigerator, the size of the door, or the front and rear lengths of the basket 21 .

At this time, the predetermined angle θ may mean an angle at which the basket 21 may not appear in the picture when the camera 70 takes a picture, and this predetermined angle θ is the It can vary depending on the left and right width and front and rear length.

The door switch 110 for detecting whether the door 20 is opened or closed may be installed on the upper side of the refrigerator. At this time, when the door switch 110 is pressed by the door 20, it is sensed that the door 20 seals the storage compartment, and when not pressed by the door 20, the door 20 closes the storage compartment. You can sense it's open.

A left hinge part 300 for rotatably installing the left door 20 to the refrigerator may be provided, and a right hinge part 320 for rotatably installing the right door 20 to the refrigerator may be provided. The left hinge part 300 and the right hinge part 320 are provided with a rotating shaft on which the door 20 can rotate and are coupled to the door 20, and a part installed in the outer case 10; It may be divided into a connection part connecting two parts. That is, it may be divided into door coupling parts 300c and 302c, outer case coupling parts 300a and 320a, and connecting parts 300b and 320b connecting them.

A left door sensor 120L for detecting rotation of the left door 20 may be installed in the left hinge part 300 .

A right door sensor 120R for detecting rotation of the right door 20 may be installed in the right hinge part 320 .

The left door sensor 120L and the right door sensor 120R may independently detect the rotation angle of each door.

10 is a view for explaining an operation method of the door sensor. Hereinafter, it will be described with reference to FIG. 10 .

The door sensor 120 may include a light emitting unit 122 for irradiating light and a light receiving unit 124 for receiving the light irradiated from the light emitting unit 122 .

The light irradiated from the light emitting unit 122 may be reflected through the upper surface of the door 20 and transmitted to the light receiving unit 124 . In this case, it is also possible to be composed of a material that reflects well so that light reflection can be stably made.

When the light irradiated from the light emitting unit 122 is transmitted to the light receiving unit 124 , it may be detected that the door 20 is rotated by an angle smaller than the angle at which the door sensor 120 is installed. That is, it may mean that the door 20 is rotated at an angle smaller than a predetermined angle θ, or that the storage chamber 22 is not rotated in a sealed state.

On the other hand, if the light irradiated from the light emitting unit 122 is not transmitted to the light receiving unit 124 , it may be detected that the door 20 is rotated by an angle greater than the angle at which the door sensor 120 is installed. That is, the door 20 is rotated in a state greater than a predetermined angle θ, which may mean a state in which the user can access the storage chamber 22 .

To this end, as shown in FIG. 9 , the left hinge part 300 and the right hinge part 320 may be mounted over the top surface of the outer case 10 of the body and the top surface of the door 20 . Accordingly, as the door 20 rotates with respect to the rotation centers 302 and 322 , the uppermost surface of the door or a portion (eg, a stepped portion) of the upper surface of the door may function as a light receiving unit. Of course, the left door sensor 120L and the right door sensor 120R may be installed in the respective connection units 300b and 320b to perform the function of the light emitting unit. Due to the positional relationship between the sensors 120L and 120R and the door, the sensors 120L and 120R determine when the door opens at a certain angle (θ) or more and when the door closes at a certain angle (θ) can

11 is a diagram specifically illustrating a left hinge part and a right hinge part. 11 , other components of the refrigerator are omitted for convenience of explanation. Hereinafter, it will be described with reference to FIG. 11 .

The door sensor may be disposed in front of the inner case, that is, in front of the opening (ie, a position where the user stands) in order to detect the rotation of the door.

A left door sensor 120L for detecting the rotation of the left door 20 may be installed in the left hinge part 300 .

A right door sensor 120R for detecting the rotation of the right door 20 may be installed in the right hinge part 320 .

The left door sensor or the right door sensor may be disposed in a region between the opening and the respective rotation centers of the respective hinges. That is, it may be disposed on the connecting portions 300b and 320b of each hinge portion.

The door sensor 120 may be connected to other components by a connector c and a wire in order to receive electricity or transmit a signal to the outside. In this case, the door sensor 120 may provide signal information obtained by being connected to the above-described control unit 100 .

The left hinge part 300 may have a rotation center 302 through which the left door 20 is rotated. In this case, the left door sensor 120L may be disposed on the right side with respect to the rotation center 302 . The space on the right from the rotation center 302 may or may not be located below the left door sensor 120L, depending on the rotation angle of the left door 20, the upper surface of the left door 20, particularly the left door 20, is located below the left door sensor 120L. It is a space that can Accordingly, it is preferable that the left door sensor 120L is positioned at the connection part 300b of the left hinge part 300 . Through this, when the size of the door or the size of the basket is changed, it is possible to flexibly change the predetermined angle θ. That is, the space on the right side of the connection part 300b can be easily changed, and accordingly, the position of the left door sensor 120L in the connection part 300b can be easily changed.

The wire w connected to the left door sensor 120L may be connected through the rotation center 302 . Also, the wire w may be connected to the refrigerator body through the outer case 10 .

The right hinge part 320 may have a rotation center 322 through which the right door 20 is rotated. In this case, the right door sensor 120R may be disposed on the left side with respect to the rotation center 322 . The space left from the rotation center 322 is a space in which the right door 20 may or may not be located under the right door sensor 120R according to the rotation angle of the right door 20 . Therefore, for the same reason, it is preferable that the right door sensor 120R is installed in the connection part 320b of the right hinge part 320 .

The time point detected by the door sensor 120 may be a capture point of a picture for each area inside the storage compartment 22 , specifically, the first area 42 or the third area 62 . . That is, the time sensed by the door sensor 120 may mean a capture time of a picture taken in the external storage space of the drawer 50 .

Meanwhile, since the door sensor 120 irradiates and receives light, the door sensor 120 irradiates the light downward. The downwardly irradiated light is reflected upward again by the door 20 .

The door sensor 120 and the door 20 may maintain a short distance to receive a sufficient amount of light from the door sensor 120 . In this case, it is also possible to maintain a vertical distance between the door sensor 120 and the door 20 at a maximum of 20 mm.

12 is an example of a screen provided to a user. Hereinafter, it will be described with reference to FIG. 12 .

In FIG. 12A , the first area 42 , the second area 52 , and the third area 62 are all provided to the user, and in FIG. 12B , the second area 52 and the third area 62 are provided to the user. In the form provided to the user, FIG. 12C shows the form in which the first area 42 and the second area 52 are provided to the user.

First, it will be described with reference to FIG. 12A.

One picture taken by the camera 70 shows a first image showing the first region 42 , a second image showing the second region 52 , and the third region 62 . It can be divided into a third image that That is, one picture or a plurality of pictures may be divided by area. In this case, the first image, the second image, and the third image may mean a shape cut in a range showing a corresponding area from a single photographed photo.

The divided photo may be independently provided to the user as shown in FIG. 12A . That is, a picture of the corresponding area is provided to the user individually, so that the user can easily obtain information about the food stored in the corresponding area.

The display 14 may be arranged such that a plurality of storage areas can be displayed in a frame.

That is, the second image showing the second region 52 is disposed on the left and right sides of the upper side of one rectangular frame, and the third image showing the third region 62 is located at the lower side of the rectangular frame on the left and to the right, respectively.

A plurality of regions may be arranged as an image having the same width in one frame on the display 14 . To this end, the controller may crop the photographed picture, correct the size, and provide it to the display 14 . At this time, the corrected image is displayed at the corresponding position of the frame.

At least some of the images provided on the display are corrected to have the same width, so that the user has a feeling similar to looking at the storage room when the refrigerator door is actually opened, thereby having a planar arrangement effect.

The display 14 may provide two overlapping regions through one screen to provide food information in a state that an actual user cannot see at once.

In particular, since the second image and the third image are spaces arranged to overlap in the state in which the drawer is withdrawn, the user cannot simultaneously acquire information of the two storage spaces in the state in which the drawer is withdrawn. However, the display 14 may simultaneously provide two pieces of storage space information.

The controller 100 may update (replace) each image individually to provide the user with information on the type of food accommodated in each area.

For example, when the part corresponding to the second image is upgraded through the photo taken by the camera and the part corresponding to the third image does not need to be upgraded, only the second image may be replaced.

On the contrary, when the part corresponding to the third image is upgraded through the picture taken by the camera and the part corresponding to the second image does not need to be upgraded, only the third image may be replaced.

It is possible to upgrade the second image and the third image in one frame independently of each other and regardless of other images.

And it is also possible that only the second image disposed on the left side of the two second images disposed on the left and right sides is upgraded, or only the second image disposed on the right side is upgraded. In this case, the third image may also be individually upgraded on the left and right sides.

That is, the display 14 provides a frame in which a plurality of images are provided to provide information about an item stored in the storage room to the user, but the plurality of images can be individually upgraded regardless of other images. In particular, since the second image and the third image are images selected from different photos, the controller determines which region the corresponding photo represents, and it is possible to upgrade each image.

In this case, the second image and the third image may have the same or substantially similar width. That is, the portions corresponding to the second image and the third image obtained from the photo may be converted to the same width by the correction of the controller and provided to the display.

When the sizes of the second image and the third image are different from each other, the number of pixels of the second image and the third image in the corresponding photo, that is, different sizes according to pixels, may be selected. Alternatively, it is also possible to select the second image and the third image with the same number of pixels and correct them in the control unit to have different sizes when displayed on the display 14 .

By arranging the second image and the third image vertically, it is possible to have a flat feel of the vertical arrangement. The area originally shown by the second image is disposed above the area shown by the third image. Accordingly, the user can intuitively recognize the image provided by the display, and thus can easily grasp the information provided by the image.

Of course, the display 14 may provide the user with a screen in which the first image and the third image are simultaneously replaced. On the contrary, a screen in which only the second image is replaced may be provided to the user while the first image and the third image remain the same.

The refrigerator compartment shelf shown in FIG. 12A may mean the first area 42 , the vegetable compartment may mean the second area 52 , and the multi-storing corner may mean the third area 62 . Of course, it is possible to have various arrangements and to be transformed into various shapes according to the user's preference or the convenience of the image provider.

Meanwhile, the screen of FIG. 12A may be provided to the display 14 installed in the refrigerator, but may also be provided to a separate external device other than the refrigerator, for example, an external terminal device such as a smart phone. Accordingly, the user can obtain information about the food stored in the refrigerator while going out of the house, and use it for shopping. Accordingly, the user can determine the type of food stored in the refrigerator even if the user does not open the door. The screen provided to the user may be displayed by dividing one storage room into a plurality of storage areas. Accordingly, the user can easily obtain information classified by each location on the food stored at each location in one storage room.

At this time, the image shown in the vegetable compartment shows an area from the front end of the drawer 50 to the front end of the shelf 40 vertically when the drawer 50 is withdrawn. The image shown in the multi-storage corner shows the area of the overlapping portion when the drawer 50 is retracted in a state not covered by the door 20 . The shelf shows the user the area above the shelf 40 .

In one embodiment of the present invention, the screen provided to the user does not show one storage room as it is, but separate images are arranged in each part separated from each other, so that the user can easily obtain information. In other words, it can be said that the photo shown in FIG. 7 is not used as it is, but each area is separated from one photo and used independently and individually.

As shown in FIG. 12B , a screen in a form different from that of FIG. 12A may be provided. In this case, the camera 70 may take a picture including the second area 52 and the third area 62 . Otherwise, even if the camera 70 takes a picture including all of the first area 42 , the second area 52 , and the third area 62 , images corresponding to the two areas are provided to the user. It is possible to provide only

Another type of screen may be provided as shown in FIG. 12C . In this case, the camera 70 may take a picture including the first area 42 and the second area 52 . Otherwise, even if the camera 70 takes a picture including all of the first area 42 , the second area 52 , and the third area 62 , images corresponding to the two areas are provided to the user. It is possible to provide only

That is, according to an embodiment of the present invention, it is also possible to provide a related image to the user after photographing all three areas. However, as shown in FIGS. 12B and 12C , it is possible to provide the user with at least an image corresponding to the internal storage area of the drawer, that is, the second area and an image corresponding to a different area other than the second area.

13 is a view for explaining a method of adjusting a picture taken by a camera. Hereinafter, it will be described with reference to FIG. 13 .

13 may be an example of a screen displayed on the display 14 . Of course, if the refrigerator is not provided with a display, FIG. 13 may mean a screen displayed on a separate display device of an operator when manufacturing the refrigerator. It may also be a screen displayed on the user's mobile phone.

In a state in which the camera 70 is installed, the camera 70 may not be accurately installed at a desired position due to assembly tolerances.

Therefore, in the embodiment of the present invention, it is possible for an operator or a user to change the control line 15 that divides each area through a picture or an image taken by the camera 70 .

The adjustment line 15 may include a vertical adjustment line and a horizontal adjustment line. The vertical adjustment line may be placed at the center or both ends of the image. The user may select a portion from which the image provided on the display 14 is cut by moving the vertical adjustment line up and down and the horizontal adjustment line left and right through the buttons shown on the screen.

Meanwhile, a limit line 16 that limits the movable range of the adjustment line 15 may also be shown. In this case, the limit line 16 may be arranged to have a predetermined interval on the left and right with respect to the vertical adjustment line. In addition, the limit line 16 may be arranged to have a predetermined interval vertically with respect to the horizontal adjustment line.

By providing the limit line 16 , if an image of a desired degree is not obtained despite the movement of the adjustment line 15 , it is determined that the image is defective and the installation position of the camera 70 can be adjusted.

In a form in which two drawers are provided in the storage compartment as shown in FIG. 13 , the vertical adjustment line may be disposed between the two drawers.

And the horizontal adjustment line may be disposed at the front end of the shelf.

In a state in which the horizontal adjustment line and the vertical adjustment line are moved, the lower left corner of the intersection of the horizontal adjustment line and the vertical adjustment line may constitute an image of the left portion of the second region or the third region. In this case, the corresponding image may extend from the horizontal adjustment line to a specific number of pixels in the horizontal direction, and may extend from the vertical adjustment line to a specific number of pixels in the vertical direction.

A lower right corner of the intersection of the horizontal adjustment line and the vertical adjustment line may constitute an image of a right portion of the second region or the third region. In this case, the corresponding image may extend from the horizontal adjustment line to a specific number of pixels in the horizontal direction, and may extend from the vertical adjustment line to a specific number of pixels in the vertical direction.

An upper side of the horizontal adjustment line may constitute an image of the first region. In this case, the corresponding image may extend from the horizontal adjustment line to a specific number of pixels in the vertical direction.

Meanwhile, as shown in FIG. 13 , it is also possible to separately show the left and right photos of the third area corresponding to the vegetable compartment to check whether a photo of the portion corresponding to the drawer is obtained.

When the adjustment by the user or operator is completed, it is possible to input that the movement of the adjustment line 15 is completed by clicking the input completion button. Accordingly, when the installation error of the camera is small, it is possible to adjust the positions of vertical and/or horizontal lines dividing each area in the photographed picture. Through this, parts corresponding to a specific area can be clearly displayed. Of course, when there are a plurality of regions divided vertically or horizontally, a plurality of the aforementioned horizontal or vertical adjustment lines may be provided.

FIG. 14 is a perspective view of the camera, FIG. 15 is a view showing an essential part of the camera, and FIG. 16 is a cross-sectional view of FIG. 14 . Hereinafter, it will be described with reference to FIGS. 14 to 16 .

The camera 70 may include a camera module 71 including a camera lens 71a and a body 71b, and cases 73 and 74 in which the camera module 71 is accommodated.

The camera module 71 may include an image sensor that converts image information passing through the camera lens 71a into a digital signal.

The case may include a first case 73 and a second case 74 coupled to the first case 73 to form a predetermined space therein. The camera body 71b and the camera lens 71a may be accommodated in the first case 73 and the second case 74 . The coupling degree may be maintained between the first case 73 and the second case 74 by a separate sealing member to prevent moisture or dust from entering the inside.

Since the camera body 71b and the camera lens 71a are inserted into the space formed by the first case 73 and the second case 74, the inner case 12 and the outer case 10 ) does not come into contact with the foaming agent filled in between. In addition, the camera module 71 may be protected without direct contact with the cold air accommodated in the storage chamber 22 .

The first case 73 may include a transparent window 80 installed at the front end of the camera lens 71a and a heater 84 providing heat to the transparent window 80 . The transparent window 80 is preferably made of a transparent material.

It is possible to prevent dew from forming on the transparent window 80 by the heat provided by the heater 84 .

If the heater 84 is installed on the camera lens 71a to prevent dew formation, the camera lens 71a may be deformed. Since the occurrence of aberration and permanent damage due to such deformation increases, in the embodiment of the present invention, the transparent window 80 does not directly contact the camera range 71a.

As a material of the transparent window 80, both plastic and glass can be applied from a camera image acquisition point of view, but plastic has poor thermal conductivity. Therefore, when plastic is used as the transparent window 80, power consumption may be increased compared to using glass. Therefore, the transparent window 80 is preferably made of a glass material to reduce power consumption.

When the thermal conductivity is increased, the heat supplied by the heater 84 may be easily conducted from the transparent window 80 to a portion not in direct contact with the heater 84 . For example, when the thermal conductivity is lowered, the temperature of the portion adjacent to the heater 84 is high, while the temperature of the portion away from the heater 84 is lowered, so that it is difficult to prevent dew condensation. Also, since more heat energy must be supplied by the heater 84, energy efficiency is reduced.

Meanwhile, since the heater 84 directly makes a direct surface contact with the transparent window 80, heat transfer efficiency may be improved.

The heater 84 is disposed on the opposite side of the surface where the transparent window 80 is exposed to the storage chamber 22 so that the heat generated by the heater 84 does not pass through the transparent window 80 , and the storage chamber (22) can be prevented from being supplied.

In particular, the heater 84 may include a hot wire coil for generating heat, and the hot wire coil may make surface contact with the transparent window 80 .

In addition, it is preferable that the transparent window 80 and the camera lens 71a are provided with a predetermined distance g. This is to consider deformation that may be generated by supplying heat to the transparent window 80 or assembly tolerance.

In addition, the camera 70 may include a cable 79 that receives electricity supplied from the outside and transmits a signal related to the acquired picture. The cable 79 may electrically connect the camera module 71 to an external component.

Meanwhile, the case is provided with a first seating piece 75 capable of fixing the camera 70 to other components. At this time, the first seating piece 75 is provided in any one of the first case 73 or the second case 74 and is irrelevant.

The first seating piece 75 may include a surface having a predetermined area, and the camera 70 may be disposed at a desired installation position to have a desired angle of inclination.

The first seating piece 75 is provided on both sides of the camera 70, so that the camera 70 can be fixed on the left and right sides by external components.

A second seating piece 76 formed to have a predetermined area and to be stepped may be formed in the second case 74 . The second seating piece 76 is provided in a different part from the first seating piece 75, so that the camera 70 is positioned at a specific location and at a predetermined angle to other components, similarly to the first seating piece 75 . to ensure stable bonding. Accordingly, the camera 70 may be fixed to have three or more supporting points with respect to other components at a predetermined angle through the first seating piece 75 and the second seating piece 76 . Details of the support structure of the camera 70 will be described later.

The camera 70 may have a resolution of VGA Resolution, and a resolution of VGA (640x480) or 300,000 pixels may be used in consideration of image data transmission speed and processing time.

The interface of the camera 70 is a USB type and can be driven using 5V, 500mA USB power.

In addition, the current value supplied to the camera 70 may be 87 mA in the standby mode and 187 mA (67 mA in the heater) in the operation mode. Accordingly, power consumption proportional to the supplied current value may be consumed. At this time, the operation of the camera 70 is divided into a standby mode and an operation mode, and it is possible to always apply power to the heater 84 in order to prevent condensation on the surface of the transparent window 80 .

Meanwhile, the camera 70 operates in a standby mode when a rated voltage is applied, and when the door switch 110 detects that the door 20 is opened, the camera 70 can be switched to an operation mode to take a picture. That is, when power is applied to the refrigerator, the camera operates in a standby mode, and when the door switch 110 detects that the door 20 is opened, it can be switched to an operation mode for taking a picture.

17 is a view for explaining the arrangement of the heater. Hereinafter, it will be described with reference to FIG. 17 .

The camera 70 may have a horizontal angle of view of 115°, a vertical angle of view of 95°, and a diagonal angle of view of 145°. That is, it is possible to arrange the camera 70 so that the horizontal angle of view is larger than the vertical angle of view. Through this arrangement, when the width direction length (horizontal direction in FIG. 17) of the storage compartment 22 is longer than the front-back direction length (vertical direction in FIG. 17) of the storage compartment 22 through this arrangement, food information about the storage compartment 22 It is possible to obtain a photograph that can provide

Meanwhile, the heater 84 may be formed along the outline of the transparent window 80 . In this case, the heater 84 may be arranged to form a rectangle so as not to block the angle of view of the camera 70 .

In addition, when the heater 84 has a rectangular shape, heat can be supplied to the transparent window 80 over a relatively large area, so that the energy efficiency of the heater 84 can be improved.

In particular, when a gap g is provided between the camera lens 71a and the transparent window 80 as shown in FIG. 16 , the angle of view with respect to the transparent window 80 forms a rectangular shape as shown in FIG. 17 . If there is no gap g and the transparent window 80 and the camera lens 71a are in contact, the heater 84 comes into contact with one surface of the transparent window 80, so that the camera lens 71a ) may be provided in the form of a square having a length equal to or greater than the diameter of the .

However, as described above, since the gap g exists and the vertical and horizontal angles of view are different from each other, the heater 84 preferably has a rectangular shape with one side longer than the other side.

18 is a view illustrating a picture taken while the camera is mounted on the refrigerator with the camera twisted in the vertical direction (front and back direction), and FIG. 19 is a picture taken while the camera is rotated horizontally (left and right direction) on the refrigerator It is a drawing. Hereinafter, it will be described with reference to FIGS. 18 and 19 .

18A is a state in which the camera is rotated upward by 5 degrees, FIG. 18B is installed at a desired position, and FIG. 18C is a state in which the camera is rotated downward by 5 degrees. The picture shown in FIG. 18 may provide the user with information about the food stored in the storage room, but if it is deviated at a larger angle, the desired information may not be provided to the user.

19A is a state in which the camera is turned to the left by 4 degrees, FIG. 19B is a state in which the camera is turned to the left by two degrees, and FIG. 19C is a state in which the camera is installed at a desired position.

When compared to a picture in which the camera is installed to rotate in the vertical direction as shown in FIG. 18, a picture in which the camera is not installed at a desired angle in the horizontal direction as shown in FIG. 19 provides a relatively large distorted screen.

As can be seen in FIGS. 18 and 19 , even if the installation angle of the camera is slightly different, it is necessary to manage the installation angle of the camera so that the assembly tolerance is as small as possible, since it may cause distortion of the photographed picture. Of course, minute errors can be adjusted through the adjustment line described with reference to FIG. 13 , so that an optimal picture can be provided.

20 is a view showing the camera housing is assembled, FIG. 21 is a front view of the first housing, FIG. 22 is a front view of the second housing, and FIG. 23 is a front view showing a state in which the camera is installed in the first housing and FIG. 24 is a side cross-sectional view of FIG. 23, and FIG. 25 is a cross-sectional view illustrating a state in which the camera housing is installed in the inner case. Hereinafter, it will be described with reference to FIGS. 20 to 25 .

A camera housing may be provided to install the camera 70 to the inner case 12 . That is, a camera housing for accommodating the camera shown in FIG. 14 and fixing the camera to the ceiling of the storage room may be provided.

The camera housing includes a first housing 400 coupled to a lower surface of the ceiling of the inner case 12, a second housing 410 coupled to an upper surface of the ceiling of the inner case 12, and the first housing 400 ) may include a third housing 420 and a fourth housing 430 coupled to. Here, the first housing 400 may be said to be coupled to the upper surface of the ceiling of the storage room.

The third housing 420 may close the front of the camera 70 so that parts other than the transparent window 80 of the camera 70 are not directly exposed to the storage chamber 22 . That is, the third housing 420 may function as a cover that covers the front of the camera 70 . In this case, the third housing 420 and the fourth housing 430 may be formed of one component. Of course, the fourth housing 430 may be a decorative housing interposed between the third housing 320 and the first housing 400 .

The camera 70 is assembled in the first housing 400 to manage the left and right inclination (horizontal inclination) and front and back inclination (vertical inclination) of the camera 70. can

As shown in FIG. 21 , the first housing 400 may include a first seating part 402 on which the first seating piece 75 of the camera 70 is mounted. In this case, two first seating parts 402 are provided so that the left and right first seating pieces 75 can be fastened to each other.

The first seating piece 75 may be provided to be inclined at a predetermined angle on the left and right sides of the camera 70 , and may be disposed to be inclined at a desired angle while the camera 70 is installed in the inner case 12 . .

A fastening hole 404 is provided in the first seating part 402 , so that the first seating piece 75 and the first seating part 402 can be fixed by a screw. The first seating part 402 is provided to be disposed at different heights in the horizontal direction, so that the camera 70 can be stably fixed.

The first housing 400 may include a second seating part 408 on which the second seating piece 76 of the camera 70 can be mounted. At this time, the second seating part 408 is also formed to have a predetermined area, so that the degree of inclination of the camera 70 can be fixed while making surface contact with the second seating piece 76 .

That is, the camera 70 may be coupled to the first housing 400 while in contact with the plurality of surfaces, thereby performing a function of fixing the camera 70 to be inclined at a desired angle. In other words, while being fixed to have three or more supporting points, it may be supported at a predetermined angle with respect to the first housing 400 . Of course, the horizontal angle may be fixed to be parallel to the first housing 400 , and the vertical angle may be fixed to be a predetermined angle.

The first housing 400 may include a fastening hole 406 to be coupled to the second housing 410 by a screw.

As shown in FIG. 22 , the second housing 410 has a contact surface 416 that can be in surface contact with the upper surface of the ceiling of the inner case 12 . At this time, the contact surface 416 is widely distributed while forming a rectangular shape as a whole, so that the second housing 410 can be stably seated on the upper surface of the ceiling of the inner case 12. Accordingly, the second housing ( It has a surface parallel to the ceiling of the inner case 12 of the contact surface 416 of the 410). Through this, the reference line or reference plane for the installation angle of the camera may be the same as the ceiling surface of the inner case 12 .

The second housing 410 may include an accommodation space 412 in which the cable 79 of the camera 70 can be accommodated. A through hole 418 is formed at one side of the accommodation space 412 , so that electric wires connected from other components of the refrigerator may be exposed to the accommodation space 412 . In this case, in the accommodating space 412 , the electric wire connected from other components of the refrigerator and the cable 79 may be connected. Accordingly, electricity may be supplied to the camera 70 through the cable 79 , and a signal for a picture taken by the camera 70 may be transmitted to other components.

A fastening part 414 coupled to the first housing 400 may be provided in the second housing 410 . The fastening part 414 may be coupled to the fastening hole 406 by a screw. When the fastening part 414 is coupled to the fastening hole 406, it is preferable that an error due to assembly does not occur. Therefore, it is also possible to be formed to protrude by a predetermined height so as to be in surface contact with the fastening hole 406 .

As shown in FIGS. 23 and 24 , when the camera 70 is assembled to the first housing 400 , the camera 70 may be disposed to be inclined by a set angle. As shown in FIG. 24 , one side 400a of the first housing 400 has a flat shape so as to be in contact with the inner case 12 . The camera 70 is inclined at a predetermined angle when compared to one side 400a of the first housing 400 . And the one side 400a is coupled to the lower surface of the ceiling of the inter-case 12 so as to be parallel to the contact surface 416 of the second housing 410 . Accordingly, as the camera 70 is coupled to the first housing 400 , and the first housing 400 is coupled to the second housing 420 , the installation angle of the camera 70 with respect to the ceiling surface of the storage room error can be significantly reduced.

Referring to FIG. 25 , the second housing 410 is disposed to be exposed to the space filled with the foaming liquid in the upper portion of the inner case 12 , that is, in the space between the inner case 12 and the outer case 10 . do. The second housing 410 is disposed outside the storage chamber 22 , but is disposed in a space partitioned by the outer case 10 . After the second housing 410 is attached to the inner case 12 and temporarily fixed in position, the second housing 410 is finally fixed by the foaming liquid filled between the inner case 12 and the outer case 10 . can

With the inner case 12 interposed therebetween, the first housing 400 may be coupled to the second housing 410 . Since the first housing 400 is coupled to the second housing 410 in a state in which the second housing 410 is fixed at a specific position of the inner case 12, the position of the camera 70 is can be fixed.

In a state in which the camera 70 is inclined to the first housing 400 at a specific angle, the third housing 420 and the fourth housing 430 may be coupled to the second housing 410 . . As described above, the third housing 420 and the fourth housing 430 are made of one component, and the first housing 400 is a component that performs the function of the third housing 420 . It is also possible to combine only

The third housing 420 may allow only the transparent window 80 of the camera 70 to be exposed to the storage chamber 22 , and may prevent other portions from being exposed. Accordingly, it is possible to prevent the camera 70 from being affected by moisture, dust, cold air, etc. existing in the storage chamber 22 .

The structure of the housing of the camera, the installation direction of the camera housing relative to the inner case, the installation form, etc. also have a technical meaning for reducing the condensation phenomenon occurring on the transparent window 80 . A related detailed description will be given later.

26 is a table comparing the power consumption of the camera in the present invention, and FIG. 27 is a diagram comparing the supplied standby current and the driving current. 26A shows a state in which current is supplied to the camera in the comparative example, and FIG. 27B shows a state in which current is supplied to the camera in the present invention. Hereinafter, it will be described with reference to FIGS. 26 and 27 .

In the comparative example, 70 mA of standby current is supplied to the camera 70 , and 50 mA is additionally supplied when the door is opened, so that a driving current of 120 mA is supplied to the camera 70 . In the comparative example, when the camera 70 is opened 25 times a day, 253Wh is consumed per month, and when the camera 70 is opened 50 times a day, 254Wh is consumed per month.

On the other hand, in the embodiment of the present invention, 20 mA of standby current is supplied to the camera 70 , and 100 mA is additionally supplied when the door is opened, so that 120 mA of driving current can be supplied to the camera 70 . When the camera 70 is opened 25 times a day, 74Wh is consumed per month, and when the camera 70 is opened 50 times a day, 76Wh is consumed per month.

Comparing the two experimental results, it can be seen that the power consumed by the camera 70 can be reduced by approximately 70% by lowering the standby current. Accordingly, it was confirmed that the power consumed by the camera 70 can be greatly reduced by lowering the standby current under the premise that the driving current is the same. This is because, when the refrigerator is operated continuously for 24 hours, the time during which the door is opened is relatively short. Accordingly, assuming that the current value required for shooting in the operation mode is constant, the current consumption can be reduced as the current value applied to the camera in the standby mode is lowered.

When the standby current is lowered, as shown in FIG. 27 , the time at which the driving current is supplied to the camera 70 is delayed. This is because, due to the difference between the standby current value and the driving current value, the delay time is inevitably increased. In other words, when the standby mode is switched to the driving mode, the picture is not taken immediately, but a picture is inevitably taken after a predetermined delay time has elapsed.

Accordingly, the time at which a picture is actually taken by the camera 70 is delayed. If there is a large difference between the time when the command to take a picture is transmitted to the camera 70 and the time when the picture is actually taken by the camera 70, the picture actually taken by the camera 70 preferably provides food information to the user. may not be able to provide.

In other words, the lower the standby current is, the lower the power consumed by the camera 70 is. Therefore, it is preferable to calculate the normal time elapsed for the user to be able to take a picture at a desired point in time, and select the standby current value so that the delay time is reduced rather than the corresponding time. For this reason, according to the embodiment of the present invention, it is preferable to increase the current applied step by step from the standby current value to the driving current value. That is, it is possible to increase the applied current value step by step from a relatively low standby current value to a relatively high driving current value with a sufficient delay time. That is, it is possible to prevent an abrupt increase in the current value. Therefore, power consumption can be reduced and camera stability can be achieved.

Accordingly, in the embodiment of the present invention, as shown in FIGS. 28 and 29 , the camera 70 is controlled to start driving when the door 20 is opened. That is, if it is determined that the door is opened through the door switch, it is switched from the standby mode to the driving mode. However, a valid photo among the photos taken by the camera is after at least a predetermined time has elapsed from the time the door is opened. This is because it takes physically more than a few seconds for the door to open and the user to update the food. In other words, even if the delay time is long, the picture can be acquired at the desired point in time. Therefore, in the present embodiment, it can be said that the switching time of the camera from the standby mode to the driving mode is performed according to the detection result of the door switch. And, it can be said that the reference time of selecting a valid photo is performed according to the detection result of the door sensor.

28 is a view for explaining a start point of the camera taking pictures and continuous shooting of the camera. Hereinafter, it will be described with reference to FIG. 28 .

Meanwhile, in the present invention, as shown in FIG. 4 , the door switch may be included. When the door switch 110 detects that the storage chamber 22 is opened, the controller 100 may command the camera 70 to start taking a picture. That is, when the door 20 is moved or rotated to open the storage chamber 22 , the camera 70 may start taking a picture.

On the other hand, when the door switch 110 detects that the storage chamber 22 is closed, the controller 100 may command the camera 70 to stop taking a picture. That is, when the door 20 is moved or rotated to seal the storage chamber 22 , the camera 70 may end photo taking.

As shown in FIG. 28 , when electricity is supplied to the refrigerator by an external power source, a standby current is supplied from the camera 70 ( S1 ). At this time, the camera 70 may be in a state of being driven in a standby mode. In the standby mode, the camera 70 does not take pictures.

Opening of the door 20 may be detected by the door switch 110 (S10).

When it is determined that the door 20 opens the storage chamber 22 , the camera 70 may start taking a picture ( S14 ). In particular, the camera 70 may continuously take pictures at predetermined time intervals.

At this time, since a larger current than the current supplied in the standby mode is supplied to the camera 70, the camera 70 may be operated in the driving mode. In order to supply a current greater than a standby current and a driving current to drive the camera 70 , a predetermined time period, that is, a delay time, is inevitably generated. That is, it is because a certain amount of time is actually required for the magnitude of the current supplied to the camera 70 to increase. After a delay time of a predetermined time has elapsed after the current for performing the driving mode is supplied to the camera 70, the driving current is supplied to the camera 70 so that the camera 70 can only take pictures. .

Since the camera 70 is switched to the driving mode for photographing from the time when the door 20 is opened, the start time at which a picture is taken is early, and thus the risk of not being able to take a picture at a specific time is reduced. This is because, if the standby current is supplied before the desired time point and the current value is increased at the desired time point, the time of taking the picture is delayed due to the delay time.

In addition, the number of pictures taken by the camera 70 per second can be reduced. This is because, if it is assumed that the camera 70 takes 30 pictures per second, the possibility that a change in the interior of the storage chamber 22 occurs during the time interval during which pictures are taken, that is, 1/30 second is extremely small. The inside of the storage chamber 22 can be changed by a user's action, because there is little possibility that a general person will finish an action for 1/30 second with respect to the storage chamber 22 . That is, the time interval for photographing by the camera 70 may be relatively increased. Accordingly, the relatively inexpensive camera 70 can be used, thereby reducing the cost of the refrigerator.

The photo taken by the camera 70 may be stored in the storage unit 18 (S16). In addition, if the number of pictures taken in one second by the camera 70 is reduced, the number of pictures stored in the storage unit 18 is also decreased, so that the storage unit 18 can be operated more efficiently.

Meanwhile, the door sensor 120 may detect whether the door 20 is rotated at a specific angle (S18).

If the door sensor 120 does not detect whether the door 20 is being rotated, the pictures stored in the storage unit 18 before a predetermined time may be deleted ( S90 ). That is, by deleting unprocessed photos to provide to the user, it is possible to prevent the storage unit 18 from increasing in capacity.

On the other hand, if the door sensor 120 determines that the door 20 has been rotated at a specific angle, a picture taken at that time is selected (S92). In this case, the selected photo may be processed by the controller 100 as an effective photo and provided to the user. For example, the photo processed by the controller 100 may be transmitted to the display 14 .

In order for a user to access the storage compartment 22 , a predetermined time is required to rotate the door 20 . Since the delay time of the camera 70 elapses during the corresponding time, a picture may be taken when the door 20 is rotated to a specific angle. Accordingly, the controller 100 may select a photo at a desired time.

If the camera 70 is switched from the standby mode to the driving mode to take a picture at the time the door sensor 120 detects it, there is a problem that the storage room state has to be taken after a delay time has elapsed from that point in time. can

Meanwhile, since the camera 70 starts shooting when the door 20 is opened, it is possible to supply a relatively small amount of standby current to the camera 70 . When a small amount of standby current is supplied, the delay time until the driving current is supplied increases. However, in the control flow described above, since the camera 70 starts shooting early, there is no problem that a picture cannot be taken at a desired time. .

29 is a view for explaining the start time of the camera taking pictures and the continuous shooting of the camera. Hereinafter, it will be described with reference to FIG. 29 .

Unlike FIG. 28 , in FIG. 29 , the photo selection time is determined according to the state of the drawer 50 , rather than the photo selection time being determined according to the rotational state of the door 20 . Other parts are the same, and the same parts are omitted for convenience of description. However, it can be said that the operation of the drawer 50 is premised on opening the door 20 . The reference time for selecting a specific photo may be determined by the door sensor's determination, and the reference time for selecting a photo for a specific area within the specific photo may be determined by the door sensor's determination and the drawer detection unit 130's determination. will be able Accordingly, in any case, the determination result of the door sensor is required in order to determine the reference viewpoint of the valid photo.

In order to store food in the drawer 50 or to take out the food stored in the drawer 50 to the outside, the user may withdraw the drawer 50 . In this case, the user may withdraw the drawer 50 by a sufficient length in order to use the drawer 50 .

After the user has finished using the drawer 50 , the drawer 50 may be pushed in. The time point at which this state occurs may mean the specific state described in S19.

Since the camera 70 starts shooting from the time when the door 20 is opened at the door switch 110, if the drawer 50 is placed in a specific state, the picture taken at that time is captured. It may be selected and provided to the user (S92).

When the door 20 is opened, a picture is taken by the camera 70 , that is, the movement of the drawer 50 can be monitored from a point in time at which the picture is taken, that is, when the door 20 is opened.

30 is a diagram illustrating an embodiment of a drawer sensing unit, and FIG. 31 is a diagram illustrating a method of detecting movement of a drawer in the drawer sensing unit of FIG. 30 . Hereinafter, it will be described with reference to FIGS. 30 and 31 .

Referring to FIG. 30 , the drawer sensing unit 130 is not provided in the control unit 100 , but may be implemented as a separate component.

The drawer sensing unit 130 may include one light emitting unit and two light receiving units. That is, the moving direction of the drawer 50 can be sensed by analyzing the timing at which the light irradiated from one light emitting unit is incident on the two light receiving units.

A plurality of slits 134 may be sequentially disposed in the drawer 50 to prevent light irradiated from the light emitting part of the drawer sensing unit 130 from being reflected when incident to the slits 134 . Accordingly, light may be sequentially incident on the two light receiving units.

In FIG. 31 , the x-axis means time, and the y-axis means a signal when light is received by the two light receiving units (Photo TR1 and Photo TR2). That is, in FIG. 31A, light is received before Photo TR2 in Photo TR1, and in FIG. 31B, light is received before Photo TR2 in Photo TR2.

In FIG. 31B , the user withdraws the drawer 50 to use the drawer 50 , and in FIG. 31A , the user ends the use of the drawer 50 and pulls the drawer 50 into its original position. It can mean status.

Therefore, when the drawer detection unit 130 detects a signal as shown in FIG. 31A , the control unit 100 determines that it is time to acquire a picture taken by the camera 70 , and at that time the drawer 50 It is possible to select a picture corresponding to the inside of the , that is, the third area. That is, the acquisition time of the photo corresponding to the third region may be determined according to the detection or determination result of the drawer sensing unit 130 . Of course, in the obtained photo, the part corresponding to the third region may be selected, separated, and updated independently and separately from the part corresponding to the other region.

32 is a diagram illustrating a state in which a marker is displayed on a drawer. Hereinafter, it will be described with reference to FIG. 32 .

Unlike FIGS. 30 and 31 , the movement of the drawer 50 can be detected by using a marker that is a specific mark on the drawer 50 without using a separate mechanical or electronic device. .

Various types of markers (patterns having black and white colors) may be provided on the drawer 50 . The camera 70 may analyze the movement of the marker through the photographed picture to read whether or not the drawer 50 moves and the direction of movement. At this time, it is possible to analyze the marker by the drawer detection unit 130 of the control unit 100 . That is, the drawer detection unit 130 may determine the state of the drawer 50 by analyzing the movement of the marker in the pictures taken continuously. Accordingly, the acquisition time of the photo corresponding to the third region may be determined according to the detection or determination result of the drapery detection unit 130 . Of course, in the obtained photo, the part corresponding to the third region may be selected, separated, and updated independently and separately from the part corresponding to the other region.

In FIG. 32, the marker includes an image of a simplified camera shape and the word 'smart sensor'. On the other hand, the marker may include various types of patterns, and may be provided in a form in which black and white intersect each other, so that it can be recognized by the camera 70 .

In this case, the marker may be displayed on the front upper side of the drawer 50 so that the camera 70 can recognize it.

Fig. 33 (a) is a view showing a state in which the left and right drawers are drawn out, and Fig. 33 (b) is a view showing a state in which the left drawer is retracted and the right drawer is drawn out. Hereinafter, it will be described with reference to Figs. 33 (a) and 33 (b).

In FIG. 33 , the marker is provided in the center of the upper side of the drawer 50, and an example of forming a wave-shaped pattern is shown.

The first area 42 may be expressed in one picture taken by the camera 70 . Also, one of the second area 52 and the third area 62 may be shown in one picture.

That is, the first area 42 disposed on the left and right and the second area 52 disposed on the left and right are shown in the photo taken with the left and right drawers drawn out as in FIG. 33A .

On the other hand, in the picture taken with only the right drawer drawn out as in FIG. 33(b), the first area 42 disposed on the left and the third area 62 on the left and the second area 52 on the right side ) can be shown.

That is, when the drawer 50 is withdrawn, the third area 62 is not shown in the photo, but the second area is shown. On the other hand, when the drawer 50 is drawn in, the second area 52 is not shown in the photo, but the third area 62 is shown.

The camera 70 takes pictures at the same location. The drawer 50 may be moved, and thus, the picture obtained from the camera 70 may take pictures of various areas according to the position of the drawer 50 .

Meanwhile, as shown in FIG. 33(b) , when the drawer 50 provided on the left is completely drawn into the lower space of the shelf 40 , the marker is not recognized by the camera 70 . That is, the marker is not shown in the picture taken by the camera 70 .

A method of detecting movement of a drawer using a marker will be described with reference to FIGS. 34 to 40 .

34 is a flowchart illustrating a method of operating a refrigerator according to an embodiment of the present invention.

Referring to FIG. 34 , when the door switch 110 detects that the door 20 is opened, the camera 70 starts taking a picture (S2101), and the control unit 100 performs a marker recognition operation and marker tracking. The operation starts (S2103).

Hereinafter, steps S2101 and S2103 will be described in more detail with reference to FIG. 35 .

35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.

Referring to FIG. 35 , when the door 20 is opened, the camera 70 starts taking a picture ( S2301 ).

The controller 100 controls the camera 70 to obtain an internal picture of the drawer 50 (S2303).

Next, the controller 100 processes the acquired picture (S2305), and determines whether a predetermined marker is recognized in the acquired picture (S2307).

In order to determine whether a predetermined marker is recognized, the controller 100 may compare whether a marker recognized from the acquired photo is the same as a previously stored marker.

The predetermined marker may be determined by a business operator or a user, and may be stored in the storage unit 18 or the like in advance.

The marker may be formed in a searchable area in a photo taken by the camera. Hereinafter, a marker that can be recognized and tracked by the controller 100 will be described in detail with reference to FIGS. 36 and 37 .

36 is a view for explaining a marker for detecting the withdrawal/input timing of the drawer according to an embodiment of the present invention.

Referring to (a) of FIG. 36 , a marker may be formed in an area searchable by the camera 70 from a point in time when the drawer 50 starts to open. For example, when the camera 70 is positioned at the upper end of the refrigerator to photograph the plane of the drawer 50 , the marker may be positioned at the front end of the upper end of the drawer 50 . In addition, the field of view of the camera 70 may be installed so as to be able to photograph from a state in which the drawer 50 is completely closed to a state in which the drawer 50 is completely open.

In the case where the marker is located at the top of the drawer 50, there is no obstruction at the top of the drawer 50 at the time the drawer 50 is closed after the storage of the inside of the drawer 50 is completed. planned, but not limited thereto.

Referring to FIG. 36 (b), when looking at the drawer compartment 54 and the drawer 50 in which the drawer 50 is accommodated from the viewpoint of the camera 70, the camera 70 is the drawer ( In all ranges from the state in which the 50 is started to open to the state in which the drawer 50 is fully opened, the inside of the drawer 50 including the marker may be photographed.

37 is a view for explaining the shape of a marker according to an embodiment of the present invention.

Referring to FIG. 37 , the marker may be formed in a form in which the same shape of two colors intersects and repeats.

Referring to (a) of FIG. 37 , the marker may be formed in a form in which a plurality of bands in which a white square and a black square are alternately repeated are intersected and arranged.

Referring to (b) of FIG. 37 , the marker may also be formed in a form in which a plurality of bands in which a white slanted quadrangle and a black slanted quadrangle are alternately repeated cross each other.

As described above, the repeated marker has an advantage that the control unit 100 can recognize and track the marker even when a part of the marker is hidden from the camera shooting range. However, the marker is not limited to a repeating form, nor is it limited to a plurality of colors. As described above, the marker suffices to have a shape sufficient to recognize whether the internal components of the refrigerator with mobility, such as the drawer 50, move or not.

Again referring to FIG. 35 , when a predetermined marker is recognized, the controller 100 detects movement of the recognized marker ( S2311 ). The controller 100 may continuously or periodically analyze the acquired picture to determine whether the recognized marker moves or stops. For example, when the opening operation of the drawer 50 is continued or completed, or the closing operation of the drawer 50 is started, continued, or completed, the control unit 100 may determine whether the marker moves or stops. .

Alternatively, when the predetermined marker is not recognized, the controller 100 notifies the user that the marker recognition has failed (S2309).

For example, when an obstacle is located in the marker, the camera may not be able to acquire a picture including the marker, or the controller 100 may not be able to recognize the marker from the acquired picture. In this case, the controller 100 may notify the user through the display 14 or the like of information indicating that marker recognition has failed, information indicating a method for successfully recognizing the marker, such as removing an obstacle.

Then, when the recognition of the marker fails, the controller 100 may repeatedly perform the step of recognizing a predetermined marker by processing a newly acquired picture through the camera.

Referring back to FIG. 34 , after marker recognition, when a drawer closing operation is detected (S2105), the control unit 100 includes the inside of the drawer (the third area 62) through the picture taken at that time. A final picture is acquired (S2107). At this time, in the final picture of the inside of the drawer obtained by the controller 100, the inside of the drawer just before the drawer closing operation is sensed or the inside of the drawer immediately after the drawer closing operation is sensed may be, The present invention is not limited thereto.

Whether the drawer closing operation is sensed will be described with reference to FIG. 38 .

38 is a flowchart illustrating a method for detecting a closing operation of a drawer according to an embodiment of the present invention.

Referring to FIG. 38 , the control unit 100 determines whether the end of the opening operation of the drawer 50 is detected ( S2501 ). For example, the control unit 100 recognizes the marker from the previously acquired picture of the drawer 50, tracks the movement of the marker, and when it detects that the movement of the marker has stopped, the opening operation of the drawer 50 is You can detect that it has ended.

Next, the control unit 100 determines whether the start of the closing operation of the drawer 50 is detected (S2503). For example, when the marker stops moving and starts moving again in the opposite direction, the controller 100 may sense that the closing operation of the drawer 50 has started.

Meanwhile, an operation of the control unit 100 that senses a drawer closing operation related to another embodiment of the present invention will be described with reference to FIG. 39 .

39 is a flowchart illustrating an operation of the refrigerator performed when the completion of a drawer closing operation is detected according to an embodiment of the present invention.

When the end of the drawer closing operation is sensed (S2701), the controller 100 determines whether the final photo acquisition through marker recognition is successful (S2703).

The controller 100 may detect that the closing operation of the drawer 50 has ended through image processing of the drawer detection unit 130 .

The controller 100 notifies the user that the marker recognition has failed when the final picture fails to be acquired as a result of the marker not being recognized normally due to an obstacle or the like (S2705).

The controller 100 may notify the user of the failure to recognize the marker through the display 14 or the like. This guides the user to remove the obstruction, and may induce the drawer to be pulled out for normal marker recognition and final image acquisition.

Referring back to FIG. 34 , the controller 100 displays the final image using the display 14 or the like ( S2109 ). In this case, the form provided to the user may be a portion extracted from only the portion corresponding to the third area 62 from the obtained final photograph. Next, with reference to FIG. 40, the final image control according to another embodiment of the present invention will be described.

40 is a flowchart for explaining a method of controlling an image of a specific area inside a refrigerator captured at a specific point in time according to an embodiment of the present invention.

Referring to FIG. 40 , the controller 100 may obtain a final picture ( S2901 ), and crop an image corresponding to a desired area from among the final picture. Then, the obtained final image is stored in the storage unit 18 (S2903).

Next, the controller 100 determines whether or not a user input for displaying a final image is received through a user input or the like (S2905).

For example, when acquiring the final picture, the control unit 100 may form a user input unit in the form of an icon on the display 14 composed of a touch screen along with information notifying the success of the final picture acquisition, and depending on whether the icon is selected, the user It can be determined whether the input is received. The user input unit may include a physical key as described above in addition to the icon formed on the touch screen, but is not limited thereto.

Next, when receiving a user input for displaying the final image processed from the final photo, the controller 100 displays the final image (S2907). The controller 100 may display the final image using the display 14 . The display of such a final image can be said to be displayed by replacing the existing image. Therefore, it can be called an update of the final image.

Of course, the updating of the final image may be performed automatically regardless of a user's input.

According to this embodiment of the present invention, the refrigerator may detect whether the drawer or the like moves by recognizing a marker previously formed on the drawer or the like. In addition, the refrigerator may acquire the final state of the storage stored in the drawer or the like in the form of an image using a camera installed therein. In this case, the refrigerator may determine an optimal time point at which the final storage state of the drawer or the like can be obtained by using a marker formed on the drawer or the like.

Hereinafter, the installation position of the camera and the pictures obtainable accordingly will be described.

In the embodiment of the present invention, a camera 70 is installed outside the drawer 50, that is, in the storage compartment 22, and the drawer body (ie, the drawer 50) is formed inside the drawer 50 and the drawer 50 is formed. By continuously taking pictures including the movement path of the front and rear side walls, the upper part of both walls, etc.), it is possible to simultaneously recognize the state of storage of items inside the drawer as well as the state of movement of the drawer. Of course, the picture taken by the camera 70 is taken together with the upper space of the drawer 50 , that is, a portion corresponding to the first area 42 .

Basically, the picture taken by the camera should be able to include both the opening/closing action of the drawer and the path the drawer moves. Therefore, the camera must be installed at a position where the opening/closing and movement path of the drawer can be photographed from the outside of the drawer. That is, it is preferable that the camera be installed at a position where at least the drawer can take both the closed state and the fully open state as one picture.

Furthermore, the camera 70 includes not only the drawer area but also another storage area of the storage room, for example, a shelf area where food or articles are stored by being divided by a shelf to be included in one picture with one shot. Thus, one camera can be configured to cover another storage area including the drawer area, that is, a plurality of storage areas.

To this end, the drawer 50 is preferably formed on the bottom surface of the refrigerator storage compartment. The drawer 50 may include a front wall, two side walls, a rear wall, a bottom wall, and an upper opening. The camera 70 should be provided on the upper part of the drawer so that at least a part of the inside of the drawer and the upper part of the front wall of the drawer in a state in which the drawer is fully opened can be exposed with respect to the drawer 50 .

And, a printing part (Marker) made of a specific pattern is formed or attached to any position on the upper surface of the front wall or both side walls of the drawer exposed to the camera.

At this time, since the camera 70 is fixed at a predetermined position, it is necessary to configure the position of the printing part of the drawer 50 to be relatively changed. That is, when the camera 70 receives a driving command from the controller, it continuously shoots at a constant speed, and sequentially transmits the captured images to the controller 100 . The control unit 100 tracks the position of the printing unit from the photo related to the storage of items in the refrigerator received from the camera 70 to determine information about the drawer movement as well as the open state of the drawer.

The range in which the camera can photograph may be determined by the angle of view of the camera, and it includes a vertical angle of view indicating a photographing range in the vertical direction and a horizontal angle of view determining a photographing range in the horizontal direction.

One end of the vertical angle of view of the camera 70 should include at least the front wall of the drawer when the drawer is fully opened, and the other end is located on the front wall or both sides of the drawer in the closed state. It should be decided to include up to a portion where it can be confirmed that the part is not exposed, for example, up to a part of the front surface of the upper side of the drawer that covers the opening of the upper part of the drawer.

At this time, it is possible to simultaneously recognize the drawer internal area and another storage space with one camera. Another embodiment of the present invention provides a separate storage space (hereinafter, referred to as "a separate storage space" on the bottom surface of the storage compartment where the drawer is partially overlapped in the open state between the front wall of the storage compartment and the front wall of the storage compartment when the drawer is closed. storage compartment floor) can be configured. In this case, the vertical angle of view of the camera 70 is configured to recognize another storage space (floor storage compartment) including the drawer inner area even in the shooting range including the drawer fully open and drawer fully closed states. may be

Meanwhile, according to another embodiment of the present invention for the vertical angle of view of the camera 70 , the drawer inner area and the shelf area may be photographed simultaneously. Therefore, it is possible to determine the range of the other end of the vertical angle of view so as to include up to any one of the plurality of shelves arranged at regular intervals up and down. In making the camera vertical angle to include the shelf area, a preferred example may include up to a part of the topmost shelf among the plurality of shelves, and more preferably, at least the top shelf disposed at the top of the storage compartment. to include the rear end.

According to these embodiments, it is possible to check the storage state of the items in at least two storage spaces including the drawer inner area by one camera with one picture taken with one camera.

To this end, a preferred installation position of the camera is required for the drawer formed at the bottom of the storage compartment. First, the inventors of the present invention reviewed the camera installation position disclosed or known through the published data at the time of filing the present invention. It was examined whether it was possible to photograph the inside of the storage room by installing it on the top of the door outside the refrigerator storage room.

When the camera is installed outside the storage room (door), there is an advantage that a separate anti-condensation structure is unnecessary because the camera is always exposed to the outside temperature so that no dew condensation is formed around the camera. However, it was difficult to photograph the inside of the storage chamber with a camera mounted on the door because the operation of the user to store the goods in the storage space inside the drawer and the storage chamber is always performed with the door open. In particular, an object of the present invention is to obtain a state of storage of articles inside the drawer as well as the state of movement of the drawer from the outside of the drawer. However, the user's operation in the drawer is always in an open state, and when the door is closed, the drawer is also in a closed state, which does not match the meaning of the present invention at all.

Therefore, in order to photograph at least the inside of the drawer and the movement state of the drawer, which is one aspect of the present invention, it is preferable to arrange the storage room so that the storage room can be photographed in the storage room of the main body where the drawer to be photographed exists, not outside the refrigerator. do.

On the other hand, in accordance with one aspect of the present invention, when the camera is installed with the camera direction facing the opening side of the main body storage compartment from the lower part of any one of the shelves located on the upper part of the drawer, it is impossible to photograph the shelf area first. It is difficult to install and wire the drive unit that can drive the camera on the shelf, and since a wide-angle camera must be used to capture all the movement paths of the drawer, there may be a problem in that the distortion phenomenon in the captured image increases.

The inventors of the present invention found that when the camera is installed on the side wall of the storage room in the range where the drawer can be exposed, distortion is in the image showing the storage state of the goods inside the drawer due to an asymmetric image due to the difference in distance from the camera in the left and right directions of the refrigerator storage room I knew it was serious. Therefore, it can be seen that the satisfaction of these users can be greatly reduced.

Above all, the purpose of the present invention is to detect the open/closed state and the extent of the drawer by tracking the movement state of the drawer without a separate sensor. The marker may be photographed in a relatively enlarged state (image distortion). Accordingly, the size (pixel size) occupied by the marker portion increases with respect to the entire size constituting one image, so that the position change may not be easily revealed. In addition, since the image processing time may increase due to an increase in the range to be processed to determine whether the marker position has changed, continuously transmitted images may be missed or unable to be processed.

Therefore, it was found that installing the camera on the side wall or corner of the storage room, which was already known at the time of filing of the present invention, does not correspond to the purpose of the present invention.

In one embodiment of the present invention, the camera is installed on the upper wall of the storage chamber corresponding to the upper opening surface of the drawer, and the closed state of the drawer and the fully open state of the drawer are projected on the upper surface of the storage chamber. Installed within the range, the direction of the camera toward the inside of the storage chamber toward the rear wall side of the storage chamber, the focus of the camera was installed inclined at a certain angle to face any one point on the rear wall of the storage chamber. Accordingly, it was possible to obtain an image in which distortion was minimized in the embodiments with respect to the vertical angle of the camera of the present invention described above.

In determining the position of the camera, another embodiment of the present invention is installed on the upper wall of the refrigerator storage compartment, and the front end of the uppermost shelf among the shelves disposed vertically in the storage compartment at the front opening of the storage compartment is on the upper wall of the storage compartment. It is placed between the projected positions. In addition, the direction of the camera is directed toward the inside of the storage chamber toward the rear wall of the storage chamber, and the focus of the camera is installed at a predetermined angle to face any one point on the rear wall of the storage chamber.

The camera is preferably located in the central portion of the upper portion of the storage chamber, and the printing units located on the upper surface of the front wall or both sides of the drawer to correspond to this are preferably installed on the upper surface of the front wall of the drawer. More preferably, the vicinity of the upper surface of the front wall of the drawer corresponding downward to the position of the camera installed on the upper surface of the storage chamber is proposed.

In a refrigerator in which the opening of one storage compartment is opened and closed by one or more doors, that is, the left and right doors, the camera is installed on the upper surface (ceiling surface) of the storage compartment at a position that becomes the boundary between the left and right doors, In this case, the printing part is located near the left and right borders. That is, from the viewpoint of viewing the inside of the storage compartment, it is preferable that the left drawer is located on the right upper surface of the front wall or the upper end of the right side wall, and the right drawer is located on the left upper surface of the front wall or the upper end of the left side wall. In other words, it is preferable that the printing parts are formed at a position close to the vertical lower part of the camera. This is because, as the distance from the vertical lower part to the left and right increases, the possibility of distortion of the printing part increases. An example of this is shown in FIG. 41B .

At this time, when the printing parts are installed on both sides of the upper surface of the front wall of the left and right drawers or the upper ends of both sides, the drawers used in the storage room that are opened and closed by the left and right doors can be used in common regardless of the left and right positions. can increase efficiency.

As described above, according to the camera position and the print unit position according to the present invention, all of the movement paths from the closed state to the open state of the drawer are included within the shooting range of the fixed camera. It is possible to detect not only the open/closed state of the drawer, but also the degree of opening and closing, and the stop or movement of the drawer, and at the same time, the storage status of the items inside the drawer can be detected or checked by a single camera.

In addition, since the storage state of items in the floor storage compartment installed on the floor of the storage room other than the inner region of the drawer can be checked when the drawer is closed, a plurality of storage regions can be recognized with one camera.

In addition, when the angle of view of the camera is extended to the shelf area, three storage areas can be selectively recognized with one camera, including the drawer inner area as well as the storage room floor storage compartment and the front area.

On the other hand, since the camera installed on the upper surface of the storage chamber and the printing part formed or attached to the upper part of the drawer are arranged in an approximately line up and down, the printing part is part of the entire image of the cabinet taken and transmitted by the camera. There is almost no distortion, and the space occupied by the printing part can be relatively minimized, so it is possible to minimize the time for tracking and judging the position of the printing part through image processing.

Hereinafter, a method for recognizing a marker formed in the drawer will be described.

A printing part (marker or printing paper) that can be recognized by image reading can be installed on a specific part of the drawer body exposed to the camera. By tracking the position of the marker, it is possible to detect the degree of opening/closing of the drawer, and the state of opening and closing. The marker part is configured to be exposed to the camera from the closed position to the fully open position on the movement path of the drawer, and the opening/closing state of the drawer can be determined by the position of the marker part. That is, various state information of the drawer can be determined through a change in the position of the marker in the continuously taken pictures.

As described above, the structure of the printing unit (Marker) may be formed of a specific pattern in order to track and recognize the movement or opening/closing state of the drawer. In this case, the printing unit may be disposed at various positions on the upper surface of the drawer exposed to the camera.

Preferably, the printing unit is installed at a portion that can be the boundary of the maximum opening portion of the drawer at the upper end of the front wall of the drawer or at the upper end of both sides of the wall of the drawer.

The pattern shape of the printing unit may be formed as a pattern in which patterns having a predetermined shape of a certain size are repeated in contrasting colors, and the shape may be formed in various shapes such as a square, a triangle, a circle, and a wavy shape.

Such a printing unit may have a risk of deteriorating the overall aesthetics of the refrigerator, and thus may have a rejection feeling depending on the user. Accordingly, it is necessary to minimize the repetition of the pattern so that there is no objection to the aesthetic sense felt by the user, and furthermore, a printing unit having a form that can appeal to the user can be considered.

However, this design limitation is not limited to a simple design problem, and a case in which the position of the drawer cannot be accurately recognized occurs. That is, it affects the recognition rate of the printing part of the drawer, and ultimately it is possible to result in not being able to accurately detect the position of the drawer.

That is, in consideration of the design element that needs to significantly reduce the size of the printing part, it is not possible to repeat the pattern several times with a contrasting color with a pattern of a certain shape, but it is possible to select the degree of repeating only 2 to 4 times. In this case, the control unit that processes the image captured inside the refrigerator may not be able to accurately recognize the boundary of the printing unit (marker) in the entire image.

Therefore, according to a preferred embodiment of the present invention for solving this problem, it is possible to select a contrasting color because the drawer is usually made of a bright color or transparent or translucent. Referring to FIG. 41A , the length L1 in the vertical direction with respect to the moving direction (back and forth) of the drawer may be set longer than the length L2 in the moving direction of the drawer. And forming a first color part having a certain length in the vertical direction, the color contrasting with the first color part or the same color as the drawer with about the same thickness in front or rear of the first color part with respect to the moving direction of the drawer A second color part made of a is formed, and is continuously formed in the same color as the first color part and repeatedly formed to have the same thickness and the same length. That is, it is possible to sequentially arrange the first color part and the second color part with each other. In this case, it is more preferable that the length L1 in the vertical direction is larger than the length L2 in the drawer direction in the outer size of the markers arranged as the first color part, the second color part, and the first color part.

According to another form of the marker of the present invention, both ends of the first color part at the front and rear ends are connected to each other and surround the second color part as shown in FIG. 41B. The third embodiment in the form of a marker shown above may be an example showing such a structure.

That is, in the preferred embodiment of the marker presented in the present invention, the length (L1) in the vertical direction with respect to the movement direction of the drawer is larger than the length (L2) in the movement direction of the drawer, but the portion bordering the drawer (especially the drawer) The color (dark color) that contrasts with the drawer is formed in the lower moving direction), and the bright color part that contrasts with the dark color is included in the middle.

When the printing unit is configured in this way, since it can be configured in a small size, not only can the user's requirements for aesthetics be satisfied, but also the size of the printing unit is relatively small in one whole image captured by the camera. Accordingly, the image processing area for detecting a change in the position of the printing unit is reduced by that much, so that the image processing speed can be improved. This advantageously works to match the image processing speed of the control unit to the image capturing speed that is continuously captured and transmitted, so that the accuracy of obtaining an image at an accurate time is further improved. For example, if the size of the portion to be processed in the pictures taken continuously is large, a load of the control unit is high. Therefore, it is necessary to use a very expensive control unit. However, if the size of the part to be processed is small, the load on the control unit can be reduced. Accordingly, it is possible to effectively process an image even if a low-cost control unit is used.

On the other hand, as shown in FIG. 41B , the marker may preferably include a character, a symbol, or a pattern so that the user can intuitively grasp the purpose of the marker by looking at the marker.

Hereinafter, the marker coordinate recognition of the drawer and the camera image processing method will be described.

42 is a view for explaining a method for recognizing a marker position. Hereinafter, a method for recognizing the marker position of the drawer will be described with reference to FIG. 42 .

When the user wants to store (wear) or release an item inside the drawer, the user first opens the door. When the door is separated from the front of the storage room, the door switch detects it, and the control unit receives a signal from the door switch and drives the camera in standby installed in the storage room. When the camera receiving the driving command from the controller applies the camera driving voltage instead of the standby voltage state, the camera is ready to drive and continuous shooting starts.

The user will then try to open the drawer by grabbing the handle on the front wall of the drawer installed in the refrigerator storage compartment and pulling the drawer. At this time, since the camera is continuously shooting, the image is sent to the control unit by shooting the open state of the drawer at regular intervals.

On the other hand, a marker of a certain type is formed in a portion of the drawer exposed to the camera, and the control unit receiving the image sent from the camera divides the entire image into predetermined pixels and assigns coordinate values at regular intervals, The coordinates of the marker on the coordinates of the image are analyzed.

Accordingly, when an image captured while the drawer is closed is sent to the control unit, the corresponding image marker portion will not be exposed, and the marker coordinates at this time will have a coordinate of 0.

When the user opens the closed drawer, the camera takes a picture with the marker exposed, and the camera can take a picture including the marker. When this image is sent to the control unit, the control unit recognizes the coordinate values of the markers on the entire image and compares them with the previous position. For example, when the coordinate value of the marker at point 'A' in FIG. 42 is changed from '0' to '50' as an example of a constant coordinate value, the controller can determine that the drawer is moving from the changed coordinate value. have. In particular, when it is changed from '0', it can be determined that the drawer has just begun to open.

If the user continues to open the drawer, the control unit may determine that the drawer is being opened because the drawer is moving and the coordinate value changes to a larger value through a series of processes as described above.

When the user opens the drawer to the fully opened state, the control unit recognizes that the coordinate value is no longer changed and determines that the drawer is fully opened or that the drawer is open and stopped. For example, the point of 'B' in FIG. 42 is a point where the coordinate value of the marker does not change from '90' to '90', so it is recognized as 'the drawer is in a stopped state' or as 'the drawer is completely open' can judge

In general, after the drawer is opened, regardless of whether the drawer is fully opened or partially opened, if the drawer is in a stopped state, it may be determined that the user is performing a certain operation on the drawer. In this way, even when the drawer is stopped, the camera continues to take pictures of the inside of the refrigerator and sends it to the control unit, and the control unit monitors whether there is a change in the coordinates of the marker.

When the user completes the act of putting the item into the drawer or taking the item out of the drawer, the user closes the drawer again. It is determined that the drawer is moving again. For example, point 'C' in FIG. 42 is a point where the coordinate value of the marker changes from '90' to '70', and the drawer has just started closing from the change in the direction in which the size of the coordinate value is reduced. It can be determined immediately after the user completes the operation on the drawer.

In the same way, when the coordinate value of the marker is '0', that is, when the marker is not recognized from the image, it is determined that the drawer is completely closed.

From the video taken by the camera, the control unit grasps the marker position according to the coordinates of one entire picture, and whether the drawer moves and stops, the close and open of the drawer, the degree of opening of the drawer, the drawer starts to open, The method for tracking and judging whether the drawer starts to close has been described.

Hereinafter, camera image processing and a desired image acquisition method according to the marker position of the drawer will be described. A detailed method of how to process and obtain an internal image of the drawer from the moving state of the drawer, in particular, a method of obtaining an image at a point in time determined immediately after the user completes the operation on the drawer will be described.

When the marker coordinates of the drawer change to '0' or more and it is determined that the process is moving in the direction in which the drawer is opened, the control unit detects only the coordinate change for the in-house image captured by the camera and sent to the control unit, and displays the entire picture. Buffered in a temporary buffer and no processing is performed for each storage area partitioned in the photo.

When the drawer is continuously opened to reach a fully open state or is opened to a certain extent and is in a stopped state, a situation in which the user performs an operation on the drawer proceeds. The control unit determines this state from the fact that the coordinates of the marker do not change, and from this time on, for the transmitted image, the image corresponding to the drawer area from the storage area partitioned from the entire image is moved to the drawer temporary buffer and stored. do. That is, in the section corresponding to point 'D' in the figure, the control unit cuts out the necessary part from the part partitioned by the drawer area among the entire picture inside the store and moves it to a temporary buffer for the drawer area, which is called the 'drawer internal image refresh section'. ' is defined as

In other words, the image processing method is different in the section in which the drawer is opened and the section in which the drawer is in a stationary state. The reason is that the user's action on the drawer reflects that the drawer will be in a stationary state. As a result, in the section in which the drawer moves, the image processing time for tracking the coordinate change can be increased that much, thereby increasing the efficiency of data processing.

On the other hand, when the drawer starts to close again from a stop state, the control unit determines that the user completes the task immediately from the coordinate value changing to a smaller value, and at that point in the temporary buffer for the image inside the drawer Among the temporarily buffered images, the most recent image is selected and saved. That is, the image of the inner area of the drawer is captured immediately after the user completes the operation.

The captured (stored) image of the item storage status of the most recent drawer area replaces the image of the corresponding area of the display connected online or directly to the control unit, or is transmitted to the network server. In addition, the latest image transmitted to and stored in the network server is provided so that the user can check it through a display installed in the refrigerator or a mobile terminal connected to the network server whenever a user wants.

Next, a description will be given of a method in which the drawer selects a specific section and searches for the position of the marker, rather than searching for the position of the marker for the entire photographed picture.

That is, it is possible to set a search range for the drawer area. On the movement path of the drawer, the degree of opening of the drawer is divided by section, so that the method of tracking the marker in each section can be different. Thereby, it is possible to shorten the processing (analysis) time of the picture taken and transmitted by the camera, and to match the image processing speed and the camera shooting speed.

In general, in the drawer being used in the refrigerator, even when the drawer is fully opened, the entire drawer cannot be withdrawn to such an extent that the drawer is separated from the storage compartment and falls to the outside. That is, it is common that about 50% of the drawer's total depth (the length before and after the drawer in the direction of movement of the drawer) is drawn out at the distance that the drawer can normally be drawn out, the normal opening distance.

The drawer pull-out distance may be determined as an area to which coordinates of the drawer marker should be assigned in the entire image processed by the controller. In this case, the control unit may determine the position of the drawer from the time the drawer is withdrawn. In particular, in order to detect a change in the marker coordinates of the drawer, the size corresponding to the drawer withdrawal distance becomes the data size to be processed by the controller. In addition, in order to detect a change in coordinates, the controller has to compare and determine two images at the same time, and the processing time of this process may be burdened with the ability to respond to the speed of photographing by the camera.

Preferably, the shooting speed of the camera is determined so that the speed at which the camera takes pictures (time interval at which pictures are taken) roughly matches the speed at which the controller determines and processes the image from the transmitted image.

If this method is followed, the position of the marker can be grasped without delay in the picture taken by the camera, and the position of the drawer can be finally determined. In addition, while the control unit processes the images for each area partitioned for each storage space in the entire picture of the interior, there is no difficulty in processing the new images continuously transmitted from the camera into images. In other words, depending on whether the door is open or the drawer is open from the entire image, the image for each area is cut and moved to a temporary buffer for each area. All images captured by the camera can be processed without missing will be done

However, since data for two images (previous image and current image) of the same size as described above must be processed, the processing power of the control unit needs to be increased. have. In addition, for image processing when the drawer is not opened, the processing power becomes too large, which causes a problem in that efficient part selection is not made.

In order to solve this problem, the inventors of the present invention have focused on the fact that it is practically difficult to sufficiently grasp the inside of the drawer when the withdrawal distance of the drawer is not completely open, that is, in a state in which the drawer is opened by a certain distance, A method of different data processing is provided by separating the images corresponding to the drawer area step-by-step according to the draw distance of the drawer.

43 shows the degree to which the drawer is opened. Figure 43a is a state in which the drawer is maximally opened, that is, about 50% of the entire length of the drawer is opened, Figure 43b is that the drawer is opened about 30% of the total length, and Figure 43c is that the drawer is full About 10% of the length is open.

For example, in a state in which the user opens the drawer by only about 10% of the total length of the drawer in the front and rear directions as in FIG. 43c, it becomes to the extent that the item stored inside the drawer cannot be sufficiently grasped, and in FIG. 43b about It is only when about 30% of the drawer is opened that the storage status of the items inside the drawer can be grasped to some extent, and this condition is enough to recognize the drawer as an open state.

As shown in FIG. 43A , when the drawer is maximally opened, about 50% of the entire drawer is withdrawn. In other words, when the open state of about 30% to 50% is reached, the user can store or take out the article with respect to the drawer.

In view of this, it is determined that the drawer withdrawal distance is within 30% of the 'drawer not open state', and 30% or more is divided into the 'drawer open state' by differentiating the image processing, so that the controller is It can significantly reduce the data load to be processed.

The control unit determines the image size corresponding to the movement trajectory of the drawer by imaging the image taken by the camera, and determines the pixel size to the point or distance at which the drawer is drawn out by about 30% from the determined drawer movement area to determine the image section compartmentalize

That is, when the drawer is withdrawn from the range corresponding to FIGS. 43A to 43B, the control unit may determine the movement of the coordinates of the marker. In this case, the load on the control unit required is reduced compared to the case of determining the movement of the coordinates of the marker in the range corresponding to FIGS. 43A to 43C.

In the section where the drawer is drawn out by 30% while the drawer is closed, only whether the position of the marker has reached the position of the pixel by the distance corresponding to 30% is determined. In other words, it does not compare the marker coordinates of the previous picture with the marker coordinates of the current picture (a prerequisite for determining the position change of the marker), it simply determines whether the marker position is within a certain pixel range and temporarily buffers the entire image. will do

In this section, since only one photo data is processed without the need to compare the two photo data, the data processing capability of the control unit does not need to be large, and it is possible to smoothly respond to the camera shooting speed.

On the other hand, when the drawer is opened more than 30%, that is, when the marker reaches a position of a specific pixel or more in the photo processed by the control unit, two photos (taken just before The coordinate value of the marker can be determined by comparing the taken picture with the picture taken after that).

When the search section is limited in this way, the size of the photo is about 2/5 of that of the non-limited search section. Furthermore, since the size of the photo corresponding to the two photos to be processed is about 4/5 of that of one photo in the case where it is not, the data to be processed by the control unit is smaller or substantially the same as compared to the entire photo. level can be

Therefore, since the data processing speed for tracking the coordinate change value of two photos can correspond to the speed at which the camera shoots, data processing (coordinate tracking or monitoring) is possible without missing any images captured by the camera. It is possible to prevent a time delay in processing speed even when the processing capacity of the processor is no longer increased, so that the desired image can be acquired at the correct time.

In conclusion, as shown in FIG. 44 , it is possible to divide a portion of a distance that the drawer can withdraw from a picture taken with a camera into a first section and a second section.

In the first section, it is possible to determine the position of the marker with respect to the pixel of the photographed picture. In this case, in the first section, it is possible to determine at which pixel position the marker is in one picture without comparing the coordinate values of the marker.

Meanwhile, in the first section, even if the drawer is actually opened, it is also possible to classify that the drawer is in a substantially closed state because it is difficult for a user to withdraw or bring an article into the drawer.

The second section may be set as a 'coordinate search range' that compares coordinate values of markers shown in two pictures taken continuously. In the second section, since two photos are compared, a change in the coordinates of the marker can be detected. For example, if the marker is at the same position in two consecutive photos, it may be determined that the drawer is stopped during the time the two photos are taken. If the position of the marker is changed in two consecutive photos, it may be determined that the drawer is moved during the time the two photos are taken. At this time, it may be determined whether the drawer moves in the closing direction or the drawer moves in the open direction according to the change direction of the position of the marker.

In the second section, it is also possible to distinguish that the drawer is actually in an open state because the user has an opening sufficient to take out or bring in an article to the drawer.

45 is a control flowchart according to an embodiment of the present invention. Hereinafter, it will be described with reference to FIG. 45 .

It is determined that the door 20 opens the storage chamber 22 (S10). At this time, opening and closing of the door 20 may be detected by the door switch 110 .

When the door 20 is opened, the door sensor 120 may be driven. That is, the door sensor 120 may detect whether the door 20 is rotated by a predetermined angle θ or more. That is, the door sensor 120 may not operate when the door 20 is not opened, but may start to operate when the door 20 is opened.

Also, when the door 20 is opened, the camera 70 may be operated to start photographing (S14). In this case, the camera 70 may take pictures at a predetermined time interval, for example, 10 pictures per second.

The picture taken by the camera 70 is stored in the storage unit 18 (S16). At this time, if a plurality of pictures are taken by the camera 70 , the most recent picture is stored, and the picture taken before that may be deleted due to the capacity limitation of the storage unit 18 .

On the other hand, it is possible to detect whether the drawer 50 is opened by the drawer detection unit 130 (S20). The drawer detection unit 130 may detect that the drawer 50 is withdrawn from the lower space of the shelf 40 .

And when it is sensed that the drawer 50 is closed by the drawer sensing unit 130, a picture taken at that time or a picture taken at the time closest to that time is selected as the final picture (drawer open state) ( S40). Since the camera 70 takes a predetermined number of pictures per second, it is also possible that pictures are not taken exactly at the time when the drawer 50 starts to close. However, if the drawer detection unit 130 determines the movement of the drawer 50 by analyzing the movement of the marker expressed in the picture taken by the camera 70, the picture is taken at the time it is determined to be closed. It is possible to choose

The control unit 100 divides the final picture into the first area 42 , the second area 52 , and the third area 62 ( S42 ). In this case, the control unit 100 can classify each area by specifying the number of pixels based on the control line 15 , and obtain a final image for each area.

This is because, as described above, when the drawer 50 is drawn out, the second region 52 and the third region 62 overlap each other.

Meanwhile, since information indicating that the drawer 50 is withdrawn is obtained in S20 , the controller 100 may determine that the second image is obtained.

In addition, information on the second region 52 may be updated (S46). The control unit 100 may transmit a second image showing the second region 52 to the display 14 and change the corresponding portion to be based on the final photograph.

On the other hand, if it is determined in S30 that the drawer 50 does not start to close, a picture may be taken again as in S14.

The state in which the drawer 50 is not closed in S30 may be regarded as that the user does not terminate the access to the second area 52 . That is, the user can store new food in the second region 52 or take out the stored food from the second region 52 while the drawer 50 is stopped.

On the other hand, when the drawer 50 is closed, it can be predicted that the user ends the access to the second area 52 and draws the drawer 50 into the lower space of the shelf 40 .

Meanwhile, in the state in which the drawer 50 is not opened in S20, it may be determined whether the door 20 is just before closing (S60).

At this time, whether the door 20 is just before closing may be detected by the door sensor 120 .

When the user ends access to the storage chamber 22 , the user closes the door 20 and seals the storage chamber 22 . That is, the user can store new food in the storage compartment 22 or take out the stored food from the storage compartment 22 while the door 20 opens the opening 14 .

If the user closes the door 20 , it may be determined that the food is maintained in that state until the user ends the access to the storage room and the door 20 is opened again.

If it is determined in S60 that the door 20 is just before closing, the control unit 100 selects the picture taken at the time or the picture taken at the time closest to the time as the final picture (closed state of the drawer) (S70). That is, the control unit 100 may select a picture taken at a time point at which the camera 70 finishes taking a picture or a picture taken at a point adjacent to the time point as the final picture.

The controller 100 divides the final photo into the first area 42 , the second area 52 , and the third area 62 ( S72 ). In this case, the final photo may include a first image showing the first region 42 . The latest state of the food stored in the first area 42 is because the user has finished accessing the storage chamber 22 . Accordingly, through the first image obtained at this time, the user can obtain accurate food information on the first region 42 .

Meanwhile, since information indicating that the drawer 50 has been drawn is obtained in S60, the control unit may determine that the first image and the third image are obtained. As described above, when the drawer 50 is drawn out, the second area 52 and the third area 62 overlap each other, but the first area and the third area do not overlap each other. This is because it is a location where we can film together.

In addition, information on the first region 42 and the third region 62 may be updated (S76). The control unit 100 transmits a first image showing the first region 42 and a third image showing the third region 62 to the display 14, and determines the part based on the final photograph. can be changed to

Meanwhile, although not shown in the drawings, the door 20 is rotated to seal the storage chamber 22 , and when the storage chamber 22 is closed, it is possible to stop the operation of the camera 70 . At this time, the door switch 110 may detect whether the door 20 seals the storage chamber 22 .

That is, in the present invention, in consideration of the user's end of use, the photo obtained at the corresponding time is processed to provide the user with information about the food stored in the corresponding area. Accordingly, the user can obtain accurate information about the food stored in the storage chamber 22 .

Meanwhile, the door sensor 120 , the door switch 110 , and the drawer detection unit 130 may be installed two on the left and right sides, respectively. In this case, individual information about the left and right doors and the left and right drawers may be acquired. Of course, if the drawer detector 130 detects the movement of the drawer through the analysis of the marker in the photographed picture, the drawer detector 130 is one, whereas the marker is installed in two drawers, respectively. it is possible to be

Therefore, based on this information, when only the left door is opened, it is possible to update only the left part of one picture. On the other hand, when only the right door is opened, it is possible to update only the right part of one picture.

Of course, if the above-described conditions are satisfied when the left and right doors are opened together, it is also possible to update the left and right portions of one picture and provide them to the display 14 .

46 is a control flowchart illustrating a modification example of FIG. 45 . Hereinafter, it will be described with reference to FIG. 46 .

In the modified example of FIG. 46 , S60 is a form in which S62 and S64 are specifically divided, and only the difference portion will be described in detail.

That is, whether or not the door 20 is just before closing can be specifically determined by determining whether the door 20 is rotated in the closing direction (S62) and whether the door 20 is rotated more than a certain angle (θ). have.

In other words, when the door 20 is rotated in the closing direction ( S60 ) and the door is rotated at a predetermined angle or less ( S64 ), the photographing by the camera 70 may be terminated.

In order to obtain information about the food stored in the storage chamber 22 from the picture taken by the camera 70 , interference from the door 20 as well as the basket installed on the door 20 should not occur. Therefore, the expression that the door 20 is just before closing may mean a state in which the door 20 and other components are not expressed in the picture taken by the camera 70 .

47 is a control flowchart for explaining another modified example of FIG. 46 . Hereinafter, it will be described with reference to FIG. 47, but only the difference from FIG. 46 will be described.

In another modified example according to FIG. 47 , unlike in FIG. 46 , the rotation direction of the door 20 is not determined.

If it is determined in S12 that the door has been rotated by a certain angle or more, the door 20 is rotated in the opening direction at the corresponding moment. Accordingly, when the user rotates the door 20 again and reaches a state in which the door 20 is rotated by a predetermined angle, it may correspond to a case in which the user closes the door 20 .

Therefore, in another modified example, if S62 in FIG. 46 is omitted, and it is determined that the drawer is not opened in S20, S64 of determining whether the door is rotated by a predetermined angle or less is immediately performed.

48 is a control flowchart illustrating another modification of FIG. 45 . Hereinafter, it will be described with reference to FIG. 48 .

In FIG. 48, a control flow for detecting the movement of the drawer 50 by a marker will be described. For convenience of explanation, overlapping parts are omitted, and only the different parts will be described.

Overall, S20 and S30 may be specifically divided into steps S22, S24, and S26.

After the photo is stored in the storage unit 18 as in S16, it is determined whether the marker is recognized by the camera 70 (S22).

Since the camera 70 takes a picture from the upper side to the lower side of the inner case 12 , a case may occur where the marker is not visible from the camera 70 . That is, when the handle portion of the drawer 50 is covered at the front end of the shelf 40 , that is, in a state in which the drawer 50 is not drawn out at all, the marker may not be recognized.

When the marker is recognized in the picture taken by the camera 70, it can be determined whether the marker is stopped (S24). In this case, as described above, it is also possible to determine the movement of the marker in the first section and the second section after recognizing the movement path of the marker in one picture, and to determine the movement of the marker only in the second section.

Since the camera 70 takes a plurality of pictures at predetermined time intervals, it is possible to determine whether the marker moves or not by analyzing the pictures while comparing them. Since the marker is displayed on the drawer 50, when the marker is moved, it can be determined that the drawer 50 is also moved.

In addition, it may be determined whether the marker has moved more than a certain distance from the position in which the drawer 50 is closed (S26).

If the marker does not move more than a certain distance, an appropriate image for the second area 52 cannot be obtained because the drawer 50 is not sufficiently opened. For example, in the picture taken in a state in which only 1/3 of the drawer 50 is withdrawn, the inner space of the drawer 50 is not sufficiently shown, and in the image obtained from the picture, the user can view the second area (52) is not available. In this case, the predetermined distance may be determined based on whether the marker has entered a search range for determining the movement of the marker.

If it is determined that the marker has moved more than a certain distance, it can be considered that the user ends the access to the second area 52 and the use of the second area 52 is finished.

Accordingly, a photograph obtained at a time when the three conditions are satisfied or a time adjacent to the time may include information about food stored in the second area 52 .

49 is a view showing a modified example of FIG. 45 differently. Hereinafter, it will be described with reference to FIG.

In FIG. 49 , unlike FIG. 45 , the camera is not driven immediately when the door 20 is opened by the door switch 110 .

When the door switch 110 determines that the door 20 is open, the door sensor 120 is driven.

Then, the door sensor 120 determines whether the door 20 has been rotated by a predetermined angle or more (S12). At this time, when the door 20 is not rotated by a predetermined angle or more, the camera 70 does not take a picture.

When the door 20 is rotated more than a certain angle, the camera 70 takes a picture (S14). In this case, the camera 70 may take pictures at a predetermined time interval, for example, 10 pictures per second.

The picture taken at this time includes a first image of the first area 42 . On the other hand, since the door 20 is rotated by a certain angle (θ) or more, interference from the basket installed on the door 20 may not occur in the photograph. That is, the appearance of the basket installed on the door 20 may not appear in the photo.

Furthermore, in S60, the door sensor 120 may detect whether the door 20 is just before closing. In this case, the meaning that the door 20 is just before closing may mean a state in which the door 20 is rotated by a predetermined angle or less while the door 20 is rotated in a closing direction.

Until just before the door 20 is closed, the camera 70 may take a picture and stop taking the picture (S49). In FIG. 45 , the camera 70 takes pictures until the door 20 is closed and the door switch 110 is pressed near the moment it is pressed. However, in the modified example, when the door 20 is rotated to a predetermined angle by the door sensor 120 , the camera 70 stops taking pictures. In this case, the predetermined angle may mean an angle that prevents interference by the door 20 as well as the basket installed in the door 20 from occurring in the picture taken by the camera 70 .

50 is a control flowchart according to another embodiment of the present invention. Hereinafter, it will be described with reference to FIG. When it is determined that the door 20 is opened by the door switch 110 , the control flow of FIG. 50 may be continuously performed at intervals of a predetermined time.

First, it is determined by the door sensor 120 whether the door 20 is opened by a predetermined angle θ or more (S100).

When the door 20 is opened more than a certain angle, it is determined that interference does not occur in the door 20, and the camera 70 takes a picture (S110). In this case, the camera 70 may take a single picture.

And it is determined whether the drawer 50 is drawn out more than a first set distance (S120). In this case, the first set distance may mean a distance at which food information for the second area 52 can be sufficiently obtained when a picture is taken by the camera 70 . For example, the first set distance may mean 2/3 or 1/2 of the total length of the drawer 50 .

The withdrawal distance of the drawer 50 may be detected by the drawer sensing unit 130 .

On the other hand, if it is determined that the drawer 50 is drawn out over the first set distance, the control unit 100 may divide the photo by area and divide it into a plurality of images (S122).

Since the picture is obtained in a state in which the drawer 50 is withdrawn, a first image and a second image may be obtained from the picture.

The controller 100 may update the first image and the second image (S124).

On the other hand, if it is determined in S120 that the drawer 50 is not drawn out more than the first set distance, it may be determined whether the drawer 50 is drawn out by the second set distance or less (S130).

In this case, the second set distance may mean a distance at which food information for the third area 62 can be sufficiently obtained when a picture is taken by the camera 70 . For example, the second set distance may mean 1/3 of the total length of the drawer 50 . Of course, it is also possible to select a state in which the drawer 50 is completely drawn into the lower space of the shelf by selecting the second set distance as '0'.

On the other hand, if it is determined that the drawer 50 is drawn out by the second set distance or less, the control unit 100 may divide the picture by area and divide it into a plurality of images (S132).

Since the picture is obtained while the drawer 50 is withdrawn, a first image and a third image may be obtained from the picture.

The controller 100 may update the first image and the third image (S134).

In another embodiment of the present invention according to FIG. 50 , when the door 20 is opened, control is repeated, and when the door 20 closes the storage chamber 22, the control is completely terminated. Since the update of the image is no longer generated in the finished state, the final image of the food stored in the storage compartment 22 can be obtained.

51 is a view for explaining a process of updating an image in a state in which two doors and two drawers are provided to open and close the storage room. Hereinafter, it will be described with reference to FIG. 51 .

The overlapping contents in the above-described embodiment will be briefly described.

When electricity is supplied to the refrigerator, it may be individually determined whether the left door or the right door is opened by the door switches 110 installed on the left and right sides, respectively (S302 and S303).

When the door switch 110 determines that either the left door or the right door is open (S303), the camera 70 starts taking a picture (S304).

Even if any one of the left door and the right door is opened, as well as a time interval between the left door and the right door, or both of them are opened at the same time, the camera 70 can take a picture. This is because, when it is determined that the door is opened through one of the door switches, the light is turned on in the refrigerator.

When the drawer disposed on the left is drawn out more than 130 mm (S310), an image related to the left drawer is captured (S312). The image related to the left drawer may mean a portion corresponding to the left drawer among the photographed photos, and may mean the entire photographed picture.

In this case, 130 mm may be changed by a refrigerator manufacturer or a user, but may mean a withdrawal distance of a degree in which the user withdraws the drawer and accesses the storage space inside the drawer.

If the left drawer is withdrawn less than 130 mm (S310), the previous image captured by the camera 70 is maintained (S310).

And the left door sensor detects the opening angle of the left door. At this time, if the left door is opened at an angle smaller than 80 degrees, an image of the left multi storage space provided in the lower space in which the left drawer is withdrawn is captured (S332).

In this case, 80 degrees may mean an angle to the extent that the left door and various structures installed on the left door do not interfere when photographing the storage space with the camera 70 . Depending on the specific structure or size of the refrigerator, it is possible to change 80 degrees.

If the left door is opened at an angle greater than 80 degrees, it is possible to maintain the immediately preceding image taken by the camera 70 (S336).

And it is possible to sense that the left door is closed by the door switch 110 (S334).

Meanwhile, for the right drawer and the right door, it is possible to implement the same type of control flow as described in the left drawer and the left door. That is, in S320 to S324 and S340 to S346, the detailed description is omitted because the above-described contents are equally applied.

When the left door and the right door are closed, it is possible to process the image captured above in the control unit 100 and display the processed image on the display 14 (S350). In this case, the display may be a display installed in the refrigerator.

And the camera may stop taking the picture (S352).

Meanwhile, the refrigerator is connected to Wi-Fi to determine whether communication is possible (S354), and if communication is possible, it can transmit an image to an external server (S356).

The server determines whether it is synchronized with the application of the terminal used by the user (S358), and if synchronized, it is possible to display the image on the display 14 of the terminal.

Accordingly, the user may be provided with information on items stored in the refrigerator through the user terminal even if the user is not in front of the refrigerator.

If the refrigerator is not connected to Wi-Fi and cannot communicate with the outside, or is not synchronized with the application of the terminal, it is preferable to maintain the previous image on the display 14 of the user terminal and provide the immediately preceding image to the user do.

Hereinafter, components provided for taking a picture and acquiring a picture at a desired time will be schematically described.

When the door switch detects that the user starts to open the door, the control unit drives the camera to start photographing the interior of the room. The camera continuously takes pictures at a fixed frame rate per second. Since opening the user's door means taking out a desired item from the refrigerator or storing a new item in the refrigerator, by continuously shooting the inside of the refrigerator, the It is possible to monitor whether there is a change in the state of the storage space.

The storage compartment is separated by various shelves, drawers or baskets to form storage spaces, respectively, and is stored and stored in the storage space according to the shape or content of articles or containers.

The camera can monitor the storage space by dividing it into a shelf area separated by a shelf, a drawer area, and other storage areas. Through this, it is intended to allow the user to check the state of storage of goods in the area immediately after the user completes the withdrawal or warehousing operation. If two drawers are provided and two other storage areas are provided in parallel on both sides, information that allows the user to check each of the two drawers and two other storage areas through the camera is provided. can

In one embodiment of the present invention, the other storage area is formed on the floor of the storage room, is located in front of the drawer storage area, and at least a part of the storage space overlaps when the drawer is opened, and includes it in the monitoring target. present.

When the door switch detects the opening of the door, the controller turns on the door sensor included in the hinge assembly of the refrigerator to detect movement according to the rotation of the door. The door sensor may be always on, but it is also possible that the door sensor is turned on only when the door is opened by the door switch.

When the door is opened more than a certain angle, the control unit determines that the door is substantially open enough to allow the user to access the storage room to bring in or take out items, and determines the storage condition of the shelf area to be upgraded.

In addition, the drawer sensing unit detects the degree of opening of the drawer and, when the drawer is opened, determines the storage state of the drawer area to be upgraded, and when the drawer is not opened, the storage area is to be upgraded to decide

The door sensor according to an embodiment of the present invention detects when a predetermined angle has elapsed in the process of opening and closing the door, and sends related information to the control unit. Here, the door sensor may also detect the rotational direction of the door, that is, whether it is a closing direction or an open direction.

In the embodiment of the present invention, when the on signal of the door switch is detected, the control unit turns on the door sensor, and the door sensor detects when the door passes a certain angle and sends a corresponding time point to the control unit it becomes At this time, the control unit determines that the door is open. Specifically, at this time, the door is rotated by a predetermined angle and is in an open state.

On the other hand, when the user finishes taking out or put in the goods while the door is open, the opened door is closed, and when the door passes the door sensor, the door sensor sends a signal to the control unit, and the control unit sends a signal to the user from this signal. It is determined that it is just before the door closes after completing the warehousing act. Specifically, at this time, the door is rotated by a certain angle and opened, but is rotated to close.

An embodiment of the present invention will be described in detail with reference to an embodiment of the present invention for the process of capturing an image of the inside of the cabinet from a camera during the process of opening and closing the door and upgrading to the image captured immediately after the user completes the act of putting in and out of goods.

When the refrigerator door switch is turned on while the door is closed, the controller turns on the camera and the door sensor. The camera may mean being switched from the standby mode to the driving mode, and the door sensor may mean being switched to a state capable of determining the opening angle of the door.

The camera may continuously photograph the inside of the refrigerator at a predetermined speed per predetermined time. The captured image is sent to the control unit, and the control unit temporarily buffers the transmitted image as an image including the entire interior of the refrigerator.

The image [Picture (or photograph)] and the image (Image) used in this specification may be used separately from each other.

Specifically, an image (Picture or photograph) means raw data transmitted immediately after being photographed by a camera, and an image is a raw data image (picture or photograph) transmitted by the control unit. It may mean buffered, corrected, or transmitted data.

When the door switch is on and the door is opened at a certain angle and the control unit determines that the door is in an open state, the control unit determines which area the entire image contains the storage state of the item according to the open state of the drawer. will decide

For example, when the drawer is in the open state, the controller controls the area (shelf area) in which the entire stored image is stored on the refrigerator shelf and the drawer storage area (drawer area) stored in the drawer. It is judged that it contains the storage condition of the goods. On the other hand, if it is determined that the drawer is in the closed state, the controller controls the storage space inner area (storage area), which is an area where the entire stored image is formed in the refrigerator shelf area and the front end of the drawer and overlaps by opening the drawer. It is judged that it contains the state of storage of the goods.

Here, the picture of the entire refrigerator storage room captured and transmitted by the camera is divided into three different storage spaces, the shelf area is always included, and the drawer area and the storage area are the drawer areas. It is selectively included according to the open/closed state of the lower. The drawer area and the accommodation area do not coexist with each other in the photo.

Accordingly, one entire picture transmitted from the camera includes at least two areas, one of which is a shelf area, and which storage area the other part corresponds to is determined by the drawer opening/closing state.

The present invention is to transmit the recent arrival/delivery status of goods to the user by photographing at least two different spaces including the inside of the drawer, such as the inner space of the drawer and another storage space, with one camera, so that the drawer A method of upgrading by fixing only the area and the shelf area may be another embodiment of the present invention, and fixing only the drawer area and the storage space may be an embodiment of the present invention. .

On the other hand, dividing one entire picture of the storage compartment constituting one independent insulating space in the refrigerator into storage areas in the storage compartment may be determined by the number of doors that open and close the storage compartment. For example, in a refrigerator that opens and closes the storage compartment with one door, the storage area to be photographed may be divided into two or three types. When the storage compartment is opened and closed with two left and right doors, the storage area is It can be broadly divided into 4 or 6 categories.

In this case, the drawer area and the storage area may be divided to the left and right to be managed as two partitioned areas, respectively, in the entire photo, and the shelf area may be managed as one area. In this case, all the partitioned parts in the whole photo may be composed of five parts.

The shelf area is connected to the storage space facing the left and right doors. In the shelf area, items, food or vegetables may be stored across the left and right sides. Since items can be retrieved or received by opening only one of the left and right doors when stored across, even if only one of the left and right doors is opened, the entire shelf area in the captured image is integrated and replaced with the latest data. because there is a need In consideration of this point, even if only one of the left and right doors is opened, it is possible to update the entire shelf area.

In more detail, when it is determined that the drawer is opened in the state in which the door is opened, the controller separates the shelf area and the drawer area from the entire transmitted photo and temporarily buffers them. And, when the user finishes loading/unloading items in the drawer area or/and the shelf area and tries to close the drawer, that is, when the drawer is opened and then started to move in a stationary state, the most recent among the temporarily buffered images. The necessary part in the drawer area is captured (captured or cut) from the image taken in , and it is stored in the storage unit as an image of the recent storage state of the drawer area. At the same time, the image captured on the drawer area in the image inside the refrigerator displayed on the refrigerator display is replaced with the latest image to upgrade. And, along with this, the control unit transmits the captured image of the drawer area to the server system connected to the network.

Meanwhile, when it is determined that the drawer is closed because the opening of the drawer is not detected in the open state of the door, the controller determines that the entire picture of the transmitted image includes the shelf area and the storage area. Therefore, the shelf area and the storage area are separated from the entire picture and temporarily buffered.

When the user completes the operation of putting on and leaving the product in the storage area and/or the shelf area and closing the door, the door sensor detects when the door passes a specified angle and sends a signal to the control unit. The control unit determines by the signal transmitted from the door sensor that the door is just before closing (it can be expressed as just before the picture is obscured by an obstruction such as the door), and is stored in the most recent image among the temporarily buffered images. The necessary part in the area is captured and stored in the memory as an image of the recent storage state of the storage area. At the same time, the image captured on the storage area in the refrigerator image displayed on the display is replaced and upgraded.

In this case, the upgrade may mean replacing an existing image with a newer image. And, along with this, the control unit transmits the captured image of the drawer area to the server system connected to the network.

Meanwhile, when the drawer is in an open or closed state, or when it is determined that the shelf area is just before the door is closed, the most recent image among images being buffered is captured and stored in the memory as the latest storage state of the shelf area.

The image displayed on the refrigerator display is replaced by the captured image (any one or both of the recent shelf area image, the recent drawer area image, and the recent storage area image), or is transmitted to a server system or connected to the refrigerator through communication. The transmission to the mobile device of the user may be performed simultaneously while storing the captured image in the refrigerator memory, may be performed at once when the user requests it, or may be performed by the user's request signal for each individual area.

Here, one image transmitted from the camera may be buffered into one image by the controller. If it is determined that the single image is photographed with the door open, it may be determined whether the single image includes a drawer area or a storage area according to a drawer opening/closing state. As such, the image is extracted and buffered in a separate temporary buffer.

If it is determined that one image was taken when the drawer starts closing, the most recent image among the temporarily buffered images is stored in the memory, and when a new warehousing action by the user is completed, it is replaced with the latest image. . Also, if it is determined that the door is just before closing, the most recent image among the images of the temporarily buffered storage area is stored in the memory.

On the other hand, for the shelf area, only the shelf area can be extracted from the most recent image just before the door is closed among all the buffered images and stored in the memory, and a temporary buffer is prepared for a separate shelf area to extract and buffer only the shelf area. may be

Hereinafter, various embodiments in which the most recent refrigerator interior image obtained by the various methods described above are transmitted to a personal mobile terminal or a business server server will be described with reference to FIGS. 52 to 55 .

52 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 52 , when it is determined that the final picture of a specific region inside the refrigerator has been obtained through the steps described with reference to FIG. 34 ( S1101 ), the control unit 100 converts the corrected final image into an event format to the terminal ( 2) (S1103). The terminal 2 may include a smart phone, a personal digital assistant (PDA), a tablet PC, and the like, but is not limited thereto. At this time. The controller 100 may transmit to the terminal 2 not only the final image of the inside of the refrigerator, but also various information related to the final image to be described below.

In this case, the control unit 100 may transmit the final image to the terminal 2 as soon as the final image is obtained through the final photo, or store the obtained final image in the storage unit 18, and then according to a predetermined cycle An image stored in an event format may be transmitted to the terminal 2, but is not limited thereto.

Then, the terminal 2 displays the final image received according to the user's selection (S1105). For example, the terminal 2 may execute an application capable of receiving and displaying an image inside the refrigerator according to a user's selection, or may perform a multimedia service to display the received image inside the refrigerator.

According to this embodiment of the present invention, even if the user does not directly look inside the refrigerator, since the image inside the refrigerator can be checked using the mobile terminal, there is a practical benefit of remotely grasping the current interior condition of the refrigerator.

53 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 53 , when it is determined that the final picture of a specific area inside the refrigerator has been obtained as described above ( S1301 ), the controller 100 divides the final picture by area to generate an image, and then the obtained final image is It is controlled to be stored in the storage unit 18 (S1303).

Subsequently, when a user input for displaying a current refrigerator interior image, that is, a final image of a specific region of the refrigerator is received (S1305), the terminal 2 transmits a final image request command (request) to the refrigerator (S1307).

And, the control unit 100 transmits to the terminal 2 the most recently photographed and stored refrigerator interior image among the images stored in the storage unit 18 as a response to the final image request command (request), the communication unit ( 270) is controlled (S1309).

According to this embodiment of the present invention, communication between the mobile terminal and the refrigerator is performed only when the user needs it, so that it can be more efficient and economical. Hereinafter, only a case in which an image is transmitted as a response to a request is described, but a case in which an image is transmitted in the form of an event as described with reference to FIG. it is not going to be

54 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 54 , when it is determined that the final photo of a specific region inside the refrigerator has been obtained as described above ( S1501 ), the controller 100 controls so that the final image obtained from the final photo is stored in the storage unit 18 . (S1503).

Subsequently, when a business server such as the market server 3 receives a user input for displaying a current image inside the refrigerator (S1505), the market server 3 transmits a final image request command to the refrigerator (S1507) . In this case, the market server 3 may transmit a final image request command to the refrigerator according to a predetermined period without a user input.

Then, the control unit 100 transmits the most recently photographed and stored final image inside the refrigerator among the images stored in the storage unit 18 in response to an image request command (request) to the market server 3 ( S1509).

The market server 3 that has received the final image inside the refrigerator analyzes the final image (S1511). The market server 3 analyzes the final image, finds out the storage currently stored in the refrigerator, and the storage that is not currently stored, and analyzes the storage that has been stored in the refrigerator in the past but is not currently stored. It is possible to extract information necessary to provide services, such as

The market server 3 transmits the list of insufficient items extracted through the final image analysis to the corresponding refrigerator (S1513).

55 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 55 , when it is determined that the final picture of a specific area inside the refrigerator has been obtained as described above ( S1701 ), the controller 100 controls so that the final image corrected from the final picture is stored in the storage unit 18 . (S1703).

Subsequently, when a business server such as the broadcasting station server 4 receives a user input for displaying the final image inside the refrigerator (S1705), the broadcasting station server 4 transmits a final image request command to the refrigerator (S1707) . In this case, the broadcasting station server 4 may transmit a final image request command (request) to the refrigerator according to a predetermined period without user input.

Then, the control unit 100 transmits the final image inside the refrigerator most recently photographed and stored among the images stored in the storage unit 18 to the broadcasting station server 4 in response to the final image request command (request). control (S1709).

The broadcasting station server 4 receiving the final image analyzes the final image (S1711). The broadcasting station server 4 may analyze the final image, find out what is stored in the refrigerator, and extract information necessary for service provision, such as analyzing dishes that can be prepared using the currently stored storage.

The broadcasting station server 4 transmits recipe information of dishes that can be made using the currently stored storage to the corresponding refrigerator through final image analysis (S1713).

The control unit 100 displays recipe information of dishes that can be prepared using the currently stored storage received from the broadcasting station server 4 on the display 14 installed in the refrigerator (S1715).

According to such an embodiment of the present invention, there is a real benefit that a related business operator can provide a refrigerator-related service suitable for the home or business by accurately understanding the internal situation of a refrigerator located at a home or business.

FIG. 56 is a view for explaining the operation of a heater of the camera, and FIG. 57 is a view for explaining an experimental result regarding dew condensation generated by temperature on a transparent window of the camera. Hereinafter, it will be described with reference to FIG. In FIG. 56 , the heater of the type described in FIGS. 14 to 17 will be described as a reference.

Fig. 56 (a) is a graph of a form in which power is supplied intermittently to the heater 84 and is operated intermittently, and Fig. 56 (b) is a graph of a form in which the heater 84 is continuously supplied with power and continuously operated. It is a graph.

56( a ) shows a method in which the temperature of the transparent window 80 is not normally controlled, and the temperature of the storage chamber 22 is maintained in an equilibrium state, and then high power is applied instantaneously when the refrigerator camera needs to be photographed. In particular, by operating the heater 84 just before taking a picture, dew condensation generated on the transparent window 80 can be removed.

When the door 20 is opened, the transparent window 80 may come into contact with the moisture contained in the warm outside air that has entered through the door 20 . At this time, since the transparent window 80 has a relatively low temperature compared to the outside air, moisture contained in the outside air is condensed in the transparent window 80, and dew condensation occurs.

In order to remove the dew generated by the dew condensation, if the heater 84 is instantaneously driven, there may be a problem that it takes time until the dew on the transparent window 80 is actually removed. Therefore, if the time at which the photographing is required is earlier than the time at which the dew is removed, a problem that dew may form on the transparent window 80 may occur.

On the other hand, in FIG. 56(b) , the heater 84 is continuously driven, so that heat is supplied to the transparent window 80 . Since the temperature of the transparent window 80 is always maintained above the dew condensation temperature, dew condensation does not always occur in the transparent window 80 . Accordingly, even if the door 20 is opened and the transparent window 80 comes into contact with the outside air, moisture contained in the outside air does not condense on the transparent window 80 as dew. Accordingly, it is possible to prevent dew from forming on the transparent window 80 at a point in time when the camera 70 needs to take a picture.

In driving the heater 84, it is necessary to prevent an image quality problem due to dew condensation and to prevent a sudden increase in power consumption of the refrigerator due to the power consumption of the heater 84. In addition, it is also desirable to consider that the heat supplied by the heater 84 should not affect the internal temperature of the refrigerator.

The present inventors believed that it would be necessary to increase the time for turning off the heater in order to minimize the power consumption of the heater and not affect the inside of the refrigerator. However, if the heater is turned off for a long time, there is concern that condensation will occur, so set the target temperature in the camera transparent window, and the refrigerator outside temperature is 32 degrees Celsius and the relative humidity is 85% (the dew point temperature is approximately Celsius 29 degrees to 30 degrees level) (see FIG. 57). In FIG. 57A , the y-axis denotes the amount of condensation generated on the transparent window, and in 57b, the y-axis denotes the percentage of the amount of condensation actually generated at the corresponding temperature compared to the current amount of water vapor.

The on/off operation cycle of the heater derived from repeated experiments could deteriorate the image quality of the pictures taken by the camera under special temperature conditions inside the refrigerator.

Even if an appropriate on/off cycle of the heater is found, a user's refrigerator usage pattern during the time the heater is off may vary unpredictably.

For example, when the user leaves the refrigerator door open for a long time, the camera cover glass (the cover glass may mean the above-described transparent window, or various types of windows placed in front of the camera lens). There may be a large amount of dew condensation on the surface. In this case, even if the cover glass is heated by operating the heater, since it takes a lot of time for evaporation, there is a problem in image quality when shooting at that time. E In addition, a lot of time as well as a lot of power was consumed to evaporate the already generated dew from the cover glass.

In addition, when the storage compartment is a refrigerating compartment, the temperature inside the refrigerator is usually maintained between 0°C and 7°C, and the internal volume is also larger than the cover glass surface area. A large portion of the heat supplied by the heater is inevitably leaked into the cold air inside the storage room. Therefore, in order to evaporate the dew generated on the cover glass, a large amount of heat must be applied for a short time. In order to intensively supply such heat for a short period of time, a large-capacity heater is used, and since instantaneous power is greatly increased, various problems may occur in a poor power situation.

Therefore, it is preferable to prevent the amount of dew condensation from being continuously generated on the cover glass from the beginning, and in the present invention, it is characterized in that power is applied to continuously operate the heater without an on/off cycle.

As for the method of always applying power to the heater, the power may be applied to the camera heater immediately when power is applied to the refrigerator, or the refrigerator may be controlled to always apply power to the heater during normal operation of the storage room through the refrigerator controller. have.

Next, the heating temperature of the cover glass will be described. As a result of quality evaluation based on the image samples obtained for each temperature range, it was confirmed that there was no significant problem in the quality of the sample image until the temperature of the central portion of the cover glass was approximately ½ of the dew point temperature. The absence of a problem in image quality may mean a degree to which the type and quantity of items stored in the storage room can be identified through the photographed photos.

In particular, the quality of the image sample obtained at 13.5, the point (3) of FIG. 57B, was at an applicable level. Of course, it is possible to maintain the heating temperature of the cover glass at a temperature higher than the dew point, but power can be supplied to the heater so that the heating temperature of the cover glass is maintained between about 13 degrees Celsius and about 30 degrees Celsius.

On the other hand, from the experimental results, the heating temperature of the cover glass was different depending on the outer size of the cover glass under the condition that the same applied power was supplied, and there was no significant difference in the thickness of the cover glass. Therefore, since the image quality was satisfactory at 1/2 of the dew point temperature at the same cover glass size, it is preferable to set this as the lower limit, but set an appropriate upper limit in terms of power consumption (applied power) of the heater.

When heating the cover glass, the applied power near the dew point temperature (29.2 degrees Celsius) was 0.45 W, and the applied power at about 0.58 times the dew point temperature (17.1 degrees Celsius) was 0.27 W. Therefore, in the range of 0.5 to 0.7 of the dew point temperature, the power consumption is also approximately 0.5 times, so a great effect can be seen in reducing power consumption.

The mid-band of the dew point temperature at ambient conditions, which can usually vary, may be determined to be approximately 12 to 17 degrees Celsius. Accordingly, it is preferable that a current value for maintaining the temperature of the central portion of the transparent window to be approximately 12 to 17 degrees Celsius is applied to the heater.

That is, in the present invention, it is possible to obtain the effect of reducing the power consumed by the heater by maintaining the temperature of the cover glass at a temperature lower than the dew point, not the dew point.

58 is a cross-sectional view of the transparent window. Hereinafter, it will be described with reference to FIG.

The transparent window 80 may be hydrophilic coated. In this case, the transparent window 80 may refer to the above-described cover 90 .

Even if dew is momentarily formed on the transparent window 80 , the contact angle α of water may be formed within 15 degrees due to the hydrophilic coating as shown in FIG. 58 . When hydrophilic coating is applied, the surface tension of water on the corresponding surface is reduced. This is because water can spread widely on the surface of the transparent window 80 . Accordingly, the camera may be fixed to the ceiling surface of the storage chamber so that the angle between the transparent window and the ceiling surface of the storage chamber is between approximately 10 degrees and 20 degrees.

Therefore, it is possible to minimize distortion of the photographed picture due to the water formed on the transparent window 80 . In addition, the hydrophilic coating may have advantages in terms of energy efficiency since it does not need to be controlled such as supplying additional electricity after being manufactured once.

59 and 60 are diagrams schematically illustrating a structure in which a camera is installed in an inner case. For reference, FIGS. 59A and 60A are views looking up at the side of the storage compartment, and FIGS. 59B and 60B are views looking up at the upper side of the inner case. Hereinafter, it will be described with reference to FIGS. 59 and 60 . Through these embodiments, it is possible to minimize the occurrence of condensation in the cover 90 by bypassing the path through which the outside air reaches the cover 90 .

The camera 70 may be installed on the protrusion 500 protruding downward from the ceiling of the inner case 12 . In this case, the protrusion 500 may mean a portion that protrudes downward compared to other portions of the inner case 12 . These protrusions 500 may be implemented through the aforementioned camera housings.

At this time, as shown in FIG. 59 , the camera 70 is installed within the protrusion 500 to face the inner wall of the storage chamber 22 . The camera 70 can take a picture inside the storage chamber 22 through the cover 90 . Accordingly, the cover 90 is disposed at a position facing the rear wall of the storage chamber 22 from the protrusion 500 .

At this time, the cover 90 is cooled while the moisture contained in the outside air comes into contact, thereby generating dew.

However, since the cover 90 is disposed to face the rear wall of the storage chamber 22 , the path through which the outside air reaches the cover 90 may be lengthened.

That is, in order for the outdoor air passing through the door 20 to pass through the storage chamber 22 and contact the cover 90, it must pass through the path as shown in FIG. Some of the moisture it contained may condense in other parts. Accordingly, the outside air may reach the cover 90 while the amount of moisture in the storage chamber 22 is gradually reduced.

That is, even when the door 20 is open, the inside of the storage compartment 22 maintains a lower temperature than the outside of the storage compartment 22 and does not immediately change to the same temperature as the outside air, so the outside air enters the storage compartment 22 and heat exchanges this is done

In other words, if the cover 90 is disposed in a position where it is relatively difficult to contact the outside air, the amount of dew generated on the cover 90 may be reduced, and the time point at which the dew is generated may be delayed.

First, the outside air comes into contact with the front surface 501 of the protrusion 500 as it enters the cabinet. At this time, the external air in contact with the front surface 501 enters the chamber while changing the path to the left and right of the protrusion. The outside air may reach the cover 90 while in contact with the side 502 of the protrusion 500 . Accordingly, a large amount of moisture may condense before reaching the cover 90 .

In addition, the external air in contact with the front surface 501 of the protrusion 500 may descend and rise on the lower surface 503 of the protrusion. Accordingly, a large amount of moisture may be condensed while moving to the front surface 501 and the lower surface 503 .

In addition, the cover 90 is mounted to be inclined in a shape surrounded by the front surface 501 , the side surface 502 and the lower surface 503 . Accordingly, the path of the outside air and the cover 90 are not perpendicular to each other. Accordingly, it can also be prevented that the outside air reaches the cover 90 and is rapidly condensed.

In particular, the width of the protrusion 500 , in particular, the left and right widths of the front surface 5010 may be wider than the width of the camera 70 .

The embodiment shown in FIG. 60 is very similar to the embodiment shown in FIG. 59 . However, it can be said that it is different that the camera is mounted to face the vertical lower part. Accordingly, the same effect as the example shown in FIG. 59 can be obtained.

The protrusion 500 may also include a rear surface 504 . It is preferable that the vertical length of the front surface 501 is greater than the vertical length of the rear surface 504 . That is, it is possible that the front surface 501 protrudes more downward of the inner case 12 .

Therefore, in order for the outside air to reach the cover 90 , the path lengthens in the horizontal direction by the width of the protrusion 500 , and in the vertical direction, the path becomes longer due to the height difference between the protrusions 500 . will lose

Prevention of condensation in the cover 90 through the structure of the protrusion 500 or the structure of mounting the camera is very related to the installation position of the camera. That is, it is because the camera can be installed in an angle range that is inclined backward by a predetermined angle from the vertical in the lower part of the ceiling of the storage room. Accordingly, it is possible to effectively bypass the inflow path of the outside air while effectively photographing the entire area in the storage chamber in the storage chamber.

In the prior art, there are those that disclose a technique for mounting the camera on the side wall of the main body storage compartment, the upper part of the external body, the door, or the upper part of the door.

In the prior art, when a camera is mounted on a side wall of a storage room, there is a technique for providing a recess for accommodating the camera on the side wall, and mounting the camera on the recess. That is, the camera is prevented from protruding into the interior of the storage room. It is assumed that this is a configuration not to cause inconvenience to users when taking out shelves or articles in the storage room.

However, in the present invention, since the camera is installed on the ceiling of the inner case 12 , the user is not disturbed when using the storage compartment 22 . This is because, when a user wants to take an article out of the storage room, since the camera is not located in the movement path of the article or shelf, there is no reason to collide with the camera.

Furthermore, in the present invention, the cover 90 of the camera 70 is arranged to face the rear, so that the path through which the outside air reaches the cover 90 can be extended.

In Figure 59a, while the cover 90 is arranged to face the rear, the protrusion 500 protrudes downward, so that the outside air has a three-dimensional movement path in order to reach the cover 90, the movement of the outside air path becomes longer.

Although the cover 90 is disposed to face downward in FIG. 60A , the front surface 504 disposed horizontally to the opening in the protrusion 500, both ends of the front surface 504 and the front surface 504 A side 508 disposed perpendicular to the side 508 is installed. In this case, the front surface 504 and the side surface 508 may be installed to protrude further downward from the protrusion 500 , so that the path through which the outside air reaches the cover 90 may be lengthened.

That is, the high-temperature air flowing in from the outside on the opening side of the storage chamber directly reaches the cover 90 of the camera in a straight line and does not come into contact, but bypasses the left and right and downwards along the protrusion 500 and arrives. Therefore, the outside air is heat-exchanged with the cold air inside the furnace and the temperature is lowered and the amount of saturated steam decreases. Therefore, the water vapor contained in it is condensed and mixed with the cold inside the furnace and disappears. When the cover 90 is finally reached, since it has a reduced amount of water vapor, it can be sufficiently evaporated by the heater heating the cover 90 .

Accordingly, the electric power to be provided to the heater 84 is reduced, so that the electric power consumed as a whole of the refrigerator can be improved.

59 or 60, a cabinet having a storage compartment formed therein; The storage compartment has an access opening on the front side, an upper wall, a lower wall, both side walls, and a rear wall made of an insulating material, a shelf area partitioned by a plurality of shelves, and a drawer having at least one drawer comprising a region; a door (at least one of doors) in close contact with the front of the storage compartment to open and close the storage compartment; a camera device installed on the upper wall of the storage compartment between the opening of the storage compartment and a front edge of the shelves installed in the storage compartment;

The camera device accommodates electrical components necessary for driving a camera including a camera lens, and a camera module part having a camera window formed at a predetermined interval with the lens, a camera module part, and the camera module It is configured to include a camera housing part (a camera housing part) for fixing the part by seating in a predetermined position inside;

The camera housing unit includes a fixing surface in contact with the upper wall of the storage chamber, a front surface formed on an opening side of the storage chamber, side surfaces connecting the front and rear surfaces, and the camera; To provide a refrigerator that is configured to include a top surface having an opening formed so that the camera window of the module unit is exposed, and the front surface of the camera housing unit is formed higher than the opening formed in the upper surface of the camera housing unit can

The upper surface of the camera housing part may be inclined at a predetermined angle in the opposite direction of the opening of the storage chamber with respect to the upper wall surface of the storage chamber.

The camera housing unit may further have a rear surface at a position opposite to the front surface on the rear wall side of the storage chamber, and the height of the front surface may be higher than the height of the rear surface.

The opening of the upper surface of the camera housing part may be formed at a lower position than the side surface of the camera housing part.

It may further include a housing installation part (a housing installation part) having a recessed portion (a recessed portion) to correspond to a recessed space inwardly of the upper wall of the storage compartment (a recessed space).

The housing mounting part may be mounted on the upper wall of the storage chamber together with the filling of the refrigerator insulator, and the camera housing part may be fixed to the housing mounting part.

The camera module unit fixed to the camera housing unit may be accommodated in a depression formed in the housing mounting unit.

The camera module unit accommodated in the recessed portion of the housing mounting unit may be fixed to the camera housing unit to have a gap with the inner surface of the housing mounting unit.

The front and both sides of the camera housing may be formed to be higher than the opening position of the upper surface of the camera housing.

The upper surface of the camera housing may be formed so that an opening side of the storage chamber is farther from the upper wall surface of the storage chamber than a rear wall side of the storage chamber.

The camera module unit has a front portion on which a camera window is formed and a rear portion opposite thereto, and at least one flat surface is formed on at least one of the front portion and the rear portion. can

A receiving portion for accommodating the camera module is formed inside the camera housing, and the receiving portion is inclined at a predetermined angle with respect to the upper wall surface of the storage compartment to place a flat surface formed on the camera module portion. may include a seating part.

The recessed part of the housing mounting part may include a seating part in which a flat surface formed in the camera module part is placed at a predetermined angle with respect to the upper wall surface of the storage chamber.

A flat surface may be formed around the camera window of the front part of the camera module, and the flat surface may be formed to have the same height as the center of the camera lens.

61 is a view showing another type of refrigerator to which the present invention is applicable. Hereinafter, it will be described with reference to FIG.

The refrigerator of FIG. 61 is a double-door refrigerator, and the left and right sides form different storage compartments. That is, the storage compartment on the left may be a freezing compartment, the storage compartment on the right may be a freezing compartment, and the storage compartment on the left may constitute a refrigerating compartment.

A camera 70 may be installed on the ceiling of the storage chamber 22 to face the lower portion of the storage chamber 22 .

In addition, a drawer 50 that can store articles and can be drawn in and out can be installed in the storage compartment 22 .

Other contents are omitted because they are the same as those of the refrigerator described above.

FIG. 62 is a screen provided to a user from the refrigerator of FIG. 61 . Hereinafter, it will be described with reference to FIG.

The user may be provided with an image of an article stored in the drawers 50 and 51 and an image of an article stored on the upper side of the shelf 40 . One drawer 50 may be provided in the refrigerator. Accordingly, an image related to one drawer may be provided also in FIG. 62 . However, the two drawers 50 and 51 may be provided in the upper and lower parts of the bauak kx shown in FIG. 61 . In this case, it is also possible that an image relating to the inside of each drawer is provided.

In the example described with reference to FIG. 2 , the drawers are provided on the left and right, respectively. However, in FIG. 62 , the drawer may be vertically provided. Accordingly, there may be two drawer areas, and the lower drawer 50 may protrude forward compared to the upper drawer 51 . In this case, each of the drawers 50 and 51 may also be provided with a marker.

Through this, it is possible to easily grasp food information stored in the plurality of shelf areas and the plurality of drawer areas through one camera.

The method of upgrading the image provided on the user screen is omitted because it is the same as the above-described refrigerator.

63 is a view for explaining a method of adjusting a picture taken by a camera in the refrigerator according to FIG. 61 . Hereinafter, it will be described with reference to FIG.

In general, since only one drawer is provided or arranged vertically, one horizontal adjustment line 15 may be provided in the refrigerator of the form of FIG. 61 . Of course, two horizontal adjustment lines may be provided to separate the two drawers from each other. In addition, it is also possible to adjust the position of the adjustment line by arranging two vertical adjustment lines that are arranged in parallel. Of course, unlike that shown in FIG. 63 , it is also possible to select a desired image using only the horizontal adjustment line without using the vertical adjustment line.

The user may be provided with a limit line 16 indicating a limit range for moving the adjustment line 15 . In this case, the limit line 16 may be arranged in parallel with a predetermined interval.

The correction of the error through the adjustment line 15 may be the same as or similar to that of the above-described embodiment.

The present invention is not limited to the above-described embodiments, and as can be seen from the appended claims, modifications can be made by those skilled in the art to which the present invention pertains, and such modifications are within the scope of the present invention.

10: outer case 12: inner case
20: door 40: shelf
42: first area 50: drawer
52: second area 60: barrier
62: third area 70: camera
100: control unit

Claims (40)

a body having an opening in the front and an insulating wall forming a storage space therein;
a door provided rotatably with respect to the main body to open and close the opening to form a storage chamber in which the storage space can store food below room temperature;
a drawer provided in the storage compartment;
a marker provided in the drawer;
at least one shelf provided in the storage compartment and positioned above the drawer;
a camera disposed outside the drawer to photograph the drawer; and
When the drawer is drawn out from under the shelf to the front of the shelf, the camera continuously photographs the drawer. The method of claim 1,
Refrigerator further comprising a control unit for comparing the two most recent consecutive images from among the continuously taken pictures or video images. 3. The method of claim 2,
and a controller for determining a position or movement state of a drawer through a marker included in an image of a photo or video obtained by continuous shooting by the camera. 4. The method of claim 3,
The controller divides the image of the photo or video sent from the camera into predetermined pixels, assigns coordinate values at regular intervals, and determines the position of the marker on the coordinates of the image. 5. The method of claim 4,
When the control unit recognizes that the coordinate value of the marker increases from 0, the control unit determines only a coordinate change with respect to a photo or image captured by the camera and sent to the control unit. 6. The method of claim 5,
A refrigerator that partitions and deletes images of parts other than the movement path of the marker for determining the coordinate change. 5. The method of claim 4,
When the control unit recognizes that there is no change in the coordinate value of the marker, the control unit moves a portion corresponding to a drawer area from an image of a photo or video captured by the camera and sent to the control unit to a temporary storage unit and stores it . 5. The method of claim 4,
When the control unit recognizes that the coordinate value of the marker changes to a smaller value, the control unit selects the most recent image from among the images temporarily buffered in the temporary buffer, moves it to the storage unit, stores it, displays it on the display, or Refrigerator that transmits to the terminal. 5. The method of claim 4,
The control unit determines that the drawer is not open when the position of the coordinate value of the marker is within a set first section, and determines that the drawer is open when the drawer pull-out distance is in a second section that exceeds the set section refrigerator to do. 10. The method of claim 9,
The refrigerator is defined as a first section when the drawer is within 30% of a withdrawal distance of the drawer, and a second section when the drawer is pulled out by 30% or more. 10. The method of claim 9,
The controller is configured to vary a method of determining a position of a drawer with respect to a photo or image transmitted from the camera according to a state in which the drawer is not opened and a state in which the drawer is opened. 12. The method of claim 11,
In the first section in which the drawer is determined as the closed state, the controller determines only whether the position of the pixel in the transmitted current image has reached the position of the pixel by a distance corresponding to the set withdrawal distance. 13. The method of claim 12,
The controller partitions and deletes or temporarily buffers a portion of the image except for the position of the marker. 12. The method of claim 11,
In the image processed by the controller, when the marker reaches a position greater than or equal to a specific pixel, which is the second section, two pictures (the image of the picture taken immediately and the picture taken after image) to determine the coordinate values of the markers. 15. The method of claim 14,
The control unit divides the drawer except for the position of the marker in the image and moves it to a temporary storage unit for storage. 15. The method of claim 14,
The control unit compares coordinate values of markers shown in two consecutive images in the second section, and determines whether the moving direction of the drawer, the stationary state of the drawer, or the fully opened state of the drawer. 10. The method of claim 9,
A refrigerator in which the speed at which the camera takes pictures (time interval at which pictures are taken) matches the speed at which the control unit determines and processes the transmitted picture or image from the image. 5. The method of claim 4,
When the marker is not detected from the image, the controller sets the coordinate value of the drawer to 0 and determines that the drawer is in a closed state. 18. The method of claim 17,
The controller determines that the drawer moves when the coordinate value of the marker changes from 0. 5. The method of claim 4,
Further comprising a door switch for detecting the opening and closing of the door,
In a state in which only the door is opened, the controller moves and stores the image corresponding to the shelf area to a temporary storage unit. 20. The method of claim 19,
When the door is closed, the controller selects a most recent image from among the images stored in the temporary storage unit and stores it in the storage unit, displays it on the display, or transmits the selected image to the terminal. 5. The method of claim 4,
Further comprising a door switch for detecting the opening and closing of the door,
When it is detected that the door is open and the drawer is also open, the controller moves the image corresponding to the drawer part from the image for the photo or video transmitted from the camera to a temporary storage unit and stores the image. 23. The method of claim 22,
When the control unit determines that the drawer has started to close, the refrigerator selects the most recent image from among the images stored in the temporary storage unit and stores it in the storage unit, displays it on the display, or transmits it to the terminal; 23. The method of claim 22,
The control unit separates the image portion corresponding to the shelf area and moves to the temporary storage unit for storage. 25. The method of claim 24,
When the door is closed, the control unit selects a most recent image from among the images stored in the temporary storage unit and stores it in the storage unit, displays it on the display, or transmits the selected image to the terminal. The method of claim 1,
A refrigerator including a marker in each image of a picture or video continuously taken by the camera. 27. The method of claim 26,
A refrigerator comprising a storage unit for storing the image obtained by continuous shooting by the camera. 27. The method of claim 26,
Refrigerator including images having different marker positions in the image. The method of claim 1,
The camera is disposed above the drawer so that when the drawer is withdrawn, the camera can photograph a plane of the drawer. 27. The method of claim 26,
In the process of withdrawing the drawer, the image of the photo or video captured by the camera includes the inside of the drawer and the marker. The method of claim 1,
A marker is a refrigerator in which the shape of the same shape or color is repeated at regular intervals. The method of claim 1,
Wherein the marker has a color or shape that contrasts with the drawer at the border with the drawer. 33. The method of claim 32,
A refrigerator having an inner portion of the marker having a color or shape contrasting with an outer periphery of the marker. The method of claim 1,
The marker is a refrigerator in which a length in a moving direction of the drawer is smaller than a length in a direction perpendicular to a length in the moving direction of the drawer. The method of claim 1,
Further comprising a display for displaying an image for the photo or video of the storage room taken by the camera,
The display is a refrigerator for displaying the image by dividing it for each storage area. 36. The method of claim 35,
The image displayed by the display includes an image of an inner region of the drawer. 37. The method of claim 36,
The image of the inner region of the drawer is an image of a photo or video taken when a closing operation of the drawer is sensed. 38. The method of claim 37,
When the closed state of the drawer is detected and an image of a photo or video for a storage area inside the drawer is secured, the refrigerator displays a user input unit on the display so that the user can check it. The method of claim 1,
Even when the drawer is in a stopped state or is completely withdrawn, the camera continues to take pictures to obtain an image of the drawer. 40. The method of claim 39,
A drawer withdrawal length in a state in which the drawer is fully drawn out is 50% of a length before and after the drawer.

Images