KR20210036887A - Refrigerator and Controlling Method for the same - Google Patents

Abstract

The present invention relates to a refrigerator and a control method thereof and, more specifically, to a refrigerator and a control method thereof, which can provide information on food stored in the refrigerator even when a user does not open a door. According to one embodiment of the present invention, the refrigerator comprises: a storage compartment; a drawer provided to be movable in the storage compartment, and provided with a marker; a camera photographing the drawer from the outside of the drawer; and a control unit detecting the position of the marker from a picture continuously photographed through the camera, and determining state information of the drawer, which includes at least one of the drawing degree, a drawing status, a movement direction, a stopping state, or a moving state of the drawer.

Description

Refrigerator and Controlling Method for the same

The present invention relates to a refrigerator and a control method thereof, and more particularly, to a refrigerator capable of providing information on food stored in the refrigerator even if a user does not open a door, and a control method thereof.

In general, a refrigerator is a device for supplying cold air generated by driving of a refrigeration cycle, and is a device for storing food in a low-temperature state. In the case of a conventional refrigerator, only the function of storing food in a low-temperature state could be performed. However, in recent years, the need for additional functions in addition to the food storage function is increasing.

A refrigerator is a device that accommodates and stores certain items, and in order to check the interior, there is no choice but to open the refrigerator door to check the inside. In addition, when a user attempts to buy a product at a market or a mart, when he cannot grasp the amount and type of food stored in the refrigerator, there is an inconvenience in that he cannot buy the same food repeatedly or buy the necessary food.

According to the prior art Japanese Patent Laid-Open Patent No. 3450907 and Japanese Patent Laid-Open Publication No. 2004-183987, a camera is installed on a door to photograph the inside of a refrigerator. In addition, Japanese Patent Laid-Open Publication No. 2001-294308 discloses a technology in which a camera is installed in a cabinet, a drawer, and a door, respectively.

However, according to the reviewed prior art, there is a problem in that a plurality of cameras must be used in order to photograph various storage areas because the shooting range of the camera installed in the refrigerator is limited.

Therefore, while reducing the number of cameras installed in the storage room, it is necessary to increase the shooting efficiency of the camera by increasing the range of the area photographed by the camera.

In addition, according to the prior art, there are various problems in obtaining an image of the inside of the drawer. For example, a specific method of acquiring a picture taken at a desired point in time, a concern about condensation generated in the camera, and a concern for improving power consumption by the camera have not been made in detail.

Specifically, according to the prior art, when there are a plurality of storage areas including drawers in the storage, there is a problem in that specificity for providing optimal information to a user in consideration of the specificity and location relationship of each area is lacking.

The present invention basically aims to solve the above-described problem.

Through an embodiment of the present invention, as to solve the above problems, the present invention provides a storage compartment display device of a refrigerator and a control method thereof that provides information on food stored in the refrigerator even if the user does not open the refrigerator door. To provide.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator through which a user can intuitively recognize information on food stored in each of a plurality of storage areas in a storage room, and a control method thereof.

In addition, through an embodiment of the present invention, a user can intuitively recognize the positional relationship between a plurality of storage areas, and a refrigerator capable of providing stored food information substantially consistent with the user's visual point of view, and its We want to provide a control method.

In addition, through an embodiment of the present invention, by linking the driving time of the camera and the shooting time of the camera with the opening of the door and/or the rotation angle of the door, power consumption due to the camera is reduced, and food through the camera at the optimal time point. An object of the present invention is to provide a refrigerator capable of acquiring storage information and a control method thereof.

In addition, through an embodiment of the present invention, it is intended to prevent the quality of a picture taken by a camera from deteriorating due to condensation.

In addition, through an embodiment of the present invention, it is possible to provide the user with the latest information on the food stored in the refrigerator.

In addition, through an embodiment of the present invention, it is intended to provide the user with food storage information in an optimal state by allowing the administrator or the user to directly correct the area of the picture displayed to the user.

In addition, through an embodiment of the present invention, by installing a camera on a fixed body to prevent the camera from shaking, it is intended to provide a refrigerator capable of providing optimal food storage information to a user and a control method thereof.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof capable of effectively grasping and providing food storage information in a drawer while simplifying the configuration by grasping the state information of a drawer in software.

In addition, through an embodiment of the present invention, a refrigerator capable of providing the latest food storage information for each of a plurality of storage areas inside a storage room including a drawer through continuous shooting through one camera and a control method thereof is provided. I want to.

In addition, through an embodiment of the present invention, food stored in another storage area in which the inner space of the drawer and at least part overlap vertically or in another interior space of the drawer in which at least a part of the drawer overlaps vertically. It is intended to provide a refrigerator capable of providing storage information using a single camera and a control method thereof.

In addition, through an embodiment of the present invention, it is possible to prevent the quality of a picture taken by the camera from deteriorating due to condensation.

In addition, according to the exemplary embodiment of the present invention, a refrigerator capable of reducing the cost of the refrigerator may be provided by limiting the number of cameras installed in the storage unit to one.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof that can effectively prevent an increase in power consumption due to a camera.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof that can minimize the load of the control unit and the memory unit and can effectively and efficiently process continuously photographed pictures.

In order to implement the above object, according to an embodiment of the present invention, there is provided a storage compartment; A drawer provided to be movable in the storage compartment and provided with a marker; A camera that photographs the drawer from the outside of the drawer; And, by detecting the position of the marker in the photographs continuously taken through the camera, the drawer status information including at least one of the draw degree, the withdrawal status, the moving direction, the stationary state, and the moving state is determined. A refrigerator comprising: a control unit to be provided.

The control unit may only perform image processing on an area within a preset range in which the marker may be located in a single photo.

The control unit may determine a coordinate value of the marker in the photographed picture from reference coordinates of a path in which the marker is movable.

The reference coordinate may be a one-dimensional coordinate parallel to a moving direction of the drawer. For example, if the coordinate value in the state in which the drawer is closed and immediately before the drawer is effectively opened is 0, it may be determined that the drawer is further opened as the coordinate value increases.

The controller may compare two consecutively photographed pictures, and determine whether the drawer is moved or not and a moving direction according to a change in the position of the marker. For example, when the coordinate value is increased, it may be determined that the drawer is being opened, and in the opposite case, it may be determined that the drawer is being closed. In addition, when the coordinate value is unchanged, it may be determined that the drawer is in a stopped state.

In the section (first section) from the drawer to a state in which the drawer is drawn in by a predetermined distance (the first section), the control unit determines the position of the marker in a single photo, and reaches a state in which the drawer is drawn from the predetermined distance to the maximum withdrawn state. In the section (second section), the control unit may determine a change in the position of the marker by comparing two consecutive pictures.

It is preferable that the second section is equal to or smaller than the length of the first section.

It is preferable that the photographing time interval of the camera is equal to or longer than the time it takes for the controller to compare the positions of the markers located in the second section in two consecutive photos.

It is preferable that the photographing time interval of the camera is equal to or longer than the time required for the control unit to determine the position of the marker.

The marker may be disposed on the upper surface of the front wall of the drawer.

The drawer includes a drawer disposed on a left side and a drawer disposed on a right side from a left and right center of the storage compartment, and the marker may be installed on each drawer.

The camera is fixed to the upper ceiling of the drawer at the left and right center positions of the storage compartment, and the marker may be provided in each of the two drawers so as to be close to the left and right center positions of the storage compartment.

The marker may include a color different from the color of the drawer located at the border of the marker. Through this, it is possible to recognize the marker more effectively.

The marker preferably includes a vertical side defined parallel to a moving direction of the drawer and a horizontal side defined perpendicular to the moving direction of the drawer, and the horizontal side is preferably longer than the vertical side.

It is preferable that the marker includes a first color part and a second color part having a different color from the first color part, and the first color part and the first color part are alternately arranged.

It is preferable that the markers are repeatedly arranged and include at least one shape having different colors.

In order to implement the above object, according to an embodiment of the present invention, there is provided a storage compartment; A drawer provided to be movable under the storage compartment and provided with a marker; A camera provided so as to be fixed to the ceiling of the storage room to photograph the drawer from the outside of the drawer; And, by detecting the position of the marker in the photographs continuously taken through the camera, the drawer status information including at least one of the draw degree, the withdrawal status, the moving direction, the stationary state, and the moving state is determined. A refrigerator comprising: a control unit to be provided.

The control unit may detect that the position of the marker in the continuously photographed picture is changed based on the position of the marker in the state in which the drawer is completely retracted, and may determine the state information of the drawer.

In order to reduce the load of the control unit in the section from the drawer in state to the state where it is withdrawn by a predetermined distance (the first section) and the section from the state in which the drawer is drawn in the predetermined distance to the state in which the drawer is maximum (second section), The method of tracking the marker and processing the image in the control unit and whether or not to store the photographed image may be different.

Among the pictures taken in the second section, a picture taken just before the drawer is closed may be captured and provided to the user. That is, at this time, a picture for a corresponding region may be updated using the photographed picture.

It is preferable that the second section is shorter than that of the first section.

In the second section, the position of the marker may be detected by sequentially comparing two consecutively photographed pictures.

In order to achieve the above object, according to an embodiment of the present invention, a storage compartment formed by a fixed heat insulating wall and provided with an opening in a main body of a refrigerator; A door rotatably provided in the main body to open and close the opening; A camera that is provided to be fixed to the storage compartment and photographs the inside of the storage compartment to update food information in the storage compartment; A door switch for sensing opening and closing of the door; A door sensor detecting an opening angle of the door; And, according to the detection result of the door switch, the time point of switching between the standby mode of the camera and the driving mode for starting the photographing is determined, and, according to the detection result of the door sensor, a valid photo for updating among the photographed photographs. A refrigerator including a control unit for determining a reference time point for selecting a may be provided.

In order to achieve the above object, according to an embodiment of the present invention, a storage compartment formed by a fixed heat insulating wall and provided with an opening in a main body of a refrigerator; A door rotatably provided in the main body to open and close the opening; A drawer provided in the storage room; The ceiling of the storage room so as to photograph an area in which food is stored in the outer space of the drawer in the storage room (a first area) and an area in which food is stored in the inner space of the drawer in the storage room (a second area) A camera provided to be fixed to the camera; And, through the photographing point of the picture in which the first area and the second area are taken together, a portion of the first area (a picture of the first area) and a portion of the second area (a picture of the second area) are Separately, it may include a control unit that separates and stores individually.

In order to achieve the above object, according to an embodiment of the present invention, a storage compartment formed by a fixed heat insulating wall and provided with an opening in a main body of a refrigerator; A door rotatably provided in the main body to open and close the opening; A drawer provided in the storage room; The ceiling of the storage room so as to photograph an area in which food is stored in the outer space of the drawer in the storage room (a first area) and an area in which food is stored in the inner space of the drawer in the storage room (a second area) A camera provided to be fixed to the camera; Separation of a part for the first area (photo of the first area) and a part for the second area (photo of the second area) in the photo through the time point of the photo in which the first area and the second area are taken together Thus, a control unit for individually classifying and storing; In addition, it is possible to provide a refrigerator including a display that displays the first region picture and the second region picture to be distinguished from each other.

In order to achieve the above object, according to an embodiment of the present invention, a storage compartment formed by a fixed heat insulating wall and provided with an opening in a main body of a refrigerator; A door rotatably provided in the main body to open and close the opening; A drawer provided in the storage room; The ceiling of the storage room so as to photograph an area in which food is stored in the outer space of the drawer in the storage room (a first area) and an area in which food is stored in the inner space of the drawer in the storage room (a second area) A camera provided to be fixed to the camera; The second area in the picture taken at a specific point in time when the drawer is drawn in and then drawn in is separated from the picture taken at a specific point in time when the door is opened and closed. A refrigerator may be provided, comprising: a controller configured to separate, separate, store, and update the first region photo and the second region photo separately from each other by separating a portion (second region photo) of the image.

In order to achieve the above object, according to an embodiment of the present invention, a storage compartment formed by a fixed heat insulating wall and provided with an opening in a main body of a refrigerator; A door rotatably provided in the main body to open and close the opening; A drawer provided in the storage room; At least one shelf provided in the storage compartment and positioned above the drawer; And to photograph the area (first area) where food is stored by the shelf in the outer upper space of the drawer and the area (second area) where food is stored in the inner space of the drawer in the storage room, A refrigerator including a camera provided to be fixed to the ceiling of the storage compartment between the front end of the shelf and the opening may be provided.

In the above-described embodiments of the present invention, each feature may be implemented in combination with each other unless contradictory or exclusive.

According to the present invention, it is not necessary for the user to open the door to obtain information on the food stored in the refrigerator, so that leakage of cold air contained in the storage compartment can be prevented. Therefore, it is possible to improve the energy efficiency of the refrigerator by preventing unnecessary loss of cold air.

In addition, in the present invention, since the latest information on food stored in the refrigerator can be provided to the user, reliability of the food information provided to the user can be improved.

In addition, in the present invention, since information on food stored in various locations can be provided by one camera, only a structure for installing one camera is added, so that design can be made easily. In particular, it is possible to reduce the cost incurred by the use of the camera.

According to the present invention, condensation is prevented on the camera installed in the refrigerator, so that a picture taken by the camera can be stably provided to the user.

According to the present invention, an individual can remotely grasp the current internal situation of a refrigerator, and can receive food-related information necessary for the individual from an external business operator.

In addition, according to the present invention, a screen similar to the image captured by the camera for the inside of the drawer can be provided as the user actually sees when using the refrigerator.

In addition, according to the present invention, a location that cannot be viewed at a time can be provided to a user as a flat image on a single screen because they are overlapped due to spatial arrangement.

1 is a front view of a refrigerator according to the present invention.
Figure 2 is a view in which the door is opened in Figure 1;
3 is a view showing a third area disposed on the floor of the storage room.
4 is a control block diagram of an embodiment according to the present invention.
5 is a view for explaining the angle of view of the camera.
Figure 6 is a side cross-sectional view of Figure 5;
7 is a screen taken by the camera in the state of FIG. 6.
8 is a diagram for explaining selection of a location of a camera.
9 is a cross-sectional view showing a main part of the refrigerator.
10 is a view for explaining an operation method of the door sensor.
11 is a view showing in detail a left hinge portion and a right hinge portion.
12 is a screen provided to a user.
13 is a diagram for explaining a method of adjusting a picture taken by a camera.
14 is a perspective view showing a camera.
15 is a view showing a main part of the camera.
16 is a cross-sectional view of FIG. 14;
Fig. 17 is a diagram explaining the arrangement of heaters.
18 is a view illustrating a picture taken while the camera is mounted in a refrigerator in an up-down direction.
19 is a view for explaining a picture taken while the camera is rotated in a horizontal direction in a refrigerator.
20 is a view showing a state in which the camera housing is assembled.
21 is a front view of the first housing.
22 is a front view of the second housing.
23 is a front view showing a state in which the camera is installed in the first housing.
Fig. 24 is a side cross-sectional view of Fig. 23;
25 is a view showing a cross-section of the camera housing installed in the inner case.
26 is a table comparing power consumption of cameras in the present invention.
Fig. 27 is a diagram comparing supplied standby current and driving current.
28 and 29 are views for explaining a starting point of photographing by the camera and continuous shooting by the camera.
30 is a diagram showing an embodiment of a drawer sensing unit.
FIG. 31 is a diagram illustrating a method of detecting movement of a drawer by the drawer detection unit of FIG. 30;
Fig. 32 is a diagram showing a state in which a marker is displayed on a drawer.
Figure 33 (a) is a view showing a picture taken in a state in which the left and right drawers are withdrawn.
Fig. 33(b) is a diagram showing a state in which the left drawer is pulled in and the right drawer is pulled out.
34 is a flowchart illustrating a method of operating a refrigerator according to an embodiment of the present invention.
35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.
36 is a view for explaining a marker for detecting a drawer withdrawal point according to an embodiment of the present invention.
37 is a view for explaining the shape of a marker according to an embodiment of the present invention.
38 is a flowchart illustrating a method of detecting a drawer closing operation according to an embodiment of the present invention.
39 is a flowchart illustrating an operation of a refrigerator performed when a drawer closing operation is detected completion according to an embodiment of the present invention.
40 is a flowchart illustrating a method of controlling an image of a specific area inside a refrigerator photographed at a specific point in time according to an embodiment of the present invention.
41 is a diagram showing various types of markers.
42 is a diagram for explaining a method of recognizing a marker position.
43 is a view showing the degree to which the drawer is opened.
Fig. 44 is a diagram explaining movement of a marker.
45 is a control flowchart according to an embodiment of the present invention.
46 is a control flowchart for explaining a modification example of FIG. 45;
47 is a control flowchart for explaining another modified example of FIG. 46;
48 is a control flowchart for explaining another modified example of FIG. 45;
49 is a modified example in which FIG. 45 is modified differently.
50 is a control flowchart according to another embodiment of the present invention.
51 is a view for explaining a process of updating an image in a state in which two doors and two drawers are provided to open and close a storage room.
52 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
53 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.
54 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.
55 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.
56 is a view for explaining the operation of the heater of the camera.
57 is a view for explaining an experiment result of condensation generated by temperature in a transparent window of a camera.
58 is a cross-sectional view of a transparent window.
59 and 60 are views schematically showing a structure in which a camera is installed in an inner case.
61 is a view showing another type of refrigerator to which the present invention is applicable.
62 is a screen provided to a user from the refrigerator according to FIG. 61;
FIG. 63 is a view for explaining a method of adjusting a picture taken by a camera in the refrigerator according to FIG. 61;

Hereinafter, preferred embodiments of the present invention capable of realizing the above objects will be described with reference to the accompanying drawings.

1 is a front view of a refrigerator according to the present invention, and FIG. 2 is a view in which a door is opened in FIG. 1. Hereinafter, it will be described with reference to FIGS. 1 and 2.

The refrigerator according to the present invention includes a top mount-type in which a freezer compartment in which food is stored and a refrigerator compartment is divided up/down so that the freezer compartment is disposed on the upper side of the refrigerator compartment, and the freezer compartment and the refrigerator compartment are divided into left/right sides. The same can be applied to side-by-side-type refrigerators.

However, in this embodiment, for convenience of explanation, a bottom freezer-type in which the freezing chamber and the refrigerating chamber are divided up/down and the freezing chamber is disposed below the refrigerating chamber will be described.

The main body of the refrigerator includes an outer case 10 forming an overall appearance when viewed by a user from the outside, and an inner case 12 forming a storage compartment 22 in which food is stored. A predetermined space may be formed between the outer case 10 and the inner case 12 to form a passage through which cool air is circulated. Meanwhile, an insulating material is filled between the outer case 10 and the inner case 12 so that the inside of the storage compartment 22 can maintain a relatively low temperature compared to the outside.

In addition, in a machine room (not shown) formed in a space between the outer case 10 and the inner case 12, a refrigerant cycle device for circulating a refrigerant to generate cool air is installed. The freshness of foods stored by keeping the inside of the refrigerator at a low temperature using a refrigerant cycle device can be maintained. The refrigerant cycle device includes a compressor for compressing a refrigerant, and an evaporator (not shown) that converts a liquid refrigerant into a gaseous state to exchange heat with the outside.

The refrigerator is provided with doors 20 and 30 for opening and closing the storage compartment. In this case, the doors may each include a freezing compartment door 30 and the refrigerating compartment door 20, and each of the doors is rotatably installed on the main body of the refrigerator by a hinge at one end thereof. The freezing compartment door 30 and the refrigerating compartment door 20 may be formed in plural. That is, as shown in FIG. 2, the refrigerating compartment door 20 and the freezing compartment door 30 may be installed in a form that is opened around both corners of the refrigerator toward the front.

A foaming agent is filled between the outer case 10 and the inner case 12 so that insulation may be insulated between the outside and the storage compartment 22.

The storage compartment 22 forms a space insulated from the outside by the inner case 12 and the door 20. When the door 20 closes the storage compartment 22, the storage compartment 22 may form a space that is isolated from the outside and insulated. In other words, the storage compartment 22 is formed by the door 20. It can be said to be a space that is isolated from the outside through the insulating wall and the insulating wall by the case (10, 12).

The cold air supplied from the machine room can flow in various places in the storage room 22, so that the food stored in the storage room 22 can be maintained at a low temperature.

The inner case 12 may constitute a barrier 60 on the bottom of the storage chamber 22. A barrier 60 is installed at the lower end of the storage compartment 22 to separate the freezing compartment and the refrigerating compartment. The barrier 60 may have a predetermined thickness and may be formed on the inner case 12. The barrier 60 may extend in a horizontal direction.

The storage compartment 22 may include a shelf 40 on which food is mounted. In this case, a plurality of shelves 40 may be provided, and food may be mounted on each shelf 40. The shelf 40 may divide the interior of the storage compartment in a horizontal direction.

In the storage chamber 22, a drawer 50 capable of withdrawing or drawing in is installed. Food and the like are accommodated and stored in the drawer 50. Two drawers 40 may be disposed in the storage compartment 22 to the left and right. The user may open the left door of the storage compartment 22 to access the drawer disposed on the left. On the other hand, in order for the user to access the drawer disposed on the right, the right door of the storage compartment 22 may be opened.

A space in which food is stored may be formed in the barrier 60. Since the barrier 60 is not provided on the door but is provided on the inner case 12, it does not move according to the rotation of the door and remains fixed. Therefore, the user can stably store or withdraw food.

The storage compartment 22 is divided into a space located on the upper side of the shelf 40, a space formed by the drawer 50, a space formed in the barrier 60, etc., and a space in which food is stored. It can be divided into a plurality of these.

At this time, each space is a storage space disposed inside one storage compartment 22, and cold air supplied to the storage compartment 22 may be moved to each space. In other words, since each space is partitioned to the extent that cold air can move from each other, it has a different meaning from the storage room described above.

In particular, unlike a storage compartment that forms an insulated space, each space may have a temperature difference in one storage compartment, but does not form a space that is insulated from each other.

The cold air supplied to one storage room does not move freely to another storage room, but the cold air supplied to one storage room can freely move to each partitioned space installed inside one storage room. That is, the cold air located on the upper side of the shelf 40 is movable to the space formed by the drawer 50.

In addition, the present invention may include one camera 70 for taking a picture of the inside of the storage compartment 22. At this time, the camera 70 is provided to maintain a fixed position, so it is possible to photograph the same part.

In particular, the camera 70 is installed downward on the upper wall of the inner case 12 so that a picture of the food stored in the storage compartment 22 can be taken. The photographed picture may have a shape viewed by the user, that is, a shape viewed by the user when using an actual refrigerator.

In particular, the camera 70 is installed at a position corresponding to the inside of the drawer when the drawer is fully open, so that the captured picture can provide a screen similar to that of the user viewing the inside of the drawer from above.

In this case, the storage compartment 22 may include a first area 42 in which food is placed on the shelf 40 and a second area 52 defined by an inner space of the drawer 50. In addition, when the drawer 50, which is a storage space different from the first region and the second region, is drawn out, a third region 62 overlapping the second region 52 may be included. This part will be described in more detail later.

The first area 42 and the second area 52 may have different heights as a whole within the storage chamber 22. That is, the first region 42 and the second region 52 may be lowered in the order.

Photos taken by the camera 70 or various information about the refrigerator may be provided to the user through the display 14 provided on the front of the refrigerator. In addition, the user can control the refrigerator through the display 14.

The cold air supplied to the storage compartment 22 may be moved to the first region 42 and the second region 52, and the cold air located in each region may also be moved to other regions.

3 is a diagram illustrating a third area disposed on the floor of the storage room. Hereinafter, it demonstrates with reference to FIG.

The barrier 60 may be provided with a recessed portion so that food may be stored therein, and this area may be referred to as the third area 62. The third area 62 is a space formed in the barrier 60.

In this case, the third area is a storage space disposed at a location different from that of the first area and the second area, and may provide a user with another space for food storage.

The third area 62 may store foods with a high frequency of use such as eggs, and for this purpose, it is possible to provide a cover 68 for opening and closing the third area 62. At this time, it is preferable that the cover 68 is made of a transparent material, so that the food stored in the third area 62 can be photographed even if the cover 68 is covered when the camera 70 is photographed from the top. .

Meanwhile, the drawer 50 is disposed above the barrier 60. Accordingly, when the drawer 50 is inserted into the lower space of the shelf, the user can access the third area 62.

On the other hand, when the drawer 50 is pulled out from the lower space of the shelf, the drawer 50 is disposed above the third area 62, and the user approaches the third area 62. Can not.

That is, when the drawer 50 is drawn toward the user, the drawer 50 and the third area 62 overlap each other. In this case, when the user looks down the drawer 50, the third area 62 is not visible to the user, and the inside of the drawer 50 may be seen.

On the other hand, when the drawer 50 is not drawn toward the user, the drawer 50 and the third area 62 are disposed so as not to overlap each other. When the user looks down the third area 62, the inside of the drawer 50 is not visible to the user, and only the third area 62 may be shown.

The first area 42, the second area 52, and the third area 62 may have different heights in the storage chamber 22 as a whole. That is, the first region 42, the second region 52, and the third region 62 may be lowered in order.

The cold air supplied to the storage compartment 22 may be moved to the first region 42, the second region 52, and the third region 62, and the cold air located in each region is transferred to another region. Can also be moved.

That is, an area photographed by the camera 70 may be a plurality of storage spaces inside the storage chamber 22. The camera 70 takes a picture of the inside of one storage room 22, but information on a plurality of partitioned storage spaces may be included in the inside of one picture.

In this case, the camera 70 may take a picture so that a plurality of storage spaces are included in a single picture. In particular, it is possible to take different storage spaces for one picture depending on the time it is taken.

For example, a camera arranged in a fixed form captures a first area and a second area, or selectively selects a second area and a third area according to the relationship between the time when the picture is taken and the position of the door or the drawer. It is possible to shoot with. Details about this will be described later.

4A is a control block diagram of an embodiment according to the present invention. Hereinafter, it demonstrates with reference to FIG. 4A.

The present invention may include a control unit 100 that divides and manages one picture taken by the camera 70 into a plurality of images. In this case, the image may mean a part of the photo processed or corrected by the control unit 100.

The controller 100 may command the camera 70 to take a picture, and may receive a picture taken by the camera 70.

In addition, the controller 100 may provide a part of the divided image to the display 14 to provide the user with the latest information on the food stored in the storage compartment 22. In this case, the display 14 may be installed on the front of the refrigerator, or may form a separate device separated from the refrigerator. That is, a user can receive an image of the storage room 22 and obtain information through an external communication terminal such as a mobile phone.

In this case, the controller 100 may divide the picture taken by the camera 70 for each area, divide it into a plurality of independent images, and provide it to the display 14. At this time, the picture selected by the control unit 100 may include information on the state of the storage room in which the user finally accesses the storage room 22 and withdraws food from the storage room or puts food in it.

An embodiment of the present invention may include a door switch 110 capable of detecting whether the door 20 opens or closes the storage compartment 22. At this time, the door switch 110 is provided in the outer case 10, and when the door 20 comes into contact with the outer case 10, it is sensed that the door 20 seals the storage compartment 22. can do. In addition, when the door 20 does not contact the outer case 10, it may be detected that the door 20 opens the storage compartment 22.

In addition, an embodiment of the present invention may include a door sensor 120 that detects a rotation angle of the door 20. In this case, the door sensor 120 may detect a rotation direction of the door 20 and a rotation angle of the door 20. For example, when the door 20 is rotated by a certain angle or more, a change occurs in the door sensor 120, so that the door sensor 120 can detect that the door 20 is rotated by a certain angle or more. have. In addition, when the door 20 is rotated in a specific direction, it is possible to detect a change in pulse generated according to the direction to detect the rotation direction of the door 20. Of course, the door sensor 120 may be modified in various forms other than the above-described method.

In addition, the door sensor 120 may include a light-emitting unit and a light-receiving unit to determine whether or not light irradiated from the light-emitting unit is transmitted to the light-receiving unit and detect the rotation angle of the door 20.

In particular, the door sensor 120 may be driven only when the door switch 110 determines that the door 20 has opened the storage compartment 22.

Meanwhile, when the door sensor 120 determines that the door 20 is rotated at a specific angle, the camera 70 may take a picture. In this case, even if the door switch 110 detects the opening of the door 20, the camera 70 does not immediately start taking a picture, but when the door sensor 120 reaches a specific angle, the picture Will be filmed.

When the camera 70 starts to take pictures, it is possible to continuously take pictures at predetermined time intervals. Of course, if the door sensor 120 determines that the door 20 is open by a certain angle, the camera 70 can continue to take a picture until a command to stop taking a picture is provided instead of a single picture. Do.

The control unit 100 may include a drawer detection unit 130 that detects whether the drawer 50 is withdrawn or pulled in. In this case, the drawer detection unit 130 may mean a portion in which the picture taken by the camera 70 reads an image in the control unit 100. The drawer detection unit 130 may detect the movement of the drawer 50 as well as the direction in which the drawer 50 moves.

In particular, the drawer detection unit 130 may be implemented by software. The drawer detection unit 130 may detect the position of the drawer 50 by using information captured from a picture taken by the camera 70.

The drawer 50 is provided with a recognition mark as a marker, and whether the drawer 50 enters a specific section, stops if it enters, or changes the direction of movement after stopping, through the picture taken by the marker. You can get information about it. The marker will be described in detail later.

Since the marker is marked on the drawer, it has the same movement as the drawer 50. Therefore, without providing a separate sensor for detecting the movement of the drawer, it is possible to obtain various information on the movement of the drawer 50 by using the information of the picture taken by the camera 70.

The picture taken by the camera 70 may be stored in the storage unit 18. In this case, the storage unit 18 may be provided in a refrigerator or may be provided in a device separate from the refrigerator. In this case, the storage unit 18 may be disposed in a refrigerator together with the display 14, in a device other than a refrigerator, a server connected to the refrigerator and a network, or a terminal connected to the refrigerator and a network together with the display 14. have.

In this case, the storage unit 18 may not store all photos taken by the camera 70. For example, when the control unit 100 does not issue a command to select a specific photo, the storage unit 18 may delete a previously captured photo and store the currently captured photo (First In , First Out). The storage unit 18 stores only some of the pictures taken by the camera 70, thereby reducing a storage capacity.

4B is a control block diagram of a modified form of an embodiment according to the present invention. Hereinafter, it demonstrates with reference to FIG. 4B.

In FIG. 4B, unlike FIG. 4A, the drawer detection unit 130 may separately detect the movement of the drawer 50 without using information acquired from a picture taken by the camera 70.

For example, the drawer sensing unit 130 may have the same shape as a plurality of Hall sensors. A plurality of Hall sensors may be installed in the movement path of the drawer 50. When the drawer 50 is moved, it is possible to sense the change of each of the plurality of Hall sensors to detect the position and the moving direction of the drawer 50.

The drawer detection unit 130 detects the movement of the drawer, and when a picture is taken by the camera 70, it is possible to grasp the point of time at which the inside view of the drawer is included in the picture taken by the camera 70. . That is, the drawer detection unit 130 may detect a point in time at which information about an article stored inside the drawer can be expressed in a picture taken by the camera.

The drawer detection unit 130 may detect the movement of the drawer and detect a point in time when the user determines that the drawer is in the latest state after using the drawer. That is, the drawer detection unit 130 may detect a point in time including the latest information on which the user has finished using the drawer in the picture taken by the camera. That is, it is possible to indirectly grasp the completion time of the receipt and release of the product or the end of the product update.

5 is a view for explaining the angle of view of the camera. Hereinafter, it demonstrates with reference to FIG.

In FIG. 5, for convenience of explanation, the drawer 50 is withdrawn so that the camera 70 can photograph the second area 52, and the shelf is omitted.

The camera 70 may be disposed on an upper wall of the inner case 12 so as to be inclined toward a rear wall of the inner case 12.

Meanwhile, a picture taken by the camera 70 may have a horizontal angle of view according to the width direction of the storage compartment 22 and a vertical angle of view according to the front-rear direction.

At this time, the horizontal angle of view according to the camera 70 is provided so that the inner space of the two drawers 50, that is, the food stored in the second area 52, can be seen while the two drawers 50 are drawn out. It is desirable.

In addition, the range of the horizontal angle of view according to the camera 70 may include at least some of the left and right ends of the shelf 40. This is because in order to obtain information on the food mounted on the upper side of the shelf 40, it is preferable to take pictures from the camera 70 to both ends of the shelf 40.

As the angle of view of the camera 70 changes, the range inside the storage compartment shown in the picture changes.

6 is a side cross-sectional view of FIG. 5, and FIG. 7 shows a screen taken by a camera in the state of FIG. 6. Hereinafter, it will be described with reference to FIGS. 6 to 7.

An opening 14 is formed in the front of the inner case 12 so that the user can access the storage compartment 22. Through the opening 14, the user can put food into the storage compartment 22 or take out food from the storage compartment 22. The door 20 may open and close the opening 14.

The camera 70 may be installed outside the drawer to take a picture while the drawer is moving or stopped to take a picture of the drawer's open state, the closed state, and the moving state. That is, when the camera 70 continuously photographs, a state in which the position of the drawer is changed according to the photographing time may be expressed in the photograph taken by the camera 70.

In addition, the camera 70 may provide a screen including the latest in/out status of the items inside the storage room 22 to the user.

The camera 70 is preferably installed within a range (l) between the opening 14 and the front end of the shelf 40. When there are a plurality of shelves 40, it is preferable that they are installed in a region (l, see FIG. 8) between the front end and the opening 14 located at the most front of the front ends. That is, it is preferable to be installed on the ceiling of the storage room corresponding to this area. Since the camera 70 has to selectively take pictures of the first area and the second area 52 or the first area and the third area 62, more than the front end of the shelf 40 It cannot be moved to the rear wall of the inner case 12.

6 shows the vertical angle of view of the camera 70, and it is preferable that one end (front) of the vertical angle of view is arranged so that the drawer 50 can be photographed from the open state to the front end of the drawer 50 Do. In addition, it is preferable that the other end (rear end) of the vertical angle of view is arranged so that the rear end of the most uppermost shelf can be photographed.

Since the second area 52 by the drawer 50 is included in the range of the vertical angle of view of the camera 70, the camera 70 provides the user with information on the second area 52 Can be obtained.

As shown in FIG. 7, the camera 70 may take pictures of the first area 42 and the second area 52 as a single picture. In this case, the camera 70 may take pictures of the two drawers 50 disposed on the left and right sides. In addition, when a plurality of shelves are provided, it can be seen that the first area 42 and the second area 52 divided by the plurality of shelves can be photographed together.

The camera 70 is installed inside the storage room 22 to take a picture including information about the inside of the storage room 22. Since the camera 70 is installed inside the storage room, it cannot be moved. That is, since it is generally fixedly installed in the storage compartment of the main body that is always fixed, the camera itself does not move. Therefore, when the camera 70 takes a picture, the camera 70 is always in a stopped state.

The picture taken by the camera 70 includes the first area 42 and the second area 52, or includes the first area 42 and the third area 62.

When the camera 70 takes a picture including the first area 42 and the second area 52, the user can use the picture to display food and food stored on the upper side of the shelf 40. Information on food stored in the lower 50 may be obtained.

On the other hand, when the camera 70 takes a picture including the first area 42 and the third area 62, the user will Information on food stored in the barrier 60 may be obtained.

The camera 70 photographs the inside of one storage room 22 and transmits the photographed picture, and the controller 100 may obtain information on a plurality of storage spaces.

The camera 70 is disposed above the first area 42, the second area 52, and the third area 62 to take a picture while looking down.

8 is a diagram illustrating selection of a camera position. Hereinafter, it demonstrates with reference to FIG.

As shown in FIG. 8, the camera may be mounted at positions L1, L2, L3, and L4.

L1 is an external space partitioned by the outer case 10 to the outside of the storage compartment 22. When the camera is installed at the position of L1, the user accesses the storage compartment 22 with the door open and uses the refrigerator, but there may be a problem that the camera is covered by the user.

As disclosed in the above published Japanese patent, when a camera is installed outside the refrigerator to take a picture inside the storage room, the image interference by the user is too great, and in particular, the drawer is covered so that the movement of the drawer cannot be tracked. In the embodiment of the present invention, it was impossible to grasp the drawer internal image and the motion state of the drawer together. In addition, the L1 position may mean that the camera is located on the door. In this case, since the door is configured to move, the camera may be moved during photographing. Therefore, it can be said that it is very difficult to obtain a stable and clear picture.

L2 is the interior of the storage compartment 22 and refers to an upper wall of the storage compartment 22, that is, a ceiling. In the embodiment of the present invention, it corresponds to the location where the camera is installed, but the problem of being obscured by the user does not occur. In addition, since the upper wall of the storage compartment 22 corresponds to the inner case 12, it is convenient to arrange an electric wire for supplying electricity to the camera. In addition, since the inner case 12 is generally fixed at all times, it can be stably mounted on the camera to be fixed.

L3 is an interior of the storage chamber 22 and is a sidewall of the storage chamber 22. At this time, the camera is installed so as to be inclined to take a picture from one side of the storage compartment 22 toward the other side of the storage compartment 22. In order for the camera to be disposed not at the top of the storage compartment 22 but in the middle based on the height of the storage compartment 22, it must be installed on the sidewall of the inner case 12.

When the camera is installed on one side wall, the camera lens must be installed to face the other side wall, and thus, there is a problem that an asymmetric image is obtained due to a difference in distance between the left and right sides of the interior of the storage compartment 22.

L4 is the interior of the storage compartment 22 and refers to one end of the shelf 40. In order for the camera to be disposed based on the height of the storage compartment 22, it must be supported by a specific structure, in which case the camera may be fixed to one end of the shelf 40.

In this case, since the distance between the drawer 50 and the camera is shorter than that of L2, there is a problem in that a camera having a larger angle of view than the state of L2 should be used. In addition, there is a problem that the degree of distortion of the obtained picture becomes severe. In addition, various problems that may cause inconvenience to the user may occur in appearance, such as installing an electric wire supplied to the camera on the shelf 40. In addition, the shelf 40 is a generally movable configuration. Therefore, there may be a problem in which the camera itself shakes when food is placed on a shelf.

The camera applied to the embodiment of the present invention has a field of view of 120 degrees, is a VGA class, and is capable of providing a performance of 20 fps.

9 is a cross-sectional view showing a main part of the refrigerator. Hereinafter, it demonstrates with reference to FIG.

A basket 21 for storing food may be provided inside the door 20. The basket 21 may be installed on a surface of the door 20 facing the storage compartment 22.

There may be a problem that the basket 21 is shown in a picture taken by the camera 70. Accordingly, in an embodiment of the present invention, it is possible to take a picture through the camera 70 by detecting whether the door 20 is rotated by a certain angle θ or more through the door sensor 120. Of course, it is also possible for the door sensor 120 to transmit information on the corresponding viewpoint to the control unit 100 so as to select a picture taken at a viewpoint adjacent to the viewpoint detected by the door sensor 120.

For example, it is possible for a certain angle θ to be between 60 degrees and 80 degrees. Of course, the angle may vary according to the capacity of the refrigerator, the size of the door, or the front and rear length of the basket 21.

At this time, a certain angle θ may mean an angle at which the basket 21 may not appear in the photo when the camera 70 takes a photo, and this certain angle θ is of the basket 21 It may vary depending on the left and right width and the length of the front and rear.

A door switch 110 that detects whether the door 20 is opened or closed may be installed on the upper side of the refrigerator. At this time, the door switch 110 detects that the door 20 seals the storage compartment when pressed by the door 20, and the door 20 opens the storage compartment when it is not pressed by the door 20. It can be detected that it is open.

A left hinge part 300 for rotatably installing the left door 20 on the refrigerator may be provided, and a right hinge part 320 for rotatably installing the right door 20 on the refrigerator may be provided. The left hinge part 300 and the right hinge part 320 are provided with a rotation shaft capable of rotating the door 20 and are coupled to the door 20, a part installed on the outer case 10, It can be divided into a connecting part connecting the two parts. That is, it may be divided into door coupling portions 300c and 302c, outer case coupling portions 300a and 320a, and connection portions 300b and 320b connecting them.

A left door sensor 120L for detecting the rotation of the left door 20 may be installed on the left hinge part 300.

A right door sensor 120R for detecting the rotation of the right door 20 may be installed on the right hinge part 320.

The left door sensor 120L and the right door sensor 120R may independently detect a rotation angle of each door.

10 is a diagram illustrating an operation method of a door sensor. Hereinafter, it demonstrates with reference to FIG.

The door sensor 120 may include a light-emitting unit 122 that irradiates light and a light-receiving unit 124 that receives light irradiated from the light-emitting unit 122.

The light irradiated from the light emitting unit 122 may be reflected through the upper surface of the door 20 and transmitted to the light receiving unit 124. In this case, it is also possible to be made of a material that reflects well so that the reflection of light can be stably made.

When the light irradiated from the light emitting unit 122 is transmitted to the light receiving unit 124, it may be detected that the door 20 is rotated by an angle smaller than the angle at which the door sensor 120 is installed. That is, it may mean that the door 20 is rotated at an angle smaller than a predetermined angle θ, or that the storage chamber 22 is not rotated while the storage chamber 22 is sealed.

On the other hand, if the light irradiated from the light emitting unit 122 is not transmitted to the light receiving unit 124, it may be detected that the door 20 is rotated by an angle greater than the angle at which the door sensor 120 is installed. That is, it may mean a state in which the door 20 is rotated to a state greater than a predetermined angle θ so that the user can access the storage compartment 22.

To this end, as shown in FIG. 9, the left hinge part 300 and the right hinge part 320 may be mounted across the top surface of the outer case 10 of the main body and the top surface of the door 20. Accordingly, as the door 20 rotates based on the rotation centers 302 and 322, a portion (eg, a stepped portion) of the top surface of the door or the top surface of the door may function as a light receiving unit. Of course, the left door sensor 120L and the right door sensor 120R may be installed at each of the connection units 300b and 320b to perform the function of the light emitting unit. Due to the positional relationship between the sensors 120L and 120R and the door, the sensors 120L and 120R can determine when the door is opened and opened at a certain angle (θ) or more and when the door is closed and closed at a certain angle (θ). I can.

11 is a view showing in detail the left hinge portion and the right hinge portion. In FIG. 11, other components of the refrigerator are omitted for convenience of description. Hereinafter, it demonstrates with reference to FIG.

The door sensor may be disposed in front of the inner case, that is, in front of the opening (ie, a position where the user is standing) in order to detect the rotation of the door.

A left door sensor 120L for detecting rotation of the left door 20 may be installed in the left hinge part 300.

A right door sensor 120R for detecting the rotation of the right door 20 may be installed on the right hinge part 320.

The left door sensor or the right door sensor may be disposed in a region between the opening and the rotation center of each of the hinge parts. That is, it may be disposed on the connection portions 300b and 320b of each hinge part.

The door sensor 120 may be connected to other components by a connector c and a wire in order to receive electricity or transmit a signal to the outside. In this case, the door sensor 120 may be connected to the above-described control unit 100 to provide signal information obtained.

The left hinge part 300 may have a rotation center 302 through which the left door 20 is rotated. In this case, the left door sensor 120L may be disposed on the right side with respect to the rotation center 302. The space on the right side from the rotation center 302 may or may not be located under the left door sensor 120L, in which the upper surface of the left door 20, especially the left door 20, is located according to the rotation angle of the left door 20. It is a space that can be. Therefore, it is preferable that the left door sensor 120L is located at the connection part 300b of the left hinge part 300. Through this, when the size of the door or the basket size is changed, the predetermined angle θ can be flexibly changed. That is, it is because the space on the right side of the connection part 300b can be easily changed, and accordingly, the position of the left door sensor 120L in the connection part 300b can be easily changed.

The wire w connected to the left door sensor 120L may be connected through the rotation center 302. In addition, the wire w may be connected to the refrigerator body through the outer case 10.

The right hinge part 320 may have a rotation center 322 through which the right door 20 is rotated. In this case, the right door sensor 120R may be disposed on the left side with respect to the rotation center 322. The left space from the rotation center 322 is a space in which the right door 20 may or may not be located under the right door sensor 120R according to the rotation angle of the right door 20. Therefore, for the same reason, it would be preferable that the right door sensor 120R is installed on the connection part 320b of the right hinge part 320.

The point of time sensed by the door sensor 120 may be a point of time for capturing a picture of each area inside the storage compartment 22, specifically, the first area 42 or the third area 62. . That is, the point of time sensed by the door sensor 120 may mean a point of time of capturing a picture of the drawer 50's external storage space.

Meanwhile, since the door sensor 120 irradiates and receives light, the door sensor 120 irradiates the light downward. The light irradiated downward is reflected upwardly by the door 20 again.

The door sensor 120 and the door 20 may maintain a short distance so that a sufficient amount of light can be received from the door sensor 120. At this time, the vertical distance between the door sensor 120 and the door 20 may be maintained at a maximum of 20 mm.

12 is an example of a screen provided to a user. Hereinafter, it demonstrates with reference to FIG. 12.

In FIG. 12A, the first area 42, the second area 52, and the third area 62 are all provided to the user, and FIG. 12B shows the second area 52 and the third area 62 The first area 42 and the second area 52 are provided to the user in FIG. 12C.

First, it will be described with reference to FIG. 12A.

One picture taken by the camera 70 shows a first image showing the first area 42, a second image showing the second area 52, and the third area 62. It can be classified as a third image. That is, one picture or a plurality of pictures may be divided by area. In this case, the first image, the second image, and the third image may mean a shape cut from a single photographed photo to a range showing a corresponding region.

The divided photos may be independently provided to the user as shown in FIG. 12A. That is, photos of the corresponding area are individually provided to the user, so that the user can easily obtain information on the food stored in the corresponding area.

The display 14 may be arranged so that a plurality of storage areas can be shown in a frame.

That is, a second image showing the second area 52 is disposed on the left and right sides of one rectangular frame, and a third image showing the third area 62 is disposed on the left side of the rectangular frame. And are placed on the right side respectively.

A plurality of regions may be arranged as an image having the same width in one frame on the display 14. To this end, the controller may cut the photographed picture, correct the size, and provide it to the display 14. At this time, the corrected image is shown at the corresponding position of the frame.

At least some of the images provided on the display are corrected to have the same width, so that the user has a feeling similar to the view of the storage room while the refrigerator door is actually opened, thereby having a planar arrangement effect.

The display 14 provides two overlapped regions through one screen, thereby providing food information in a state that an actual user cannot see at a time.

In particular, since the second image and the third image are spaces arranged to overlap when the drawer is fetched, the user cannot obtain information on the two storage spaces at the same time when the drawer is fetched. However, the display 14 may provide two pieces of storage space information at the same time.

The control unit 100 may individually update (replace) each image to provide a user with information on the type of food accommodated in each area.

For example, if a part corresponding to the second image is upgraded through a picture taken by the camera, and the part corresponding to the third image does not need to be upgraded, only the second image may be replaced.

On the contrary, when the part corresponding to the third image is upgraded through the picture taken by the camera, and the part corresponding to the second image does not need to be upgraded, only the third image may be replaced.

In one frame, the second image and the third image may be upgraded independently of each other and independent of other images.

And, of the two second images disposed on the left and right, only the second image disposed on the left side may be upgraded, or only the second image disposed on the right side may be upgraded. At this time, it is also possible to individually upgrade the third image on the left and right.

That is, the display 14 provides a frame in which a plurality of images are provided to provide information on an article stored in a storage room to a user, but the plurality of images can be individually upgraded regardless of other images. In particular, since the second image and the third image are images selected from different photographs, the controller may determine which region the corresponding photograph represents and upgrade each image.

In this case, the second image and the third image may have the same or substantially similar widths. That is, the portions corresponding to the second image and the third image acquired from the photo may be converted to the same size by the correction of the control unit and provided to the display.

When the sizes of the second image and the third image are different, the number of pixels of the second image and the third image, that is, the size of each pixel may be differently selected in the corresponding photo. Otherwise, the second image and the third image may be selected with the same number of pixels and corrected by the controller to have different sizes when displayed on the display 14.

By arranging the second image and the third image vertically, it is possible to have a flat feeling of the vertical arrangement. Originally, the area shown by the second image is disposed above the area shown by the third image. Therefore, the user can intuitively recognize the image provided by the display, and thus can easily grasp the information provided by the image.

Of course, the display 14 may provide the user with a screen in which the first image and the third image are simultaneously replaced. In contrast, while the first image and the third image remain the same, a screen in which only the second image is replaced may be provided to the user.

The refrigerator compartment shelf illustrated in FIG. 12A may refer to the first region 42, the vegetable compartment may refer to the second region 52, and the multi-storage corner may refer to the third region 62. Of course, it is also possible to change into various arrangements and various forms according to the user's preference or the convenience of the image provider.

Meanwhile, the screen of FIG. 12A may be provided to the display 14 installed in the refrigerator, but may also be provided to an external device other than the refrigerator, for example, an external terminal device such as a smartphone. Therefore, the user can obtain information on food stored in the refrigerator while going out from home, and can use it for shopping. Therefore, the user can grasp the type of food stored in the refrigerator even if the door is not opened. The screen provided to the user may be displayed by dividing one storage room into a plurality of storage areas. Accordingly, the user can easily obtain information classified by location of foods stored at each location in one storage room.

At this time, the image shown in the vegetable room shows a region from the front end of the drawer 50 to the front end of the shelf 40 in the vertical direction when the drawer 50 is withdrawn. The image shown in the multi-storage corner shows an area of a portion overlapping when the drawer 50 is retracted in a state that is not covered by the door 20. The shelf shows a region above the shelf 40 to the user.

In an exemplary embodiment of the present invention, since the screen provided to the user does not show a single storage room as it is, the user can easily obtain information because separate images are arranged in separate portions from each other. In other words, rather than using the picture shown in FIG. 7 as it is, it can be said that each area is separated from one picture and used independently and individually.

As shown in FIG. 12B, a screen having a different shape from that of FIG. 12A may be provided. In this case, the camera 70 may take a picture including the second area 52 and the third area 62. Otherwise, even if the camera 70 takes a picture including all of the first area 42, the second area 52, and the third area 62, the user can receive an image corresponding to two areas. It is possible to provide only.

Another type of screen may be provided as shown in FIG. 12C. In this case, the camera 70 may take a picture including the first area 42 and the second area 52. Otherwise, even if the camera 70 takes a picture including all of the first area 42, the second area 52, and the third area 62, the user can receive an image corresponding to two areas. It is possible to provide only.

That is, according to an embodiment of the present invention, it is possible to provide a related image to a user after all three areas are photographed. However, as shown in FIGS. 12B and 12C, it is possible to provide a user with at least an image corresponding to the drawer's internal storage area, that is, a second area and an area other than the second area.

13 is a diagram illustrating a method of adjusting a picture taken by a camera. Hereinafter, it demonstrates with reference to FIG.

13 may be an example of a screen displayed on the display 14. Of course, if the refrigerator is not equipped with a display, FIG. 13 may refer to a screen displayed on a separate display device of a worker when manufacturing the refrigerator. It may also be a screen displayed on the user's mobile phone.

In a state in which the camera 70 is installed, due to an assembly tolerance, the camera 70 may not be accurately installed at a desired position.

Accordingly, in the embodiment of the present invention, it is possible to change the adjustment line 15 that divides each area through a photo or an image captured by the operator or the user.

The adjustment line 15 may include a vertical adjustment line and a horizontal adjustment line. Vertical adjustment lines can be placed in the center or at both ends of the image. The user may select a portion in which the image provided on the display 14 is cut by moving the vertical adjustment line up and down and the horizontal adjustment line left and right through the buttons shown on the screen.

Meanwhile, a limit line 16 defining a movable range of the adjustment line 15 may also be shown. At this time, the limit line 16 may be arranged to have a predetermined distance left and right around the vertical adjustment line. In addition, the limit line 16 may be arranged so as to have a predetermined interval vertically around the horizontal adjustment line.

Since the limit line 16 is provided, if a desired image is not obtained despite the movement of the adjustment line 15, it is determined as a defect and the installation position of the camera 70 can be adjusted.

As shown in FIG. 13, in a form in which two drawers are provided in the storage compartment, a vertical adjustment line may be disposed between the two drawers.

And the horizontal adjustment line may be disposed at the front end of the shelf.

When the horizontal adjustment line and the vertical adjustment line are completely moved, the lower left of the intersection of the horizontal adjustment line and the vertical adjustment line may form an image of the second region or the left portion of the third region. In this case, the image may range from the horizontal adjustment line to a specific number of pixels in the horizontal direction, and may range from the vertical adjustment line to a specific number of pixels in the vertical direction.

The lower right of the intersection of the horizontal adjustment line and the vertical adjustment line may constitute an image of the second region or the right portion of the third region. In this case, the image may range from the horizontal adjustment line to a specific number of pixels in the horizontal direction, and may range from the vertical adjustment line to a specific number of pixels in the vertical direction.

An upper side of the horizontal adjustment line may constitute an image of the first region. In this case, the image may be in the vertical direction from the horizontal adjustment line to a specific number of pixels.

Meanwhile, as shown in FIG. 13, it is possible to separately show left and right photos of the third area corresponding to the vegetable compartment, and confirm whether or not a photo of the part corresponding to the drawer is acquired.

When the adjustment by the user or the operator is completed, by clicking the input completion button, it is possible to input that the movement of the adjustment line 15 has been completed. Therefore, when the installation error of the camera is small, it is possible to adjust the position of the vertical line and/or the horizontal line separating each area in the photographed picture. Through this, parts corresponding to a specific area can be clearly displayed. Of course, when there are a plurality of vertically partitioned or left and right regions, a plurality of horizontal adjustment lines or vertical adjustment lines described above may be provided.

14 is a perspective view showing a camera, FIG. 15 is a view showing a main portion of the camera, and FIG. 16 is a cross-sectional view of FIG. 14. Hereinafter, it will be described with reference to FIGS. 14 to 16.

The camera 70 may include a camera module 71 including a camera lens 71a and a body 71b, and cases 73 and 74 in which the camera module 71 is accommodated.

The camera module 71 may include an image sensor that converts image information passing through the camera lens 71a into a digital signal.

The case may include a first case 73 and a second case 74 that is coupled to the first case 73 to form a predetermined space therein. The camera body 71b and the camera lens 71a may be accommodated in the first case 73 and the second case 74. The coupling degree may be maintained between the first case 73 and the second case 74 by a separate sealing member so that moisture or dust does not flow into the interior.

Since the camera body 71b and the camera lens 71a are inserted into the space formed by the first case 73 and the second case 74, the inner case 12 and the outer case 10 ) It will not come into contact with the foaming agent that is filled in between. In addition, the camera module 71 may be protected without direct contact with cold air accommodated in the storage compartment 22.

The first case 73 may be provided with a transparent window 80 installed at a front end of the camera lens 71a and a heater 84 that provides heat to the transparent window 80. It is preferable that the transparent window 80 is made of a transparent material.

It is possible to prevent dew from forming on the transparent window 80 by the heat provided by the heater 84.

If the heater 84 is installed on the camera lens 71a to prevent condensation, the camera lens 71a may be deformed. Since the possibility of occurrence of aberration and permanent damage due to such deformation is increased, in the embodiment of the present invention, the transparent window 80 is not in direct contact with the camera range 71a.

As the material of the transparent window 80, both plastic and glass can be applied from the viewpoint of capturing a camera image, but plastic has poor thermal conductivity. Therefore, when plastic is used as the transparent window 80, power consumption may be increased compared to using glass. Therefore, the transparent window 80 is made of a glass material, and it is preferable to reduce power consumption.

When the thermal conductivity increases, heat supplied by the heater 84 can be easily conducted from the transparent window 80 to a portion that does not directly contact the heater 84. For example, when the thermal conductivity is lowered, the temperature of the portion adjacent to the heater 84 is high, while the temperature of the portion away from the heater 84 decreases, so that it is difficult to prevent condensation. In addition, since more heat energy must be supplied by the heater 84, energy efficiency decreases.

Meanwhile, since the heater 84 makes direct surface contact with the transparent window 80, heat transfer efficiency may be improved.

The heater 84 is disposed on the opposite side of the surface where the transparent window 80 is exposed to the storage compartment 22, so that the heat generated from the heater 84 does not pass through the transparent window 80, the storage compartment It can be prevented from being supplied to (22).

In particular, the heater 84 may include a heating coil that generates heat, and the heating coil may make surface contact with the transparent window 80.

In addition, it is preferable that the transparent window 80 and the camera lens 71a are provided to have a predetermined distance g. This is to consider deformation or assembly tolerance that may be generated by supplying heat to the transparent window 80.

In addition, the camera 70 may include a cable 79 that receives electricity supplied from the outside and transmits a signal related to the acquired photograph. The cable 79 may electrically connect the camera module 71 to an external component.

Meanwhile, in the case, a first seating piece 75 capable of fixing the camera 70 to another component is provided. At this time, the first seating piece 75 is provided in either the first case 73 or the second case 74 and thus is irrelevant.

The first seating piece 75 may include a surface having a predetermined area, and may be disposed to have the camera 70 at a desired installation position to have a desired angle of inclination.

The first seating pieces 75 are provided on both sides of the camera 70 so that the left and right sides of the camera 70 may be fixed by external components.

A second seating piece 76 formed to have a predetermined area and stepped may be formed in the second case 74. The second seating piece 76 is provided in a different part from the first seating piece 75, so that the camera 70 is positioned at a specific position on other components like the first seating piece 75. Make sure it can be combined stably. Accordingly, the camera 70 may be fixed to have three or more support points with respect to other components at a predetermined angle through the first seating piece 75 and the second seating piece 76. Details of the supporting structure of the camera 70 will be described later.

For the camera 70, a VGA Resolution class may be applied, and a VGA (640x480) or 3 megapixel resolution may be used in consideration of an image data transmission rate and processing time.

The interface of the camera 70 is a USB type and can be driven using a 5V, 500mA USB power supply.

In addition, the current value supplied to the camera 70 may be 87mA in the standby mode and 187mA in the operation mode (67mA for the heater). Therefore, it is possible to consume power consumption proportional to the supplied current value. At this time, the operation of the camera 70 is divided into a standby mode and an operation mode, and power may be always applied to the heater 84 to prevent condensation on the surface of the transparent window 80.

Meanwhile, when the rated voltage is applied, the camera 70 operates in a standby mode, and when the door switch 110 detects that the door 20 is open, the camera 70 is switched to an operation mode to take a picture. That is, when power is applied to the refrigerator, the camera operates in a standby mode, and when the door switch 110 detects that the door 20 is open, it may be switched to an operation mode for photographing.

17 is a view for explaining the arrangement of heaters. Hereinafter, it demonstrates with reference to FIG.

The camera 70 may have a horizontal angle of view of 115°, a vertical angle of view of 95°, and a diagonal angle of view of 145°. That is, the camera 70 may have a horizontal angle of view larger than a vertical angle of view. Through this arrangement, food information on the storage compartment 22 when the width direction length (horizontal direction in FIG. 17) of the storage compartment 22 is longer than the front and rear length (vertical direction in FIG. 17) of the storage compartment 22 You can obtain a picture that can provide.

Meanwhile, the heater 84 may be formed along the outline of the transparent window 80. In this case, the heater 84 may be disposed to form a square so as not to obscure the angle of view of the camera 70.

In addition, when the heater 84 has a rectangular shape, since heat can be supplied to the transparent window 80 in a relatively large area, energy efficiency for the heater 84 can be improved.

In particular, when a gap g is provided between the camera lens 71a and the transparent window 80 as shown in FIG. 16, the angle of view for the transparent window 80 is formed in a rectangular shape as shown in FIG. 17. If there is no gap g and the transparent window 80 and the camera lens 71a are in contact with each other, the heater 84 contacts one surface of the transparent window 80, and thus the camera lens 71a ) May be provided in a square shape having a side having a length equal to or greater than the diameter.

However, as described above, since the gap g exists and the vertical angle of view and the horizontal angle of view are different from each other, the heater 84 preferably has a rectangular shape in which one side is longer than the length of the other side.

FIG. 18 is a diagram illustrating a picture taken when the camera is mounted in a vertical direction (front and rear direction) in a refrigerator, and FIG. 19 is a diagram illustrating a picture taken in a state where the camera is rotated in a horizontal direction (left and right direction) in a refrigerator. It is a drawing. Hereinafter, a description will be given with reference to FIGS. 18 and 19.

FIG. 18A shows the camera turned upward by 5 degrees, FIG. 18B is installed at a desired position, and FIG. 18C shows a state that has turned downward by 5 degrees. The picture of the degree shown in FIG. 18 may provide information on food stored in the storage room to the user, but if it is twisted at a larger angle, desired information cannot be provided to the user.

FIG. 19A is a state in which the camera is turned to the left by 4 degrees, FIG. 19B is a state in which the camera is turned to the left by 2 degrees, and FIG. 19C is a state in which the camera is installed at a desired position.

When compared to a picture installed so that the camera rotates in the vertical direction as shown in FIG. 18, a picture in which the camera is not installed at a desired angle in the horizontal direction as shown in FIG. 19 provides a relatively largely distorted screen.

As can be seen in FIGS. 18 and 19, even though the camera's installation angle is a small difference, since it can cause a large distortion of a photographed picture, it is necessary to manage the installation angle of the camera so that the assembly tolerance is as small as possible. Of course, a minute error can be adjusted through the adjustment line described with reference to FIG. 13, so that an optimal picture can be provided.

FIG. 20 is a view showing an assembled state of the camera housing, FIG. 21 is a front view of the first housing, FIG. 22 is a front view of the second housing, and FIG. 23 is a front view showing a state in which the camera is installed in the first housing 24 is a side cross-sectional view of FIG. 23, and FIG. 25 is a cross-sectional view illustrating a state in which the camera housing is installed in the inner case. Hereinafter, a description will be given with reference to FIGS. 20 to 25.

In order to install the camera 70 in the inner case 12, a camera housing may be provided. That is, a camera housing for accommodating the camera shown in FIG. 14 and fixing the camera to the ceiling of the storage room may be provided.

The camera housing includes a first housing 400 coupled to a lower surface of the ceiling of the inner case 12, a second housing 410 coupled to the upper surface of the ceiling of the inner case 12, and the first housing 400. ) May include a third housing 420 and a fourth housing 430 coupled to. Here, the first housing 400 may be said to be coupled to the upper surface of the ceiling of the storage room.

The third housing 420 may close the front of the camera 70 so that a portion other than the transparent window 80 of the camera 70 is not directly exposed to the storage compartment 22. That is, the third housing 420 may function as a cover covering the front surface of the camera 70. At this time, the third housing 420 and the fourth housing 430 may be formed of one component. Of course, the fourth housing 430 may be a decor housing interposed between the third housing 320 and the first housing 400.

The camera 70 is assembled in the first housing 400 to manage the left and right inclination (horizontal inclination) and the front and rear inclination (vertical inclination) of the camera 70. I can.

As shown in FIG. 21, the first housing 400 may include a first seating portion 402 on which the first seating piece 75 of the camera 70 is seated. At this time, two first seating portions 402 may be provided so that the left and right first seating pieces 75 may be respectively fastened.

The first seating piece 75 may be provided to be inclined at a predetermined angle to the left and right sides of the camera 70, so that the camera 70 may be disposed to be inclined at a desired angle while the camera 70 is installed in the inner case 12. .

The first seating portion 402 is provided with a fastening hole 404, so that the first seating piece 75 and the first seating portion 402 may be fixed by a screw. The first seating portion 402 is provided to be disposed at different heights in the horizontal direction, so that the camera 70 can be stably fixed.

The first housing 400 may include a second seating portion 408 on which the second seating piece 76 of the camera 70 can be seated. At this time, the second seating portion 408 is also formed to have a predetermined area, so that the degree of inclination of the camera 70 can be fixed while making surface contact with the second seating piece 76.

That is, the camera 70 is coupled while being in contact with the first housing 400 by a plurality of surfaces, so that the camera 70 may perform a function of fixing so that the camera 70 is inclined at a desired angle. In other words, while being fixed to have three or more support points, it may be supported to have a predetermined angle with respect to the first housing 400. Of course, the horizontal angle may be fixed to be parallel to the first housing 400, and the vertical angle may be fixed to a predetermined angle.

The first housing 400 may have a fastening hole 406 to be coupled to the second housing 410 by a screw.

As shown in FIG. 22, the second housing 410 has a contact surface 416 capable of making surface contact with the upper surface of the ceiling of the inner case 12. At this time, the contact surface 416 is widely distributed while forming a rectangular shape as a whole, so that the second housing 410 can be stably seated on the upper surface of the ceiling of the inner case 12. Accordingly, the second housing ( The contact surface 416 of 410 has a surface parallel to the ceiling of the inner case 12. Through this, the reference line or the reference plane for the installation angle of the camera may be the same as the ceiling surface of the inner case 12.

The second housing 410 may include an accommodation space 412 in which the cable 79 of the camera 70 can be accommodated. A through hole 418 is formed at one side of the accommodation space 412, so that electric wires connected from other components of the refrigerator may be exposed to the accommodation space 412. In this case, the receiving space 412 may be connected to an electric wire connected from other components of the refrigerator and the cable 79. Accordingly, electricity may be supplied to the camera 70 through the cable 79, and a signal for a picture taken by the camera 70 may be transmitted to other components.

The second housing 410 may be provided with a fastening portion 414 coupled to the first housing 400. The fastening part 414 may be coupled by the fastening hole 406 and a screw. When the fastening part 414 is combined with the fastening hole 406, it is preferable that an error due to assembly does not occur. Therefore, it is also possible to be formed to protrude by a predetermined height so as to be in surface contact with the fastening hole 406.

As shown in FIGS. 23 and 24, when the camera 70 is assembled to the first housing 400, the camera 70 may be disposed in an inclined form by a set angle. As shown in FIG. 24, one side surface 400a of the first housing 400 has a shape forming a plane so as to be in contact with the inner case 12. The camera 70 is inclined at a predetermined angle when compared to the one side surface 400a of the first housing 400. In addition, the one side surface 400a is coupled to the lower surface of the ceiling of the inter case 12 so as to be parallel with the contact surface 416 of the second housing 410. Accordingly, as the camera 70 is coupled to the first housing 400 and the first housing 400 is coupled to the second housing 420, the installation angle of the camera 70 with respect to the ceiling surface of the storage room The error of can be significantly reduced.

Referring to FIG. 25, the second housing 410 is disposed to be exposed to a space filled with a foaming liquid in the upper portion of the inner case 12, that is, a space between the inner case 12 and the outer case 10. do. The second housing 410 is disposed outside the storage compartment 22, but is disposed in a space partitioned by the outer case 10. After the second housing 410 is adhered to the inner case 12 and temporarily fixed, the second housing 410 will be finally fixed by a foaming solution filled between the inner case 12 and the outer case 10. I can.

With the inner case 12 interposed therebetween, the first housing 400 may be coupled to the second housing 410. When the second housing 410 is fixed to a specific position of the inner case 12, the first housing 400 is coupled to the second housing 410, so that the position of the camera 70 is Can be fixed.

When the camera 70 is inclined to the first housing 400 at a specific angle, the third housing 420 and the fourth housing 430 may be coupled to the second housing 410. . As described above, the third housing 420 and the fourth housing 430 are composed of one component, so that the first housing 400 performs the function of the third housing 420 It is also possible to combine only.

The third housing 420 may allow only the transparent window 80 of the camera 70 to be exposed to the storage compartment 22 and may not expose other portions. Accordingly, it is possible to prevent the camera 70 from being affected by moisture, dust, or cold air existing in the storage compartment 22.

The structure of the camera housing and the installation direction and installation shape of the camera housing with respect to the inner case also have a technical meaning for reducing condensation occurring in the transparent window 80. A detailed description will be provided later.

26 is a table comparing power consumption of cameras in the present invention, and FIG. 27 is a diagram comparing supplied standby current and driving current. 26A illustrates a state in which current is supplied to the camera in the comparative example, and FIG. 27B illustrates a state in which current is supplied to the camera in the present invention. Hereinafter, a description will be given with reference to FIGS. 26 and 27.

In the comparative example, a standby current of 70mA is supplied to the camera 70, and then 50mA is additionally supplied at the time when the door is opened, and a driving current of 120mA is supplied to the camera 70. In the comparative example, when the camera 70 is opened 25 times a day, 253 Wh is consumed per month, and when opened 50 times a day, 254 Wh is consumed per month.

On the other hand, in the embodiment of the present invention, a standby current of 20mA is supplied to the camera 70, and then 100mA is additionally supplied at a time when the door is opened, so that a driving current of 120mA may be supplied to the camera 70. When the camera 70 is opened 25 times a day, 74Wh is consumed per month, and when opened 50 times a day, 76Wh is consumed per month.

Comparing the two experimental results, it can be seen that the power consumed by the camera 70 can be reduced by approximately 70% by lowering the standby current. Therefore, it was confirmed that the power consumed in the camera 70 can be greatly reduced by lowering the standby current under the premise that the driving current is the same. This is because, when the refrigerator is operated continuously for 24 hours, the time the door is opened is relatively short. Accordingly, when the current value required for photographing in the operation mode is constant, the current consumption can be reduced as the current value applied to the camera in the standby mode is lowered.

When the standby current is lowered, as shown in FIG. 27, the timing at which the driving current is supplied to the camera 70 is delayed. This is because the delay time inevitably increases due to the difference between the standby current value and the driving current value. In other words, when switching from the standby mode to the driving mode, photography is not immediately performed, and a predetermined delay time elapses, so that the photo is taken.

Therefore, the point in time at which the camera 70 actually takes a picture is delayed. If the difference between the time when the command to take a picture is transmitted to the camera 70 and the time when the camera 70 actually takes a picture is large, the picture actually taken by the camera 70 preferably provides food information to the user. May not be available.

In other words, the lower the standby current is, the lower the power consumed by the camera 70 is. However, the delay time may increase in increasing the driving current for driving the camera 70. Therefore, it is preferable to calculate a normal time elapsed before the user can take a picture at a desired time point, and select a standby current value so that the delay time is reduced rather than the corresponding time. For this reason, according to the embodiment of the present invention, it is desirable to increase the current applied step by step from the standby current value to the drive current value. That is, a current value applied from a relatively low standby current value to a relatively high driving current value can be increased step by step with a sufficient delay time. That is, it is possible to prevent a rapid increase in the current value. Therefore, it is possible to reduce the power consumption and at the same time achieve stability of the camera.

Accordingly, in an embodiment of the present invention, as shown in FIGS. 28 and 29, the camera 70 is controlled to start driving when the door 20 is opened. That is, when it is determined that the door is opened through the door switch, the standby mode is switched to the driving mode. However, among the pictures taken by the camera, valid pictures are after at least a predetermined time has elapsed from the time the door is opened. This is because the door has to be opened and the user physically needs to update the food in a matter of seconds or more, i.e., even if the delay time is long, it is possible to acquire a picture at a desired time. Accordingly, in this embodiment, it can be said that the timing of switching from the standby mode to the driving mode of the camera is performed by the detection result of the door switch. In addition, it can be said that the reference time point for selecting a valid photo is performed by the detection result of the door sensor.

28 is a diagram illustrating a starting point of photographing by the camera and continuous photographing by the camera. Hereinafter, it demonstrates with reference to FIG.

Meanwhile, in the present invention, as shown in FIG. 4, the door switch may be included. When the door switch 110 detects the opening of the storage compartment 22, the controller 100 may command the camera 70 to start taking a picture. That is, when the door 20 is moved or rotated to open the storage compartment 22, the camera 70 may start taking pictures.

On the other hand, when the door switch 110 detects that the storage compartment 22 is closed, the controller 100 may command the camera 70 to stop taking pictures. That is, when the door 20 is moved or rotated and the storage compartment 22 is closed, the camera 70 may end photographing.

As shown in FIG. 28, when electricity by external power is supplied to the refrigerator, standby current is supplied from the camera 70 (S1). In this case, the camera 70 may be in a state of being driven in a standby mode. In the standby mode, the camera 70 does not take a picture.

Opening of the door 20 may be detected by the door switch 110 (S10).

When it is determined that the door 20 opens the storage compartment 22, the camera 70 may start taking a picture (S14). In particular, the camera 70 may continuously take pictures at predetermined time intervals.

At this time, since a current larger than the current supplied in the standby mode is supplied to the camera 70, the camera 70 may be performed in a driving mode. In order to drive the camera 70, a predetermined time, that is, a delay time, inevitably occurs in order to supply a current larger than the standby current and the driving current. That is, it is because a certain amount of time is actually required to increase the magnitude of the current supplied to the camera 70. After a delay time of a predetermined time has to elapse after a current for performing a driving mode is supplied to the camera 70, a driving current is supplied to the camera 70 so that the camera 70 can only take a picture. .

Since the camera 70 is switched to the driving mode for photographing from the point when the door 20 is opened, the start point at which the picture is taken is early, so that the risk of not being able to take a picture at a specific point in time is reduced. This is because if the standby current is supplied before the desired time point, and the current value is increased at the desired time point, the time point of photographing is delayed due to the delay time.

In addition, the number of photos taken in one second by the camera 70 can be reduced. Assuming that the camera 70 takes 30 pictures per second, the possibility that the inside of the storage compartment 22 changes during the time interval at which the pictures are taken, that is, 1/30 seconds, is extremely small. The interior of the storage compartment 22 may be changed by a user's action, because a general person is less likely to end the action for 1/30 second with respect to the storage compartment 22. That is, the time interval taken by the camera 70 may be relatively increased. Accordingly, since the camera 70, which is relatively inexpensive, can be used, the cost of the refrigerator can be reduced.

The picture taken by the camera 70 may be stored in the storage unit 18 (S16). In addition, when the number of pictures taken in one second by the camera 70 decreases, the number of pictures stored in the storage unit 18 decreases, so that the storage unit 18 can be operated more efficiently.

Meanwhile, the door sensor 120 may detect whether the door 20 is rotated at a specific angle (S18).

If the door sensor 120 does not detect whether the door 20 is rotated, a picture before a predetermined time may be deleted from the pictures stored in the storage unit 18 (S90). In other words, the unprocessed photo may be deleted to provide it to the user, thereby preventing an increase in the capacity of the storage unit 18.

On the other hand, when the door sensor 120 determines that the door 20 has been rotated at a specific angle, a picture taken at the time is selected (S92). At this time, the selected photo may be processed by the control unit 100 as a valid photo and provided to the user. For example, a photo processed by the control unit 100 may be transmitted to the display 14.

In order for the user to access the storage room 22, a certain time is required to perform the operation of rotating the door 20. Since the delay time of the camera 70 elapses during that time, a picture may be taken when the door 20 is rotated at a specific angle. Accordingly, the controller 100 may select a picture at a desired point in time.

If the door sensor 120 switches from the standby mode to the driving mode so that the camera 70 takes a picture at the time it is sensed, there is a problem that the storage room state after the delay time has elapsed from the time point has no choice but to be photographed. I can.

Meanwhile, since the camera 70 starts photographing when the door 20 is opened, it is possible to supply a relatively small amount of standby current to the camera 70. If the standby current is supplied less, the delay time until the driving current is supplied increases, but in the above-described control flow, the camera 70 starts photographing early, so that the problem of not taking a picture at a desired time does not occur. .

29 is a diagram illustrating a start point of photographing by the camera and continuous shooting of the camera. Hereinafter, it demonstrates with reference to FIG. 29.

In FIG. 29, unlike FIG. 28, a photo selection time point is not determined according to a rotation state of the door 20, but a photo selection time point is determined according to a state of the drawer 50. Other parts are the same, and the same parts are omitted for convenience of description. However, it can be said that the operation of the drawer 50 is based on the opening of the door 20. The reference point of time for selecting a specific picture may be determined by the judgment of the door sensor, and the reference point of time for selecting a picture for a specific area within the specific picture is determined by the judgment of the door sensor and the judgment of the drawer detection unit 130. I will be able to. Therefore, in any case, the determination result of the door sensor is required in order to determine the reference point of time of the effective photo.

In order to store food in the drawer 50 or take out food stored in the drawer 50 to the outside, the user may withdraw the drawer 50. In this case, the user may withdraw the drawer 50 by a sufficient length to use the drawer 50.

After the user has finished using the drawer 50, the drawer 50 may be pushed in. The time point at which this state occurs may mean a specific state described in S19.

Since the camera 70 starts photographing from the point when the door 20 is opened by the door switch 110, if the drawer 50 is placed in a specific state, the picture taken at that point is taken. It may be selected and provided to the user (S92).

When the door 20 is opened, a picture is taken by the camera 70, and movement of the drawer 50 may be monitored from a point in time when the picture is taken, that is, a point in time when the door 20 is opened.

FIG. 30 is a diagram illustrating an embodiment of a drawer sensing unit, and FIG. 31 is a diagram illustrating a method of detecting a movement of a drawer in the drawer sensing unit of FIG. 30. It will be described below with reference to FIGS. 30 and 31.

Referring to FIG. 30, the drawer detection unit 130 may not be provided in the control unit 100, but may be implemented as a separate component.

The drawer sensing unit 130 may include one light-emitting unit and two light-receiving units. That is, by analyzing the point in time when the light irradiated from one light-emitting unit is incident on the two light-receiving units, the moving direction of the drawer 50 may be detected.

A plurality of slits 134 are continuously arranged in the drawer 50 so that light irradiated from the light emitting portion of the drawer sensing unit 130 is not reflected when incident on the slit 134. Accordingly, light can be sequentially incident on the two light-receiving units.

In FIG. 31, the x-axis represents time, and the y-axis represents a signal when light is received from the two light receiving units (Photo TR1 and Photo TR2). That is, in FIG. 31A, light is received from Photo TR1 before Photo TR2, and in FIG. 31B, light is received from Photo TR2 before Photo TR2.

In Fig. 31b, it means a moving state in which the user withdraws to use the drawer 50, and in Fig. 31a, the user finishes using the drawer 50 and pulls the drawer 50 into its original position. Can mean state.

Therefore, when the drawer detection unit 130 detects a signal as shown in FIG. 31A, the control unit 100 determines that it is a time point to acquire a picture taken by the camera 70, and at that time the drawer 50 It can be selected as a picture corresponding to the inside of, that is, the third area. That is, the acquisition time point of the photo corresponding to the third area may be determined according to a result of detection or determination by the drawer detection unit 130. Of course, the portion corresponding to the third area in the obtained photo may be independently and individually selected, separated, and updated from the portion corresponding to the other area.

32 is a diagram illustrating a state in which a marker is displayed on a drawer. Hereinafter, it demonstrates with reference to FIG.

Unlike FIGS. 30 and 31, even if a separate mechanical or electronic device is not used, the movement of the drawer 50 may be detected using a marker that is a specific mark on the drawer 50. .

Various types of markers (patterns having black and white colors) may be provided on the drawer 50. The camera 70 may analyze the movement of the marker through the photographed picture, and read whether the drawer 50 moves and the movement direction. In this case, the analysis of the marker may be performed by the drawer detection unit 130 of the control unit 100. That is, the drawer detection unit 130 may determine the state of the drawer 50 by analyzing the movement of the marker in the pictures that are successively taken. Accordingly, the timing of acquiring a photo corresponding to the third area may be determined according to a result of detection or determination by the detection unit 130 as it is dark. Of course, the portion corresponding to the third area in the obtained photo may be independently and individually selected, separated, and updated from the portion corresponding to the other area.

In FIG. 32, the marker includes a simplified image of the shape of the camera and the word'smart sensor'. Meanwhile, the marker may include various types of patterns, and may be provided in a form in which black and white cross each other, so that the camera 70 can recognize it.

In this case, the marker may be displayed on the front upper side of the drawer 50 so that the camera 70 can recognize it.

FIG. 33(a) is a view showing a picture taken in a state in which the left and right drawers are withdrawn, and FIG. 33(b) is a view showing a state in which the left drawer is pulled in and the right drawer is pulled out. Hereinafter, description will be made with reference to Figs. 33(a) and 33(b).

In FIG. 33, the marker is provided in the center of the upper side of the drawer 50, and an example in which a wave-shaped pattern is formed is shown.

The first area 42 may be expressed in a single picture taken by the camera 70. In addition, one of the second region 52 or the third region 62 may be shown in one picture.

That is, as shown in FIG. 33(a), the first area 42 disposed on the left and right and the second area 52 disposed on the left and right are shown in the picture taken with the left and right drawers withdrawn.

On the other hand, as shown in FIG. 33(b), in the picture taken with only the right drawer drawn out, the first area 42 disposed on the left and right, the third area 62 on the left, and the second area 52 on the right side. ) Can be shown.

That is, when the drawer 50 is withdrawn, the third area 62 is not shown in the picture, but the second area is shown. On the other hand, when the drawer 50 is inserted, the second area 52 is not shown in the picture and the third area 62 is shown.

The camera 70 takes a picture at the same location. The drawer 50 can be moved, and thus, the picture acquired by the camera 70 according to the position of the drawer 50 can take pictures of various areas.

On the other hand, as shown in FIG. 33(b), when the drawer 50 provided on the left is completely inserted into the lower space of the shelf 40, the marker is not recognized by the camera 70. As shown in FIG. That is, a marker is not shown in the picture taken by the camera 70.

A method of detecting the movement of a drawer using a marker will be described with reference to FIGS. 34 to 40.

34 is a flowchart illustrating a method of operating a refrigerator according to an embodiment of the present invention.

Referring to FIG. 34, when the door switch 110 detects the opening of the door 20, the camera 70 starts taking a picture (S2101), and the controller 100 performs a marker recognition operation and a marker tracking. The operation is started (S2103).

Hereinafter, steps S2101 and S2103 will be described with reference to FIG. 35 in order to describe steps S2101 and S2103 in more detail.

35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.

Referring to FIG. 35, when the door 20 is opened, the camera 70 starts taking a picture (S2301).

The controller 100 controls the camera 70 to obtain an internal picture of the drawer 50 (S2303).

Subsequently, the controller 100 processes the acquired photo (S2305), and determines whether a predetermined marker is recognized in the acquired photo (S2307).

In order to determine whether a predetermined marker is recognized, the controller 100 may compare whether a marker recognized from an acquired photo is the same as a previously stored marker.

The predetermined marker may be determined by a business operator or a user, and may be stored in the storage unit 18 or the like in advance.

The marker may be formed in a searchable area in a picture taken from a camera. Hereinafter, markers that can be recognized and tracked by the controller 100 will be described in detail with reference to FIGS. 36 and 37.

36 is a diagram for describing a marker for detecting a drawer withdrawal point according to an embodiment of the present invention.

Referring to FIG. 36A, a marker may be formed in a searchable area by the camera 70 from a point in time when the drawer 50 starts to open. For example, when the camera 70 is located at the top of the refrigerator and can photograph the plane of the drawer 50, the marker may be located at the front end of the top of the drawer 50. In addition, the field of view of the camera 70 may be installed such that the drawer 50 can be taken from a completely closed state to a fully opened state.

When the marker is located at the top of the drawer 50, it is assumed that there is no obstacle at the top of the drawer 50 at the time when the drawer 50 is closed after the storage of the drawer 50 is organized. It is planned, but is not limited thereto.

Referring to (b) of FIG. 36, when looking at the drawer 54 and the drawer 50 in which the drawer 50 is accommodated from the viewpoint of the camera 70, the camera 70 is a drawer ( The inside of the drawer 50 including the marker can be photographed in all ranges from the state where 50) starts to open to the state where the drawer 50 is fully open.

37 is a diagram for describing a shape of a marker according to an embodiment of the present invention.

Referring to FIG. 37, the marker may be formed in a form in which the same shape of two colors crosses and repeats.

Referring to FIG. 37A, the marker may be formed in a form in which a plurality of bands in which a white square and a black square are alternately repeated are intersected and arranged.

Referring to FIG. 37B, the marker may be formed in a form in which a plurality of bands in which a white inclined rectangle and a black inclined rectangle are alternately repeated are intersected and arranged.

As described above, the repetitive marker has an advantage that the control unit 100 can recognize and track the marker even when a part of the marker is hidden from the camera shooting range. However, the marker is not limited to a repetitive shape, nor is it limited to a plurality of colors. As described above, the marker is sufficient in a form capable of recognizing whether or not an internal component of the refrigerator having a mobility such as the drawer 50 is moved.

Again, referring to FIG. 35, when a predetermined marker is recognized, the controller 100 detects movement of the recognized marker (S2311). The controller 100 may analyze and process the continuously or periodically acquired photo to determine whether the recognized marker moves or stops. For example, when the opening operation of the drawer 50 is continued or completed, or the closing operation of the drawer 50 is started, continued or completed, the control unit 100 may determine whether the marker is moved or stopped. .

Alternatively, if the predetermined marker is not recognized, the control unit 100 notifies the user that the marker recognition has failed (S2309).

For example, when an obstacle is located on the marker, the camera may not be able to acquire a picture including the marker, or the controller 100 may not be able to recognize the marker from the acquired picture. In this case, the controller 100 may notify the user of information indicating that the marker recognition has failed, information notifying a method for successfully recognizing the marker, such as removing obstructions, to the user through the display 14 or the like.

Subsequently, when the marker recognition fails, the controller 100 may repeatedly perform the step of recognizing a predetermined marker by processing a newly acquired photo through the camera.

Referring back to FIG. 34, after recognizing the marker, when a drawer closing operation is detected (S2105), the controller 100 includes the inside of the drawer (the third area 62) through a picture taken at the time point. A final picture is obtained (S2107). At this time, in the final picture of the inside of the drawer acquired by the control unit 100, the inside of the drawer just before the drawer closing operation is detected or the inside of the drawer immediately after the drawer closing operation is detected, It is not limited thereto.

Whether the drawer closing operation is detected will be described with reference to FIG. 38.

38 is a flowchart illustrating a method of detecting a drawer closing operation according to an embodiment of the present invention.

Referring to FIG. 38, the controller 100 determines whether the end of the opening operation of the drawer 50 is detected (S2501). For example, when the controller 100 recognizes a marker from the previously acquired picture of the drawer 50, tracks the movement of the marker, and detects that the movement of the marker has stopped, the opening operation of the drawer 50 is performed. It can be detected that it has ended.

Subsequently, the control unit 100 determines whether the start of the closing operation of the drawer 50 is detected (S2503). For example, when the marker stops moving and then starts moving again in the opposite direction, the controller 100 may detect that the closing operation of the drawer 50 has started.

Meanwhile, an operation of the control unit 100 detecting a drawer closing operation according to another embodiment of the present invention will be described with reference to FIG. 39.

39 is a flowchart illustrating an operation of a refrigerator performed upon detection of completion of a drawer closing operation according to an embodiment of the present invention.

When the end of the drawer closing operation is detected (S2701), the control unit 100 determines whether or not the final photo acquisition through marker recognition has succeeded (S2703).

The control unit 100 may detect that the closing operation of the drawer 50 is terminated through image processing by the drawer detection unit 130.

As a result of the marker not being normally recognized due to an obstacle or the like, the controller 100 notifies the user that the marker recognition has failed when the final photo acquisition has failed (S2705).

The controller 100 may inform the user of information that the marker recognition has failed through the display 14 or the like. This guides the user to remove the obstruction, and may induce the drawer to be pulled in and out for normal marker recognition and final image acquisition.

Referring back to FIG. 34, the controller 100 displays the final image using the display 14 or the like (S2109). In this case, the form provided to the user may be a portion obtained by extracting only a portion corresponding to the third area 62 from the obtained final picture. Next, a final image control according to another embodiment of the present invention will be described with reference to FIG. 40.

40 is a flowchart illustrating a method of controlling an image of a specific area inside a refrigerator photographed at a specific point in time according to an embodiment of the present invention.

Referring to FIG. 40, the controller 100 may obtain a final photo (S2901) and crop an image corresponding to a desired area among the final photo. Then, the obtained final image is stored in the storage unit 18 (S2903).

Subsequently, the control unit 100 determines whether or not a user input for displaying a final image is received through a user input or the like (S2905).

For example, when obtaining a final picture, the controller 100 may form a user input unit in the form of an icon together with information notifying the success of obtaining the final picture on the display 14 composed of a touch screen. It is possible to determine whether an input is received. The user input unit may include a physical key or the like as described above in addition to the icon formed on the touch screen, but is not limited thereto.

Subsequently, when receiving a user input for displaying the final image processed from the final photo, the controller 100 displays the final image (S2907). The controller 100 may display the final image using the display 14. The final image may be displayed by replacing the existing image. Therefore, it can be said to be an update of the final image.

Of course, the update of the final image may be performed automatically regardless of the user's input.

According to such an embodiment of the present invention, the refrigerator may detect whether the drawer or the like is moved by recognizing a marker previously formed on the drawer or the like. In addition, the refrigerator may acquire a final state of a storage object stored in a drawer or the like in the form of an image using a camera installed therein. In this case, the refrigerator may determine an optimal time point at which a final storage state such as a drawer can be obtained by using a marker formed on a drawer or the like.

Hereinafter, an installation location of the camera and a photograph obtainable therefrom will be described.

In an embodiment of the present invention, a drawer body (ie, a drawer 50) forming a drawer 50 with the drawer 50 is installed outside the drawer 50, that is, in the storage compartment 22. It is possible to simultaneously recognize the drawer's movement status as well as the drawer's internal storage status by continuously taking pictures including the movement path of the front and rear side walls, the upper part of both side walls, etc.). Of course, the picture taken by the camera 70 is taken together with the upper space of the drawer 50, that is, a portion corresponding to the first area 42.

Basically, the picture taken by the camera should be able to include both the opening and closing motion of the drawer and the path the drawer moves. Therefore, the camera should be installed in a position that can take a picture of the drawer's opening and closing and moving paths from the outside of the drawer. That is, it is preferable that the camera is installed at least in a position in which the drawer is closed and completely opened as a single picture.

Furthermore, the camera 70 includes not only the drawer area but also another storage area of the storage room, for example, a shelf area that is divided by a shelf and stores food or items in one photo by taking a single picture. Thus, a single camera can be configured to cover another storage area including a drawer area, that is, a plurality of storage areas.

For this purpose, the drawer 50 is preferably formed on the bottom of the refrigerator storage compartment. The drawer 50 may include a front wall, two side walls, a rear wall, a bottom wall, and an upper opening. The camera 70 should be provided on the drawer so that at least a part of the drawer 50 and the upper front wall of the drawer in a state in which the drawer is completely open can be exposed.

In addition, a print part (Marker) made of a specific pattern is formed or attached to a position on the front wall or the upper surface of both side walls of the drawer exposed to the camera.

At this time, since the camera 70 is fixed at a predetermined position, it is necessary to configure the position of the printing unit of the drawer 50 to change relatively. That is, when the camera 70 receives a driving command from the control unit, the camera 70 continuously photographs at a constant speed, and sequentially transmits the captured images in the storage room to the control unit 100. The control unit 100 tracks the location of the printing unit from a picture of the storage of goods in the storage unit transmitted from the camera 70 to determine information on the movement of the drawer as well as the open state of the drawer.

The range in which the camera can shoot may be determined by the angle of view of the camera, which is composed of a vertical angle of view, which means a shooting range in a vertical direction, and a horizontal angle of view, which determines a shooting range in the horizontal direction.

One end of the vertical angle of view of the camera 70 must include at least the drawer front wall in a state where the drawer is completely open, and the other end is a print located on the drawer front wall or both side walls in the drawer closed state. It should be determined to include a portion where it can be confirmed that the part is not exposed, for example, a portion of the front side of the upper side of the drawer covering the upper opening of the drawer.

At this time, it is possible to simultaneously recognize another storage space as well as the drawer's internal area with one camera. Another embodiment of the present invention is a separate storage space on the bottom of the storage compartment (hereinafter, at least partially overlapped with the drawer in the open state) between the drawer front wall in the closed state and the front opening of the storage compartment. Storage compartment floor storage compartment) can be configured. In this case, the vertical angle of view of the camera 70 may be configured to recognize another storage space (floor storage compartment) including the drawer inner area even in the shooting range including the drawer fully open and the drawer fully closed state. May be.

On the other hand, another embodiment of the present invention regarding the vertical angle of view of the camera 70 may simultaneously photograph the drawer inner area and the shelf area. Accordingly, it is possible to determine the range of the other end of the vertical angle of view so as to include up to any one of a plurality of shelves arranged vertically at a predetermined interval. In the camera vertical angle including the shelf area, a preferred example may include up to a part of the topmost shelf among the plurality of shelves, and more preferably, at least of the uppermost shelf disposed at the top of the storage compartment. It is to include the rear end.

According to these embodiments, it is possible to check the storage state of articles in at least two storage spaces including the drawer inner area by one camera with one picture taken with one camera.

To this end, a preferred installation position of the camera is required for the drawer formed at the bottom of the storage chamber, and the inventors of the present invention first reviewed the camera installation position disclosed or known through the data published at the time of filing of the present invention. It was installed on the top of the door outside the storage compartment of the refrigerator to examine whether it was possible to photograph the interior of the storage compartment.

When the camera is installed outside the storage room (door), there is an advantage that a separate condensation prevention structure is unnecessary because condensation is not formed around the camera because the camera is always exposed to external temperature. However, it is difficult to photograph the inside of the storage room with a camera mounted on the door because the operation of storing items inside the drawer and in the storage space in the storage room by the user is always performed with the door open. In particular, an object of the present invention is to obtain not only the movement state of the drawer from the outside of the drawer, but also the state of storage of the article inside the drawer. However, the user's work in the drawer is always in the open state of the door, and the drawer is also in a closed state when the door is closed, and thus does not correspond at all to the gist of the present invention.

Accordingly, in order to photograph at least the inside of the drawer and the movement state of the drawer, which is an object of the present invention, it is preferable to arrange the storage room so that the storage room can be photographed in the main storage room in which the drawer to be photographed exists, not outside the refrigerator. Do.

On the other hand, in accordance with the gist of the present invention, if the camera is installed so that the camera is directed toward the opening side of the main storage compartment from the bottom of one of the shelves located above the drawer, first, it is impossible to photograph the shelf area. It is difficult to install and wire a driving unit capable of driving a camera on a shelf, and a wide-angle camera must be used to capture all the movement paths of the drawer, so there may be a problem of increasing distortion in the captured image.

The inventors of the present invention believe that when a camera is installed on the side wall of the storage compartment within the range where the drawer can be exposed, distortion is caused in the image showing the storage state of the drawer because of the asymmetric image due to the difference in distance from the camera in the left and right directions of the refrigerator storage compartment. I could see that it was severe. Therefore, it can be seen that the satisfaction of these users can be greatly reduced.

Above all, one purpose of the present invention is to track the movement state of the drawer without a separate sensor to detect the open/closed state and the degree of the drawer.In the asymmetric image, printing located on the front wall of the drawer or on the upper surfaces of both side walls It may be photographed in a state in which the marker is relatively enlarged (image distortion). Accordingly, the size (pixel size) occupied by the marker portion with respect to the entire size constituting one image may be increased, so that the position change may be difficult to reveal relatively well. In addition, since the range to be processed to determine whether the marker position is changed may increase, the image processing time may increase, and thus, there may be cases in which consecutively transmitted images are missed or cannot be processed.

Accordingly, it was found that installing a camera on a side wall or corner of a storage room known at the time of filing of the present invention did not meet the spirit of the present invention.

According to an embodiment of the present invention, the camera is installed on the upper wall of the storage compartment corresponding to the upper opening surface of the drawer, and the state in which the drawer is closed and the state in which the drawer is completely open are projected on the upper surface of the storage compartment. It was installed within the range, and the camera was installed at a certain angle so as to face the interior of the storage compartment toward the rear wall side of the storage compartment, and the focus of the camera toward a point on the rear wall of the storage compartment. As a result, it was possible to obtain an image in which distortion is minimized for the above-described embodiments of the vertical angle of the camera of the present invention.

In determining the location of the camera, another embodiment of the present invention is installed on the upper wall of the storage compartment of the refrigerator, and the front end of the uppermost shelf among the shelves arranged up and down in the storage compartment at the front opening of the storage compartment is located on the upper wall of the storage compartment. It is to place it between the projected positions. In addition, the direction of the camera is toward the rear wall side of the storage room, and the camera is installed at a certain angle so as to face the inside of the storage room, and the focus of the camera is toward a point on the rear wall of the storage room.

The camera is preferably located at the center of the upper portion of the storage chamber, and to correspond thereto, a printing unit located on the drawer front wall or both side walls is preferably installed on the upper surface of the front wall of the drawer. More preferably, the vicinity of the upper surface of the drawer front wall corresponding downward to the camera position installed on the upper surface of the storage compartment is proposed.

In a refrigerator that opens and closes the opening of one storage room with one or more doors, that is, the left and right doors, the camera is installed on the upper surface (ceiling surface) of the storage room at the boundary between the left and right doors, and drawers respectively present on the left and right. In this case, the printing unit is located near the left and right borders. That is, it is preferable that the left drawer is located at the top right of the front wall or at the top end of the right side wall, and the right drawer is located at the top left of the front wall or at the top end of the left side wall from the perspective of the inside of the storage room. In other words, it is preferable that the printing portions are formed at a position close to the vertical lower part of the camera. This is because, as the distance from the vertical lower portion to the left and right increases, there is a higher fear of distortion of the printing unit. This example is shown in Fig. 41B.

At this time, when printing units are installed on both the upper surface of the front wall of the left and right drawers or at the upper ends of both sides, the drawers used in the storage compartment opened and closed by the left and right doors can be used in common regardless of the position of the left and right sides. So you can increase the efficiency.

As described above, according to the camera position and the printing unit position according to the present invention, all the movement paths from the closed state to the open state are included within the shooting range of the fixed camera. It is possible to detect not only the opening and closing state of the lower, but also the degree of opening and closing, and the stop or movement of the drawer, and at the same time, the storage state of the items inside the drawer can be detected or confirmed by a single camera.

In addition, since the storage status of the items in the floor storage bin installed on the floor of the storage room other than the drawer inner area can be checked while the drawer is closed, a plurality of storage areas can be recognized with a single camera.

In addition, when the angle of view of the camera extends to the shelf area, three storage areas can be selectively recognized with one camera, not only the drawer inner area, but also the storage compartment at the bottom of the storage room and the front area.

On the other hand, since the camera installed on the upper surface of the storage room and the printing parts formed or attached to the top of the drawer are arranged in a roughly vertical line, the printing part in the entire image of the storage room that was photographed and transmitted by the camera. There is almost no distortion of the printer, and the space occupied by the printing unit can be relatively minimized, so the time to track and determine the location of the printing unit through image processing can be minimized.

Hereinafter, a method of recognizing a marker trace formed on a drawer will be described.

A printing unit (marker or printing paper) that can be recognized by image reading may be installed on a specific part of the drawer body exposed to the camera. By tracking the position of the marker unit, the degree of opening/closing of the drawer, and the opening and closing states of the drawer may be detected. The marker unit is configured to be exposed to the camera from the closed position to the fully open position on the movement path of the drawer, and the open/close state of the drawer may be determined based on the position of the marker unit. That is, it is possible to determine various state information of the drawer through the change in the position of the marker in the pictures to be taken continuously.

As described above, the print unit (Marker) structure may be formed of a specific pattern in order to track and recognize the movement status or opening/closing status of the drawer. In this case, the printing unit may be disposed at various positions on the upper surface of the drawer exposed to the camera.

It is preferable that the printing unit be installed at the top of the front wall of the drawer or at the top of both side walls of the drawer at a portion that can be a boundary between the maximum open portion of the drawer.

The pattern shape of the printing unit may be formed as a pattern in which patterns having a predetermined shape of a predetermined size are repeated in a contrasting color, and the shape may be formed in various shapes such as a square, a triangle, a circle, and a wave shape.

Such a printing unit may have a fear of deteriorating the overall aesthetics of the refrigerator, and thus may have a sense of rejection depending on the user. Therefore, it is necessary to minimize the repetition of the pattern so that there is no objection to the aesthetic sense that the user feels, and furthermore, a printing unit in a form that can appeal the aesthetic sense to the user can be considered.

However, this limitation of the design aspect is not limited to a simple design problem, and there are cases in which the position of the drawer cannot be accurately recognized. That is, it affects the recognition rate of the print unit of the drawer, and ultimately, the position of the drawer cannot be accurately detected.

That is, in consideration of the design factor that should significantly reduce the size of the printing unit, it is possible to select a degree of repeating only 2 to 4 times, not repeating several times with a contrasting color with a pattern of a certain shape. In this case, the control unit processing the image photographed inside the container may not accurately recognize the boundary of the printing unit (marker) in the entire image.

Accordingly, according to a preferred embodiment of the present invention for solving such a problem, since a drawer is usually made of a bright color, transparent or translucent, it is possible to select a color contrasting thereto. Referring to FIG. 41A, the length L1 in the vertical direction with respect to the movement direction (forward and backward) of the drawer may be set to be longer than the length L2 in the drawer movement direction. In addition, a first color part having a predetermined length is formed in a vertical direction, and a color contrasting with the first color part or the same color as the drawer with approximately the same thickness in front or rear of the first color part with respect to the direction in which the drawer proceeds. A second color part made of is formed, and is continuously repeatedly formed with the same color as the first color part with substantially the same thickness and the same length. That is, it is possible to sequentially arrange the first color part and the second color part with each other. In this case, it is more preferable that the length L1 in the vertical direction is larger than the length L2 in the drawer direction in the outer sizes of the markers arranged as the first color part, the second color part, and the first color part.

According to another marker form of the present invention, as shown in FIG. 41B, both ends of the first color part at the front and rear ends are connected to each other to surround the second color part. The third embodiment of the marker type shown above can be an example showing this structure.

That is, in the embodiment of the preferred marker presented in the present invention, the length (L1) in the vertical direction to the drawer movement direction is formed larger than the length (L2) in the drawer movement direction, For the direction of progression of the lower), a color (dark color) that contrasts with the drawer is formed, and the center of which contains a light color part that contrasts with the dark color.

When the printing unit is configured in this way, since the printing unit can be configured in a small size, it is possible to meet the requirements for aesthetics of the user, and the size of the printing unit is relatively small in the entire image captured by the camera. Therefore, the image processing area for detecting the position change of the printing unit is reduced by that amount, so that the image processing speed can be improved. This is advantageous in that the image processing speed of the controller matches the image capturing speed that is continuously photographed and transmitted, so that the accuracy of acquiring an image at a precise time point is further improved. For example, if the size of a portion to be processed in consecutive photographs is large, the load of the controller is high. Therefore, it is necessary to use a very expensive control unit. However, if the size of the portion to be processed is small, the load on the control unit can be reduced. Therefore, even if an inexpensive control unit is used, it is possible to process an image effectively.

Meanwhile, as shown in FIG. 41B, it is preferable that the marker includes a character, a symbol, or a pattern so that the user can intuitively grasp the purpose of the marker by looking at the marker.

Hereinafter, a method of recognizing the marker coordinates of a drawer and processing a camera image will be described.

42 is a diagram illustrating a method of recognizing a marker position. Hereinafter, a method of recognizing a drawer's marker position will be described with reference to FIG. 42.

When a user wants to newly store (stock) or release an item inside the drawer, the user first opens the door. When the door is separated from the front of the storage room, the door switch detects it, and the control unit drives the camera in standby installed in the storage room by receiving a signal from the door switch. When the camera receiving the driving command from the control unit applies a camera driving voltage other than the standby voltage, the camera is ready to be driven and continuous shooting is started.

The user will then try to open the drawer by holding the handle on the front wall of the drawer installed in the refrigerator storage compartment and pulling the drawer. At this time, since the camera is continuously photographing, the open state of the drawer is photographed at regular intervals and an image is transmitted to the controller.

Meanwhile, a marker of a certain shape is formed on a part of the drawer that is exposed to the camera, and the control unit receiving the image sent from the camera divides the entire image into certain pixels and assigns coordinate values at certain intervals, The coordinates of the marker are analyzed on the coordinates of the image.

Therefore, if an image captured while the drawer is closed is sent to the controller, the corresponding image marker portion will not be exposed, and the marker coordinate at this time will have a coordinate of 0.

When the user opens the drawer in the closed state, the camera takes a picture while the marker is exposed, and the camera can take a picture including the marker. When this image is sent to the controller, the controller recognizes the coordinate value of the marker on the entire image and compares it with the previous position. For example, at point'A' in FIG. 42, when the coordinate value of the marker changes from '0' to '50' as a certain coordinate value, for example, the control unit can determine that the drawer is moving from the change in the coordinate value. have. In particular, if it changes from '0', it can be judged that the drawer has just begun to open.

If the user continues to open the drawer, the control unit may determine that the drawer is open because the drawer is moving through a series of processes as described above, and the coordinate value changes to a larger value.

When the user opens the drawer to the fully open state, the control unit recognizes that the coordinate value is no longer changed and determines that the drawer is completely open or that the drawer is open and stopped. For example, the point'B' of FIG. 42 is a point where the coordinate value of the marker has not changed from '90' to '90', so it is recognized as'the drawer is stopped' or'the drawer is completely open'. I can judge.

Normally, after the drawer is opened, whether it is completely open or partially opened, if it is in a stopped state, it can be determined that the user is doing some work on the drawer. Even when the drawer is stopped in this way, the camera continues to capture an image of the interior and sends it to the control unit, and the control unit monitors whether there is a change in the coordinates of the marker.

When the user puts the item into the drawer or removes the item from the drawer, the user closes the drawer again, and the control unit confirms that the marker coordinate value for the entire image has changed from the image captured by the camera. It is determined that the drawer is moving again. For example, point'C' in FIG. 42 is a point where the coordinate value of the marker changes from '90' to '70', and the drawer has just started to close from the change in the direction in which the size of the coordinate value is reduced. It can be determined that it is just after the user has finished working on the drawer.

In the same way, when the coordinate value of the marker is '0', that is, the marker is not recognized from the image, it is determined that the drawer is completely closed.

From the video captured by the camera, the control unit determines the marker position based on the coordinates of one whole picture, and the drawer moves and stops, the drawer closes and opens, the degree of opening the drawer, and whether the drawer starts to open. Explained how to track and judge whether a drawer is starting to close or not.

Hereinafter, a method of processing a camera image according to a marker position of a drawer and a method of obtaining a desired image will be described. A detailed method of how to process and acquire an internal image of a drawer from the moving state of the drawer, in particular, a method of obtaining an image at a point in time that is determined to be immediately after the user finishes work on the drawer.

When it is determined that the drawer's marker coordinate changes to '0' and moves in the direction in which the drawer is opened, the controller detects only the change in coordinates for the image in the cabinet and sent to the control unit. It buffers in a temporary buffer and does nothing for each storage area partitioned in the photo.

When the drawer is kept open and reaches a fully open state or is opened to a certain degree and is stopped, a situation in which the user works on the drawer proceeds. The control unit determines this state from the fact that the coordinates of the marker have not changed, and from this point on, for the transmitted image, the image corresponding to the draw area is drawn from each partitioned storage area in the entire image, and then moved to a temporary buffer and stored. do. In other words, in the section corresponding to point'D' in the above figure, the control unit cuts out the necessary portion from the portion divided into the drawer area from the entire picture in the room and moves it to the temporary buffer for the drawer area.This is the'drawer internal image refresh section. It is defined as'.

In other words, the image processing method is different in the section in which the drawer is open and the section in which the drawer is stationary. The reason is that the user's work on the drawer reflects that the drawer will be stopped. As a result, in the section in which the drawer moves, the image processing time for tracking the change in coordinates can be faster that much, thereby improving the efficiency of data processing.

On the other hand, when the drawer starts to close again from the stop state, the control unit determines that the coordinate value changes to a smaller value, so that it is immediately after the user completes the task, and at that point in the temporary buffer for the internal image of the drawer. Among the temporarily buffered images, the most recent image is selected and stored. In other words, the image of the inner area of the drawer is captured immediately after the user completes the work.

The image of the storage state of the most recent drawer area captured (stored) in this way is replaced with the image of the corresponding area of the display that is online or directly connected to the control unit, or transmitted to the network server. In addition, the latest image transmitted to and stored in the network server is provided so that the user can check the latest image through a display installed in a refrigerator or a mobile terminal connected to the network server whenever desired.

Next, a description will be given of a method of searching for the position of the marker by selecting a specific section, rather than searching the position of the marker for the entire picture taken by the drawer.

That is, it is possible to set the search range for the drawer area. By dividing the degree of opening of the drawer on the movement path of the drawer by section, the method of tracking the marker in each section can be different. Accordingly, it is possible to shorten the processing (analysis) time of a photograph taken and transmitted by the camera, and match the image processing speed and the camera shooting speed.

In general, drawers used in refrigerators are not fully drawn out enough to separate the drawers from the storage compartment and fall to the outside even when the drawers are completely opened. In other words, the drawer is typically drawn out about 50% of the total depth of the drawer (the length before and after the drawer with respect to the direction of movement of the drawer).

The drawer withdrawal distance may be determined as an area to which coordinates of a drawer marker should be assigned in the entire image processed by the control unit. In this case, the control unit can determine the position of the drawer from the point at which the drawer is withdrawn. In particular, in order to detect a change in the marker coordinates of the drawer, the size corresponding to the drawer withdrawal distance by the control unit becomes the data size to be processed. In addition, in order to detect a change in coordinates, the control unit must simultaneously compare and determine two images, and the processing time of this process may incur a burden that must be able to correspond to the speed at which the camera is photographed.

It is preferable that the camera's shooting speed is determined so that the speed at which the camera shoots the inside of the room (the time interval at which a picture is taken) is approximately matched with the speed at which the controller determines and processes from the transmitted image.

If this method is followed, the position of the marker can be determined without delay with respect to the picture taken by the camera, and eventually the position of the drawer can be determined. In addition, it is not unreasonable for the controller to image and process new images continuously sent from the camera while processing images for each area divided for each storage space from the entire picture of the interior. In other words, depending on whether the door is open or the drawer is open from the entire image, the image for each area is cut and moved to a temporary buffer for each area, but all images captured by the camera can be processed without missing. It is done.

However, since data for two images (previous image and current image) of the size described above must be processed, the processing power of the control unit must be increased. Therefore, increasing the processing capacity of the control unit in accordance with these conditions can lead to a rapid increase in the processor unit cost. have. In addition, for image processing when the drawer is not opened, a problem arises in that the processing power is excessively large and efficient selection of parts is not made.

In order to solve this problem, the inventors of the present invention focused on the fact that it is practically difficult to sufficiently grasp the inside of the drawer when the drawer withdrawal distance is not completely open, that is, when the drawer is opened by a certain distance. It provides a method of differently processing data by separating the image corresponding to the drawer area in stages according to the draw distance of the drawer.

43 shows the degree to which the drawer is opened. 43A shows the drawer is opened to the maximum, that is, about 50% of the total length of the drawer is opened, and FIG. 43B shows that the drawer is opened at about 30% of the total length. About 10% of the length is open.

For example, as shown in FIG. 43C, when the user draws the drawer and opens only about 10% of the total length in the front and rear direction, the item stored inside the drawer cannot be sufficiently grasped. When the drawer is opened about 30%, it is possible to grasp the storage state of the items inside the drawer to some extent, and this state is enough to recognize that the drawer is in an open state.

As shown in FIG. 43A, when the drawer is opened to the maximum, about 50% of the entire drawer is withdrawn. In this case, the item inside the drawer can be sufficiently grasped in a completely open state. In other words, when the drawer is in an open state of about 30% to 50%, the user can store or take out items for the drawer. This section can be viewed as a substantially open state of the drawer.

In view of this, if the drawer withdrawal distance is less than 30%, it is determined as'the drawer is not open', and if more than 30% is classified as'the drawer is open', the image processing is different, so that the control unit It can drastically reduce the data load that needs to be processed.

The control unit converts the image captured by the camera into an image, determines the image size corresponding to the drawer movement trajectory, and determines the pixel size from the predetermined drawer movement area to the point at which the drawer is drawn by about 30% or a distance to the image section. Compartment.

That is, when the drawer is drawn in the range corresponding to FIGS. 43A to 43B, the controller may determine the movement of the coordinates of the marker. In this case, compared to the case of determining the movement of the coordinates of the marker in the range corresponding to FIGS. 43A to 43C, the load of the controller required is reduced.

In the section in which the drawer is closed by 30%, it is determined only whether the position of the marker reaches the position of the pixel by the distance corresponding to 30%. In other words, it does not compare the marker coordinates of the previous photo and the marker coordinates of the current photo (a prerequisite for determining the position change of the marker), but simply judges whether the position of the marker is within a certain pixel range, and temporarily buffers the entire image. It is done.

In this section, since there is no need to compare two photo data and only one photo data is processed, the data processing capability of the controller does not need to be large, and it is possible to smoothly respond to the camera shooting speed.

On the other hand, when the drawer exceeds 30% level and is opened more than that, i.e., when the marker reaches a position of a specific pixel or more in the picture processed by the control unit, two pictures (taken just before) are taken to determine the stop state of the drawer. It is possible to determine the coordinate value of the marker by comparing the photographed photo with the photo taken after that.

When the search section is limited in this way, the size of the picture is about 2/5 of the size of the picture compared to that of not limiting the search section. Furthermore, since the size of the picture corresponding to the two pictures to be processed is about 4/5 of the size of the other picture, the data to be processed by the control unit is smaller than or substantially the same as the whole picture. Level can be.

Therefore, since the data processing speed for tracking the coordinate change values of the two photos can correspond to the speed at which the camera is photographed, it is possible to process data (coordinate tracking or monitoring) without missing all images captured by the camera. It is possible to prevent a time lag in processing speed even when the processing capacity of is not increased any more, so that a desired image can be obtained at an accurate time.

In conclusion, as shown in FIG. 44, it is possible to divide a part of the distance that the drawer can withdraw from the picture taken with the camera into a first section and a second section.

In the first section, it is possible to determine the position of the marker with respect to the pixel of the photographed picture. In this case, the first section may determine at which pixel position the marker is located in a single picture without comparing the coordinate values of the markers.

On the other hand, in the first section, even if the drawer is actually open, it is difficult for the user to withdraw or carry the item into the drawer, so it is possible to classify the drawer as in a substantially closed state.

The second section may be set as a'coordinate search range' for comparing coordinate values of markers shown in two consecutively photographed pictures. In the second section, since two pictures are compared, a change in the coordinates of the marker can be detected. For example, if the markers are at the same position in two consecutive photos, it may be determined that the drawer is stopped during the time when the two photos are taken. If the position of the marker is changed in two consecutive pictures, it can be determined that the drawer has moved during the time when the two pictures are taken. In this case, it may be determined whether the drawer is moved in a direction in which the drawer is closed or in a direction in which the drawer is opened according to the direction of change in the position of the marker.

In the second section, it is possible to classify that the drawer is in a substantially open state because the opening is exposed to the user so that the user can withdraw or carry the item into the drawer.

45 is a control flowchart according to an embodiment of the present invention. Hereinafter, it demonstrates with reference to FIG. 45.

It is determined that the door 20 opens the storage compartment 22 (S10). At this time, the opening and closing of the door 20 may be sensed by the door switch 110.

When the door 20 is opened, the door sensor 120 may be driven. That is, the door sensor 120 may detect whether the door 20 is rotated by a predetermined angle θ or more. That is, the door sensor 120 may not operate when the door 20 is not opened, and may be started when the door 20 is opened.

In addition, when the door 20 is opened, the camera 70 may be operated to start taking a picture (S14). At this time, the camera 70 may take 10 pictures at a predetermined time interval, for example, 1 second.

The picture taken by the camera 70 is stored in the storage unit 18 (S16). At this time, when a plurality of photos are taken by the camera 70, the recently taken photos are stored, and the photos taken before that may be deleted due to the capacity limitation of the storage unit 18.

Meanwhile, it may be detected whether the drawer 50 is opened by the drawer detection unit 130 (S20). The drawer detection unit 130 may detect that the drawer 50 is pulled out from the lower space of the shelf 40.

And when the drawer detection unit 130 detects whether the drawer 50 is closed, the picture taken at the time or the picture taken at the closest time to the time is selected as the final picture (drawn and opened) ( S40). Since the camera 70 takes a predetermined number of pictures in one second, it is possible that the pictures are not accurately taken at the point when the drawer 50 starts to close. However, if the drawer detection unit 130 determines the movement of the drawer 50 by analyzing the movement of the marker expressed in the photo taken by the camera 70, the photo is taken at the time when it is determined to be closed. It is possible to choose.

The control unit 100 divides the final photo into the first area 42, the second area 52, and the third area 62 (S42). In this case, the control unit 100 may classify each area by specifying the number of pixels based on the adjustment line 15 and obtain a final image for each area.

As described above, when the drawer 50 is withdrawn, the second region 52 and the third region 62 overlap each other.

Meanwhile, since information indicating that the drawer 50 has been withdrawn is obtained in S20, it may be determined that the second image has been obtained by the control unit 100.

In addition, information on the second area 52 may be updated (S46). The controller 100 may transmit a second image showing the second area 52 to the display 14 and change the corresponding part to be based on the final picture.

On the other hand, if it is determined in S30 that the drawer 50 does not start to close, the picture may be taken again as in S14.

The state in which the drawer 50 is not closed in S30 may be regarded as that the user has not finished accessing the second area 52. That is, the user may store new food in the second area 52 or take out the stored food from the second area 52 while the drawer 50 is stopped.

On the other hand, when the drawer 50 is closed, it can be predicted that the user ends the access to the second area 52 and pulls the drawer 50 into the lower space of the shelf 40.

Meanwhile, in a state in which the drawer 50 is not opened in S20, it may be determined whether the door 20 is just before closing (S60).

At this time, whether the door 20 is just before closing may be detected by the door sensor 120.

When the user ends the access to the storage compartment 22, the door 20 is closed and the storage compartment 22 is sealed. That is, the user can store new food in the storage compartment 22 or take out the stored food from the storage compartment 22 with the door 20 open to the opening 14.

If the user closes the door 20, it may be determined that the food is stored in that state until the user ends access to the storage compartment and the door 20 is opened again.

If it is determined in S60 that the door 20 is just before closing, the control unit 100 selects a picture taken at the time or a picture taken at the time closest to the time as the final picture (drawn and closed) (S70). That is, the control unit 100 may select a photograph taken at a point in time when photographing by the camera 70 is finished or a photograph taken at a point in time adjacent to the point in time as the final picture.

The control unit 100 divides the final photo into the first area 42, the second area 52, and the third area 62 (S72). In this case, the final photo may include a first image showing the first region 42. The latest state of the food stored in the first area 42 is that the user has finished accessing the storage compartment 22. Accordingly, through the first image obtained at this time, the user can obtain accurate food information for the first area 42.

Meanwhile, since information indicating that the drawer 50 has been inserted is obtained in S60, it may be determined that the first image and the third image have been obtained by the control unit. As described above, when the drawer 50 is withdrawn, the second region 52 and the third region 62 overlap each other, but the first region and the third region do not overlap each other. This is because it is a location where you can shoot together.

In addition, information on the first area 42 and the third area 62 may be updated (S76). The control unit 100 transmits a first image showing the first area 42 and a third image showing the third area 62 to the display 14, so that the corresponding part is based on the final photo. Can be changed to

On the other hand, although not shown in the drawing, the door 20 is rotated to seal the storage compartment 22, and when the storage compartment 22 is closed, the driving of the camera 70 may be stopped. At this time, the door switch 110 may detect whether the door 20 seals the storage compartment 22.

That is, in the present invention, in consideration of the point in time when the user terminates use, the photograph obtained at the point in time is processed to provide the user with information on the food stored in the corresponding area. Accordingly, the user can obtain accurate information about the food stored in the storage compartment 22.

Meanwhile, the door sensor 120, the door switch 110, and the drawer detection unit 130 may be installed two at each of the left and right sides. In this case, individual information on the left and right doors and the left and right drawers can be obtained. Of course, if the drawer detection unit 130 detects the movement of the drawer through the analysis of the marker in the taken picture, the drawer detection unit 130 is one individual, whereas the marker is installed on each of the two drawers. It is possible to become.

Therefore, based on this information, when only the left door is opened, it is possible to update only the left part of one picture. On the other hand, if only the right door is open, it is possible to update only the right part of one picture.

Of course, when the left door and the right door are opened together, if the above-described conditions are satisfied, it is possible to update the left part and the right part of a single picture together and provide it to the display 14.

46 is a control flowchart illustrating a modification example of FIG. 45. Hereinafter, it demonstrates with reference to FIG.

In the modified example of FIG. 46, S60 is a form specifically divided into S62 and S64, and only the difference between them will be described in detail.

That is, whether the door 20 is just before closing can be specifically determined by determining whether the door 20 is rotated in the closing direction (S62) and whether the door 20 is rotated by a predetermined angle θ or more. have.

In other words, when the door 20 is rotated in the closing direction (S60), and the door is rotated below a certain angle (S64), photographing by the camera 70 may be terminated.

In order to obtain information on the food stored in the storage compartment 22 from the photos taken by the camera 70, interference with the basket installed in the door 20 as well as the door 20 should not occur. Therefore, the expression that the door 20 is just before closing may mean a state in which the door 20 and other components are not expressed in the picture taken by the camera 70.

47 is a control flowchart illustrating another modified example of FIG. 46. Hereinafter, description will be made with reference to FIG. 47, but only the difference from FIG. 46 will be described.

In the other modified example according to FIG. 47, unlike FIG. 46, the rotation direction of the door 20 is not determined.

If it is determined in S12 that the door has been rotated by a predetermined angle or more, the door 20 is rotated in a direction opened at the corresponding moment. Therefore, it may be the case that the user closes the door 20 when the user rotates the door 20 again and reaches a state in which the door 20 is rotated by a predetermined angle.

Therefore, in another modified example, if S62 in FIG. 46 is omitted and it is determined that the drawer is not opened in S20, S64 to determine whether the door is rotated below a certain angle is immediately performed.

48 is a control flowchart illustrating another modified example of FIG. 45. Hereinafter, it demonstrates with reference to FIG. 48.

In FIG. 48, a control flow of detecting the movement of the drawer 50 by a marker will be described. For convenience of explanation, overlapping parts are omitted, and only parts that differ will be described.

Overall, S20 and S30 may be specifically divided into steps S22, S24, and S26.

After the picture is stored in the storage unit 18 as in S16, it is determined whether the marker is recognized by the camera 70 (S22).

Since the camera 70 takes a picture from the upper side of the inner case 12 to the lower side, a case in which a marker is not visible from the camera 70 may occur. That is, when the handle portion of the drawer 50 is covered at the front end of the shelf 40, that is, when the drawer 50 is not drawn out at all, the marker may not be recognized.

When a marker is recognized in the picture taken by the camera 70, it may be determined whether the marker is stopped (S24). At this time, as described above, it is possible to determine the movement path of the marker in one picture and then divide it into a first section and a second section, and determine the movement of the marker only in the second section.

Since the camera 70 captures a plurality of pictures at predetermined time intervals, it is possible to determine whether the marker is moved by comparing and analyzing the pictures. Since the marker is displayed on the drawer 50, it can be determined that the drawer 50 is also moved when the marker is moved.

In addition, it may be determined whether the marker has moved more than a certain distance from the position where the drawer 50 is closed (S26).

If the marker has not moved more than a certain distance, since the drawer 50 is not sufficiently opened, an appropriate image for the second area 52 cannot be obtained. For example, in a picture taken with only 1/3 of the drawer 50 drawn out, the inner space of the drawer 50 is not sufficiently shown, and the image obtained from the picture shows the user the second area This is because information about (52) is not available. In this case, the predetermined distance may be determined as whether the marker has entered a search range for determining the movement of the marker.

If it is determined that the marker has moved more than a certain distance, it can be considered that the user has finished accessing the second area 52 and has finished using the second area 52.

Accordingly, a photograph obtained at a time when the three conditions are satisfied or a time adjacent to the time may include information on the food stored in the second area 52.

49 is a diagram illustrating a modified example in which FIG. 45 is modified differently. Hereinafter, it demonstrates with reference to FIG.

In FIG. 49, unlike FIG. 45, there is a difference in that the camera is not immediately driven when the door 20 is opened by the door switch 110.

When the door switch 110 determines that the door 20 is open, the door sensor 120 is driven.

Then, the door sensor 120 determines whether the door 20 is rotated by a predetermined angle or more (S12). In this case, when the door 20 is not rotated by a predetermined angle or more, the camera 70 does not take a picture.

When the door 20 is rotated by a predetermined angle or more, the camera 70 takes a picture (S14). At this time, the camera 70 may take 10 pictures at a predetermined time interval, for example, 1 second.

The photograph taken at this time includes a first image of the first region 42. Meanwhile, since the door 20 is rotated by a certain angle θ or more, interference from a basket installed on the door 20 may not occur in the photo. That is, in the picture, the shape of the basket installed on the door 20 may not appear.

Furthermore, in S60, the door sensor 120 may detect whether the door 20 is just before being closed. In this case, the meaning that the door 20 is just before closing may mean a state in which the door 20 is rotated at a predetermined angle or less while the door 20 is rotated in the closing direction.

Until just before the door 20 is closed, the camera 70 may take a picture and stop taking a picture (S49). In FIG. 45, the camera 70 takes a picture until the door 20 is closed and the door switch 110 is pressed. However, in a modified example, when the door 20 is rotated by a certain angle by the door sensor 120, the camera 70 stops taking a picture. In this case, the predetermined angle may mean an angle that prevents interference caused by the door 20 as well as the basket installed on the door 20 in the picture taken by the camera 70.

50 is a control flowchart according to another embodiment of the present invention. Hereinafter, it demonstrates with reference to FIG. 50. The control flow of FIG. 50 may be continuously performed at intervals of a predetermined time when it is determined that the door 20 is opened by the door switch 110.

First, it is determined whether the door 20 is opened by the door sensor 120 at a predetermined angle θ or more (S100).

When the door 20 is opened by a certain angle or more, it is determined that interference does not occur in the door 20, and the camera 70 takes a picture (S110). At this time, the camera 70 may take a single picture.

In addition, it is determined whether the drawer 50 has been withdrawn more than a first set distance (S120). In this case, the first set distance may mean a distance at which food information on the second area 52 can be sufficiently obtained when a picture is taken by the camera 70. For example, the first set distance may mean 2/3 or 1/2 of the total length of the drawer 50.

The draw distance of the drawer 50 may be detected by the drawer detection unit 130.

On the other hand, if it is determined that the drawer 50 has been withdrawn by the first set distance or more, the controller 100 may divide the photo by area and divide it into a plurality of images (S122).

Since the picture is obtained while the drawer 50 is drawn out, a first image and a second image may be obtained from the picture.

The controller 100 may update the first image and the second image (S124).

On the other hand, if it is determined in S120 that the drawer 50 has not been withdrawn more than the first set distance, it may be determined whether the drawer 50 has been withdrawn less than the second set distance (S130).

In this case, the second set distance may mean a distance at which food information on the third area 62 can be sufficiently obtained when a picture is taken by the camera 70. For example, the second set distance may mean 1/3 of the total length of the drawer 50. Of course, it is possible to select a state in which the drawer 50 is completely retracted into the lower space of the shelf by selecting '0' as the second set distance.

On the other hand, if it is determined that the drawer 50 is drawn out less than the second set distance, the control unit 100 may divide the photo by area and divide it into a plurality of images (S132).

Since the picture is obtained while the drawer 50 is drawn out, a first image and a third image may be obtained from the picture.

The controller 100 may update the first image and the third image (S134).

In another embodiment of the present invention according to FIG. 50, when the door 20 is opened, control is repeated, and when the door 20 closes the storage chamber 22, the control is completely terminated. In the terminated state, since the image is no longer updated, a final image of the food stored in the storage compartment 22 can be obtained.

51 is a view for explaining a process of updating an image in a state in which two doors and two drawers are provided to open and close the storage compartment. Hereinafter, it demonstrates with reference to FIG.

The overlapping content in the above-described embodiment will be briefly described.

When electricity is supplied to the refrigerator, it may be individually determined whether the left door or the right door is opened by the door switches 110 respectively installed on the left and right sides (S302 and S303).

If the door switch 110 determines that either the left door or the right door is open (S303), the camera 70 starts taking a picture (S304).

The camera 70 can take a picture even if one of the left door and the right door are opened, as well as at a time interval between the left door and the right door, or both are opened at the same time. This is because, when it is determined that the door is opened through one of the door switches, the interior of the room is lit.

When the drawer disposed on the left is drawn out more than 130mm (S310), an image related to the left drawer is captured (S312). The image related to the left drawer may refer to a part corresponding to the left drawer among the photographed photos, and may refer to the entire photographed photo.

At this time, 130mm may be changed by the refrigerator manufacturer or the user, but may mean a withdrawal distance such that the user can withdraw the drawer and access the storage space inside the drawer.

If the left drawer is drawn out less than 130 mm (S310), the image immediately before photographed by the camera 70 is maintained (S310).

And the left door sensor detects the opening angle of the left door. At this time, if the left door is opened at an angle less than 80 degrees, an image of the left multi-storage space provided in the lower space in which the left drawer is drawn is captured (S332).

In this case, 80 degrees may mean an angle such that various structures installed in the left door and the left door do not interfere when photographing the storage space by the camera 70. It is possible to change 80 degrees according to the specific structure or size of the refrigerator.

If the left door is opened at an angle greater than 80 degrees, it is possible to maintain the image immediately before photographed by the camera 70 (S336).

In addition, closing of the left door may be detected by the door switch 110 (S334).

Meanwhile, for the right drawer and the right door, it is possible to implement the same type of control flow as described for the left drawer and the left door. That is, in S320 to S324 and S340 to S346, since the above-described contents are applied in the same manner, a detailed description is omitted.

When the left door and the right door are closed, it is possible to process the image captured above by the control unit 100 and display the processed image on the display 14 (S350). In this case, the display may be a display installed in a refrigerator.

In addition, the camera may stop taking a picture (S352).

Meanwhile, the refrigerator may determine whether communication is possible by being connected to Wi-Fi (S354), and if communication is possible, it may transmit an image to an external server (S356).

The server determines whether or not the application of the terminal used by the user is synchronized (S358), and if it is synchronized, the image can be displayed on the display 14 of the terminal.

Therefore, even if the user is not in front of the refrigerator, information on the items stored in the refrigerator can be provided through the user terminal.

If the refrigerator is not connected to Wi-Fi and communication with the outside is not possible, or if the application of the terminal is not synchronized, it is desirable to maintain the previous image on the display 14 of the user terminal and provide the user with the immediately previous image. Do.

Hereinafter, components provided to take a picture and obtain a picture at a desired point in time will be schematically described.

When the door switch detects that the user has started opening the door, the controller starts to take a picture of the interior by driving the camera. The camera continuously shoots at a certain frame per second. Since the act of opening the door of the user means to retrieve a desired item from within the refrigerator or to store a new item in the refrigerator, the camera continuously photographs the inside of the refrigerator. You can monitor whether there is any change in the state of the storage space.

The storage rooms are separated into various shelves, drawers, baskets, etc. to form storage spaces, respectively, and are stored and stored in the storage space according to the shape or contents of an article or container.

The camera may monitor the storage space by dividing it into a shelf area divided by a shelf, a drawer area, and other storage areas. Through this, it is intended to enable the user to check the storage status of the goods in the area immediately after the user completes the withdrawal or warehousing operation. If two drawers are provided and two other storage areas are provided in parallel on both sides, the camera will provide information for the user to check each of the two drawers and the two other storage areas. I can.

In one embodiment of the present invention, the other storage area is formed on the bottom of the storage room, is located in front of the drawer storage area, and when the drawer is opened, at least part of the storage area is set as a storage space that overlaps, and includes it as a monitoring target. present.

When the door switch detects the opening of the door, the control unit turns on a door sensor included in the hinge assembly of the refrigerator to detect movement according to the rotation of the door. The door sensor may be always turned on, but the door sensor may be turned on only when the door is opened by the door switch.

When the door is opened by a certain angle or more, the control unit determines that the door is substantially open so that the user can access the storage room and carry in or take out items, and determine the storage state of the shelf area as a target to be upgraded.

In addition, when the drawer is opened, the drawer detection unit detects the degree of opening of the drawer, and when the drawer is opened, the storage state of the drawer area is determined as a target to be upgraded, and when the drawer is not opened, the storage area is to be upgraded. To decide.

The door sensor according to an exemplary embodiment of the present invention detects when a certain angle has elapsed in the process of opening and closing the door and sends related information to the control unit. Here, the door sensor may also detect whether the door is rotated in a direction in which it is closed or in an open direction.

In an embodiment of the present invention, when an on signal of a door switch is detected, the control unit turns on the door sensor, and the door sensor detects when the door passes a certain angle and sends the corresponding time point to the control unit. It becomes. The control unit determines that the door is open at this time. Specifically, the door is opened by rotating by a certain angle at this time.

Meanwhile, when the user's withdrawal or warehousing activity is completed while the door is open, the opened door is closed. When the door passes the door sensor, the door sensor sends a signal to the control unit, and the control unit sends a signal to the control unit. It is judged that it is just before the door is closed after the operation of entering/exiting the door. Specifically, the door is opened by being rotated by a certain angle at this time, but is in a state in which it is rotated to close.

An exemplary embodiment of the present invention will be described in detail with respect to the process of taking an image of the interior from the camera during the door opening and closing process and upgrading the image to the image taken immediately after the user finishes entering/exiting the item.

When the refrigerator door switch is turned on while the door is closed, the controller turns on the camera and the door sensor. The camera may mean switching from a standby mode to a driving mode, and the door sensor may mean switching to a state capable of determining the opening angle of the door.

The camera may continuously photograph the inside of the refrigerator at a predetermined speed per predetermined time. The captured image is sent to the controller, and the controller temporarily buffers the transmitted image as an image including the entire interior.

An image [Picture (or photograph)] and an image used in the present specification may be used separately from each other.

Specifically, an image (picture or photograph) refers to raw data that is immediately transmitted after being photographed by a camera, and an image is the raw data (picture or photograph) by the control unit. This may mean buffered, corrected, or transmitted data.

When the door switch is turned on and the door is opened at a certain angle and the control unit determines that the door is open, the control unit determines which area the entire image contains the storage state of the item according to the drawer's open state. To decide.

For example, when the drawer is in an open state, the control unit includes an area (shelf area) in which the entire stored image is stored on a refrigerator shelf, and a drawer storage area (drawer area) stored inside the drawer. It is judged to contain the storage condition of the item. On the other hand, if it is determined that the drawer is in a closed state, the control unit includes an area inside the storage space (storage area) that is an area in which the entire stored image is formed on the refrigerator shelf area and the front end of the drawer and overlaps by opening the drawer. It is judged that it contains the storage status of the goods.

Here, a picture of the entire refrigerator storage room taken and transmitted by the camera is divided into three different storage spaces, the shelf area is always included, and the drawer area and the storage area are It is selectively included according to the opening and closing state of the lower. The drawer area and the storage area do not coexist with each other in the photo.

Accordingly, one entire picture transmitted from the camera includes at least two areas, one of which is a shelf area, and which storage area the remaining part corresponds to is determined by the drawer open/close state.

In the present invention, the drawer is intended to transmit the latest in/out status of the item to the user by photographing two different spaces including at least the inside of the drawer, such as the drawer's inner space and another storage space, with one camera. Another embodiment of the present invention may be a method of performing an upgrade by fixing only the area and the shelf area, and fixing only the drawer area and the storage space may be an embodiment of the present invention. .

On the other hand, the division of a whole picture of the storage compartment constituting one independent insulating space in the refrigerator into storage areas in the storage compartment may be determined by the number of doors opening and closing the storage compartment. For example, in a refrigerator that opens and closes the storage room with a single door, the storage area to be photographed can be divided into two or three types. When the storage room is opened and closed with two left and right doors, the storage area is It can be largely divided into 4 or 6 types.

In this case, the drawer area and the storage area may be divided into left and right sides and managed as two divided areas in the entire picture, and the shelf area may be integrated and managed as one area. In this case, all of the partitioned parts in the whole picture may be composed of 5 types.

In this case, the storage spaces facing the left and right doors are connected to each other in the shelf area. In the shelf area, there may be cases where items, foods, or vegetables are stored across the left and right sides. If stored across the shelf, items can be withdrawn or received by opening only one of the left and right doors, so even if only one of the left and right doors is opened, the entire shelf area is integrated from the entire image and replaced with the latest data. Because there is a need. In consideration of this point, even if only one of the left and right doors is opened, the entire shelf area can be updated.

In more detail, when it is determined that the drawer is opened while the door is open, the control unit separates the shelf area and the drawer area from the transmitted whole picture and temporarily buffers it. And, when the user finishes the act of putting in/out items in the drawer area or/and the shelf area, and attempts to close the drawer, that is, when the drawer starts to move in a stopped state after opening, the most recent temporary buffered video The required part in the drawer area is captured (captured or cut) from the image taken in and stored as an image of the latest drawer area in the storage unit. In addition, the captured image of the drawer area in the image inside the refrigerator displayed on the refrigerator display is replaced with the latest image to be upgraded. In addition, the control unit transmits the captured image of the drawer area to the server system connected to the network.

On the other hand, when it is determined that the drawer is closed because the drawer is not opened in the open state, the control unit determines that the entire picture of the transmitted image includes the shelf area and the storage area. Accordingly, the shelf area and the storage area are separated from the entire picture and temporarily buffered.

When the user completes the work of putting in and leaving the goods in the storage area and/or the shelf area and then closes the door, the door sensor detects when the door passes a specified angle and transmits a signal to the control unit. The control unit determines that the door is just before the door is closed by the signal transmitted from the door sensor (it may be expressed as just before the picture is obscured by an obstacle such as a door), and is received from the most recently captured image among the temporary buffered images. A required portion of the area is captured and stored in the memory as an image of the latest storage state of the storage area. In addition, the image in the refrigerator compartment displayed on the display is upgraded by replacing the captured image for the storage area.

In this case, the upgrade may mean that the existing image is replaced with the latest image. In addition, the control unit transmits the captured image of the drawer area to the server system connected to the network.

On the other hand, the drawer is in an open state or in a closed state, or the shelf area captures the most recent image from the images being buffered when it is determined that the door is just before closing, and is stored in the memory as a storage state of the latest items in the shelf area.

The displayed image on the refrigerator display is replaced by the captured image (a recent shelf area image, a recent drawer area image, or a recent storage area image, or both), or transmitted to a server system or connected through communication with the refrigerator. Transmission of a user to a mobile device of a user may be performed simultaneously while storing the captured image in the refrigerator memory, and may be performed at once when a user requests it, or may be performed by a user request signal for each individual area.

Here, one image transmitted from the camera may be buffered as one image by the controller. If it is determined that the one image is taken while the door is open, it may be determined whether the drawer area or the storage area is included in the one image according to the drawer opening/closing state. As such determination is made, the corresponding image is extracted and buffered in separate temporary buffers, respectively.

If it is determined that one image was taken at the point when the drawer starts to close, the most recent image among the temporary buffered images is stored in the memory, and when a new input/output operation by the user is completed, the latest image is replaced. . In addition, if it is determined that the door is just before closing, the most recent image among the images of the temporary buffered storage area is stored in the memory.

On the other hand, the shelf area can be stored in the memory by extracting only the shelf area from the most recent image immediately before the door is closed from the entire image being buffered, and a temporary buffer is prepared for a separate shelf area, and only the shelf area is extracted and buffered. May be.

Hereinafter, various embodiments in which the most recent image inside the refrigerator obtained by various methods described above is transmitted to and used in a personal mobile terminal or a business server will be described with reference to FIGS. 52 to 55.

52 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 52, when it is determined that the final photo of a specific area inside the refrigerator has been obtained through the steps described with reference to FIG. 34 (S1101), the terminal (S1101) converts the corrected final image into an event format. Transfer to 2) (S1103). The terminal 2 may include a smart phone, a personal digital assistant (PDA), a tablet PC, and the like, but is not limited thereto. At this time. The controller 100 may transmit not only the final image inside the refrigerator, but also various types of information related to the final image described below to the terminal 2.

At this time, the control unit 100 may transmit the final image to the terminal 2 as soon as the final image is obtained through the final photo, or store the obtained final image in the storage unit 18 and then according to a predetermined cycle. An image stored in an event format may be transmitted to the terminal 2, but is not limited thereto.

Subsequently, the terminal 2 displays the final image received according to the user's selection (S1105). For example, the terminal 2 may execute an application capable of receiving and displaying an image inside the refrigerator according to a user's selection, or may display the received image inside the refrigerator by performing a multimedia service.

According to the embodiment of the present invention, even if the user does not directly look inside the refrigerator, the image inside the refrigerator can be checked using the mobile terminal, so that there is a practical advantage in that the current internal situation of the refrigerator can be recognized remotely.

53 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 53, when it is determined that the final photo of a specific area inside the refrigerator has been obtained as described above (S1301), after generating an image by dividing the final photo by area, the obtained final image is Control to be stored in the storage unit 18 (S1303).

Subsequently, when a user input for displaying a current internal image of the refrigerator, that is, a final image of a specific area of the refrigerator is received (S1305), the terminal 2 transmits a final image request command (request) to the refrigerator (S1307).

In addition, the control unit 100 transmits the most recently photographed and stored image inside the refrigerator among the images stored in the storage unit 18 to the terminal 2 in response to the final image request command. 270) is controlled (S1309).

According to such an embodiment of the present invention, communication between the mobile terminal and the refrigerator is performed only when the user needs it, so that it can be more efficient and economical. Hereinafter, only a case in which an image is transmitted as a response to a request is described, but the case in which an image is transmitted in the form of an event as described with reference to FIG. 35 is also included, and is limited thereto. It does not become.

54 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 54, when it is determined that a final picture of a specific area inside the refrigerator is obtained as described above (S1501), the control unit 100 controls to store the final image obtained from the final picture in the storage unit 18. (S1503).

Subsequently, when a business server such as the market server 3 receives a user input for displaying the current image inside the refrigerator (S1505), the market server 3 transmits a final image request command to the refrigerator (S1507). . In this case, the market server 3 may transmit a final image request command to the refrigerator according to a predetermined period without user input.

In addition, the control unit 100 transmits the most recently photographed and stored final image inside the refrigerator among the images stored in the storage unit 18 to the market server 3 in response to an image request command ( S1509).

Upon receiving the final image inside the refrigerator, the market server 3 analyzes the final image (S1511). The market server 3 analyzes the final image, finds out the storages currently stored in the refrigerator and the storages that are not currently stored, and analyzes the storages that were stored in the refrigerator in the past but not currently stored. You can extract the information necessary to provide services such as.

The market server 3 transmits the list of insufficient items extracted through the final image analysis to the corresponding refrigerator (S1513).

55 is a ladder diagram illustrating a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 55, when it is determined that the final picture of a specific area inside the refrigerator has been obtained as described above (S1701), the control unit 100 controls to store the corrected final image from the final picture in the storage unit 18. (S1703).

Subsequently, when a business server such as the broadcasting station server 4 receives a user input for displaying the final image inside the refrigerator (S1705), the broadcasting station server 4 transmits a final image request command to the refrigerator (S1707). . In this case, the broadcasting station server 4 may transmit a final image request command to the refrigerator according to a predetermined period without user input.

In addition, the control unit 100 transmits the most recently photographed and stored image inside the refrigerator to the broadcasting station server 4 as a response to the final image request command. Control (S1709).

Upon receiving the final image, the broadcasting station server 4 analyzes the final image (S1711). The broadcasting station server 4 analyzes the final image, finds out a storage object currently stored in a corresponding refrigerator, and extracts information necessary for service provision, such as analyzing dishes that can be made using the currently stored storage object.

The broadcasting station server 4 transmits recipe information of a dish that can be made using the currently stored storage material to the corresponding refrigerator through final image analysis (S1713).

The control unit 100 displays, on the display 14 installed in the refrigerator, recipe information of dishes that can be made using the currently stored storage material received from the broadcasting station server 4 (S1715).

According to an exemplary embodiment of the present invention, there is a practical advantage in that a related business operator can accurately identify the internal situation of a refrigerator located at a home or a business and provide a refrigerator-related service suitable for a corresponding home or business.

FIG. 56 is a diagram illustrating the operation of a heater of the camera, and FIG. 57 is a diagram illustrating an experiment result of condensation generated for each temperature in a transparent window of the camera. Hereinafter, it demonstrates with reference to FIG. In FIG. 56, the heater of the type described in FIGS. 14 to 17 will be described as a reference.

Figure 56 (a) is a graph of a form in which power is intermittently supplied to the heater 84 and is intermittently operated, and Figure 56 (b) is a form in which the heater 84 is continuously supplied with power and is continuously operated. It is a graph.

56(a) shows a method in which the temperature of the transparent window 80 is not normally controlled, but is maintained in an equilibrium state with the temperature of the storage compartment 22, and then a high power is instantaneously applied when a refrigerator camera needs to be photographed. In particular, by operating the heater 84 immediately before taking a picture, condensation generated on the transparent window 80 may be removed.

As the door 20 is opened, the transparent window 80 may contact moisture contained in the warm outside air that has entered through the door 20. At this time, since the transparent window 80 has a relatively low temperature compared to the outside air, moisture contained in the outside air condenses in the transparent window 80, resulting in condensation.

In order to remove the dew generated by this condensation phenomenon, if the heater 84 is momentarily driven, there may be a problem that it takes time until the dew of the transparent window 80 is actually removed. Therefore, if the time when photographing is necessary is earlier than the time when the dew is removed, there may be a problem that dew may be condensed on the transparent window 80.

On the other hand, FIG. 56(b) shows a method in which the heater 84 is continuously driven so that heat is supplied to the transparent window 80. Since the temperature of the transparent window 80 is always maintained above the temperature at which condensation occurs, condensation does not always occur in the transparent window 80. Accordingly, even if the door 20 is opened and the transparent window 80 contacts the outside air, moisture contained in the outside air is not condensed on the transparent window 80 as dew. Accordingly, it is possible to prevent dew from forming on the transparent window 80 at the moment when the camera 70 should take a picture.

In driving the heater 84, the image quality problem due to condensation should be avoided, and the power consumption of the refrigerator should not be rapidly increased due to the power consumption of the heater 84. It is also desirable to consider that the heat supplied by the heater 84 should not affect the interior temperature.

The inventors of the present invention considered that it is necessary to operate the heater by increasing the time to turn off the heater in order to minimize the power consumption of the heater and not affect the interior of the refrigerator. However, if you set the time to turn off the heater for a long time, you are concerned that condensation will occur, so you set the target temperature on the transparent window of the camera, and the outside temperature of the refrigerator is 32 degrees Celsius, and the relative humidity is 85% (the dew point temperature is approximately Celsius). The experiment was conducted under conditions of 29 degrees to 30 degrees) (see FIG. 57). In FIG. 57A, the y-axis indicates the amount of condensation generated in the transparent window, and in 57b, the y-axis indicates the percentage of the amount of condensation actually generated at a corresponding temperature compared to the current amount of water vapor.

The on/off operating cycle of the heater derived from repeated experiments could deteriorate the image quality of pictures taken by the camera under special temperature conditions inside the chamber.

Even if an appropriate on/off period of the heater is found, a user's refrigerator usage pattern during the time when the heater is turned off may be unpredictably varied.

For example, when the user has opened the refrigerator door for a long time, the camera cover glass (the cover glass may mean the above-described transparent window, or various types of windows placed on the front end of the camera lens). There is a large amount of condensation on the surface. In this case, even if the cover glass is heated by operating the heater, since evaporation takes a lot of time, there is a problem in image quality when photographing is performed at that time. E Also, it took a lot of time as well as a lot of power to evaporate the already generated dew from the cover glass.

In addition, when the storage compartment is a refrigerating compartment, the interior temperature is usually maintained between 0°C and 7°C, and the internal volume is also larger than the cover glass surface area. Heat supplied by the heater is inevitably leaked out of the cold air inside the storage compartment. Therefore, in order to evaporate the dew generated on the cover glass, a large amount of heat must be applied for a short period of time. In order to intensively supply such heat for a short period of time, a heater having a large capacity is used, and the instantaneous power used is greatly increased. Therefore, various problems may occur in a poor power condition.

Therefore, it is preferable that the amount of condensation above a certain level is not continuously generated from the beginning in the cover glass, and in the present invention, power is applied so that the heater is constantly operated without an on/off cycle.

The method of constantly applying power to the heater may be configured to apply power to the camera heater as soon as power is applied to the refrigerator, or control such that the storage compartment always applies power to the heater during normal operation through the refrigerator control unit. have.

Next, the heating temperature of the cover glass will be described. As a result of performing quality evaluation based on the image samples obtained for each temperature range, it was confirmed that there was no significant problem in the quality of the sample image until the temperature of the central part of the cover glass was approximately ½ of the dew point temperature. That there is no problem in image quality may mean a degree to be able to grasp the type and quantity of items stored in the storage room through the photographed picture.

In particular, the quality of the image sample obtained at 13.5 degrees, point (3) of FIG. 57B, was at an applicable level. Of course, it is possible to maintain the heating temperature of the cover glass above the dew point, but power can be supplied to the heater so that the heating temperature of the cover glass is maintained between about 13 degrees Celsius and about 30 degrees Celsius.

Meanwhile, according to the experimental results, the heating temperature of the cover glass differs according to the outer size of the cover glass under the condition that the same applied power is supplied, and there was no significant difference in the thickness of the cover glass. Therefore, since the image quality was satisfied at the level of 1/2 of the dew point temperature in the same cover glass size, it is desirable to set this as a lower limit, but to select an appropriate upper limit in terms of power consumption (applied power) of the heater.

When heating the cover glass, the applied power near the dew point temperature (29.2 degrees) was 0.45 W, and the applied power at about 0.58 times the dew point temperature (17.1 degrees Celsius) was 0.27 W. Therefore, in the range of 0.5 to 0.7 of the dew point temperature, the power consumption is also approximately 0.5 times, so that a great effect in reducing the power consumption can be seen.

In general, an intermediate band of the dew point temperature may be determined from about 12 degrees to 17 degrees Celsius under variable ambient conditions. Accordingly, it is preferable that a current value is applied to the heater to maintain the temperature of the central portion of the transparent window at approximately 12 to 17 degrees Celsius.

That is, in the present invention, it is possible to obtain an effect of reducing the power consumed by the heater by maintaining the temperature of the cover glass at a temperature lower than the dew point rather than the dew point.

58 is a cross-sectional view of a transparent window. Hereinafter, it demonstrates with reference to FIG. 58.

The transparent window 80 may be hydrophilic coated. In this case, the transparent window 80 may mean the above-described cover 90.

Even if dew is instantaneously formed on the transparent window 80, the contact angle α of water may be formed within 15 degrees due to the hydrophilic coating as shown in FIG. 58. When the hydrophilic coating is made, the surface tension of water on the surface decreases. Therefore, this is because water can spread widely on the surface of the transparent window 80. Accordingly, the camera may be fixed to the ceiling surface of the storage compartment so that the angle between the transparent window and the ceiling surface of the storage compartment is approximately between 10 degrees and 20 degrees.

Therefore, it is possible to minimize distortion of the photographed picture due to water condensed on the transparent window 80. In addition, since the hydrophilic coating does not require control such as supplying additional electricity after being manufactured once, it can have an advantage in terms of energy efficiency.

59 and 60 are views schematically showing a structure in which a camera is installed in an inner case. For reference, FIGS. 59A and 60A are views as viewed from the side of the storage compartment, and FIGS. 59B and 60B are views as viewed from the upper side of the inner case. Hereinafter, a description will be given with reference to FIGS. 59 and 60. Through these embodiments, it is possible to minimize the occurrence of condensation in the cover 90 by bypassing the path leading to the cover 90 by external air.

The camera 70 may be installed on the protrusion 500 protruding downward from the ceiling of the inner case 12. In this case, the protrusion 500 may mean a portion protruding downward compared to other portions of the inner case 12. This protrusion 500 may be implemented through the above-described camera housings.

In this case, as shown in FIG. 59, the camera 70 is installed within the protrusion 500 to face the inner wall of the storage compartment 22. The camera 70 may take a picture of the inside of the storage room 22 through the cover 90. Accordingly, the cover 90 is disposed at a position facing the rear wall of the storage compartment 22 from the protrusion 500.

At this time, the cover 90 is cooled while moisture contained in the outside air comes into contact with it to generate dew.

However, since the cover 90 is disposed so as to face the rear wall of the storage compartment 22, a path through which outside air reaches the cover 90 may be lengthened.

That is, in order for the outside air passing through the door 20 to pass through the storage compartment 22 and contact the cover 90, the outside air must pass through a path as shown in FIG. 59B. Some of the moisture it contains may condense in other areas. Therefore, outside air may reach the cover 90 while the amount of moisture in the storage chamber 22 is gradually reduced.

That is, even if the door 20 is open, the interior of the storage compartment 22 maintains a lower temperature than the exterior of the storage compartment 22 and does not immediately change to the same temperature as the outside air, so that the outside air enters the storage compartment 22 for heat exchange. This is done.

In other words, if the cover 90 is disposed in a position where it is relatively difficult to contact the outside air, the amount of dew generated in the cover 90 may be reduced, and the time point at which dew is generated may be delayed.

First, the outside air comes into contact with the front surface 501 of the protrusion 500 while entering the interior. At this time, the outside air in contact with the front surface 501 enters the chamber further as the path changes to the left and right of the protrusion. This outside air may reach the cover 90 while contacting the side surface 502 of the protrusion 500. Therefore, a large amount of moisture may be condensed before reaching the cover 90.

In addition, the outside air in contact with the front surface 501 of the protrusion 500 may descend and rise along the lower surface 503 of the protrusion. Accordingly, a large amount of moisture may be condensed while moving to the front surface 501 and the lower surface 503.

In addition, the cover 90 is mounted to be inclined in a form surrounded by the front surface 501, the side surface 502, and the lower surface 503. Accordingly, the path of the outside air and the cover 90 do not form vertical. Accordingly, it is also possible to prevent the external air from reaching the cover 90 and condensing rapidly.

In particular, the width of the protrusion 500, particularly the left and right widths of the front surface 5010 may be wider than the width of the camera 70.

The embodiment shown in FIG. 60 is very similar to the embodiment shown in FIG. 59. However, it can be said that the camera is mounted so as to face the vertical lower part. Therefore, the same effect as the example shown in FIG. 59 can be obtained.

In addition, the protrusion 500 may include a rear surface 504. It is preferable that the vertical length of the front surface 501 is greater than the vertical length of the rear surface 504. That is, it is possible that the front surface 501 protrudes further downward of the inner case 12.

Therefore, in order for the outside air to reach the cover 90, the path becomes longer due to the width of the protrusion 500 in the horizontal direction, and the path becomes longer due to the difference in the protruding height of the protrusion 500 in the vertical direction. You lose.

Condensation prevention in the cover 90 through the structure of the protrusion 500 or the structure in which the camera is mounted is very related to the installation position of the camera. That is, it is because the camera can be installed in an angle range inclined from the vertical to the rear by a predetermined angle from the vertical lower part of the ceiling of the storage room. Accordingly, it is possible to effectively bypass the inflow path of the outside air while effectively photographing the entire area within the storage room within the storage room.

In the prior art, there have been disclosed techniques for mounting a camera on a side wall of a main body storage compartment, an upper part outside the main body, a door or an upper part of the door.

In the prior art, when a camera is mounted on a side wall of a storage room, there is a technique of providing a depression for accommodating the camera on the side wall and mounting the camera on the depression. In other words, the camera does not protrude into the interior of the storage room. This is presumed to be a configuration in order not to cause inconvenience to the user when taking out shelves or items in the storage room.

However, in the present invention, the camera is installed on the ceiling of the inner case 12 so that the user is not disturbed when using the storage compartment 22. This is because there is no reason to collide with the camera because the camera is not located in the moving path of the item or the shelf when the user wants to withdraw the item from the storage room.

Further, in the present invention, the cover 90 of the camera 70 is arranged to face the rear, so that the path through which the outside air reaches the cover 90 can be extended.

In FIG. 59A, while the cover 90 is disposed to face the rear, the protrusion 500 protrudes downward, so that the outside air has a three-dimensional movement path to reach the cover 90, so that the outside air moves. The path becomes longer.

In FIG. 60A, the cover 90 is arranged to face downward, but the front surface 504 disposed horizontally at the opening in the protrusion 500, and the front surface 504 at both ends of the front surface 504 A side surface 508 that is disposed vertically is installed. At this time, the front surface 504 and the side surface 508 are installed so as to protrude further downward from the protrusion 500, so that a path through which outside air reaches the cover 90 may be lengthened.

That is, the high-temperature air introduced from the outside to the opening side of the storage compartment directly reaches the cover 90 of the camera and does not contact it, and arrives by detouring left and right and downward along the protrusion 500. Therefore, the outside air exchanges heat with the cold air in the chamber and the amount of saturated water vapor decreases as the temperature decreases, so that the water vapor contained is condensed and mixed with the cold air in the chamber and disappears. When it finally reaches the cover 90, since it has a reduced amount of water vapor, it can be sufficiently evaporated by the heater heating the cover 90.

Accordingly, the electric power to be provided to the heater 84 is reduced, and thus the power consumption of the refrigerator as a whole can be improved.

In the embodiment shown in Fig. 59 or 60, a main body (a cabinet) having a storage compartment formed therein; The storage compartment is a drawer having an access opening in the front, an upper wall, a lower wall, both side walls, and a rear wall made of an insulating material, a shelf area divided by a plurality of shelves, and at least one drawer Consists of areas; At least one of doors in close contact with the front of the storage compartment to open and close the storage compartment; A camera device installed on an upper wall of the storage compartment between the opening of the storage compartment and a front edge of the shelves installed in the storage compartment;

The camera device accommodates electrical components necessary for driving a camera including a camera lens, a camera module part having a camera window formed at a predetermined distance from the lens, and the camera module It is configured to include a camera housing part (a camera housing part) for mounting and fixing the part in a predetermined position inside;

The camera housing part includes a fixing surface in contact with the upper wall of the storage compartment, a front surface formed on the opening side of the storage compartment, side surfaces connecting the front and rear surfaces, and the camera It is configured to include a top surface in which an opening is formed so that the camera window of the module unit is exposed, and the front of the camera housing unit is provided with a refrigerator that is formed higher than the opening formed in the top surface of the camera housing unit. I can.

The top surface of the camera housing part may be inclined at a certain angle with respect to the upper wall surface of the storage compartment in a direction opposite to the opening of the storage compartment.

The camera housing unit may further form a rear surface at a position opposite to the front surface on a rear wall side of the storage chamber, and the height of the front surface may be formed higher than the height of the rear surface.

The opening of the upper surface of the camera housing may be formed at a position lower than the side of the camera housing.

It may further include a housing installation part having a recessed portion so as to correspond to a recessed space in the upper wall of the storage chamber.

The housing mounting part may be mounted on an upper wall of the storage compartment together with a refrigerator insulation material, and the camera housing part may be fixed to the housing mounting part.

The camera module part fixed to the camera housing part may be accommodated in a recessed part formed in the housing mounting part.

The camera module part accommodated in the recessed part of the housing mounting part may be fixed to the camera housing part so as to have a gap with the inner surface of the housing mounting part.

The front and both side surfaces of the camera housing part may be formed higher than the opening position of the upper surface of the camera housing.

The upper surface of the camera housing may be formed at an opening side of the storage compartment farther from an upper wall surface of the storage compartment than from a rear wall side of the storage compartment.

The camera module unit has a front portion on which a camera window is formed and an opposite rear portion, and at least one flat surface is formed on at least one of the front and rear portions. I can.

A receiving portion is formed inside the camera housing part, and a flat surface formed in the camera module part is placed in the receiving part at a certain angle with respect to the upper wall surface of the storage chamber. May contain a seating part.

The recessed portion of the housing mounting portion may include a seating part in which a flat surface formed on the camera module portion is placed at an angle with respect to the upper wall surface of the storage chamber.

A flat surface may be formed around the camera window of the front part of the camera module, and the flat surface may be formed to be substantially the same as the height of the center of the camera lens.

61 is a view showing another type of refrigerator to which the present invention is applicable. Hereinafter, it demonstrates with reference to FIG. 61.

The refrigerator of FIG. 61 is a double-door refrigerator, and the left and right sides form different storage compartments. That is, the storage compartment on the left is a freezing compartment, the storage compartment on the right is a freezing compartment, and the storage compartment on the left may constitute a refrigeration compartment.

A camera 70 may be installed on the ceiling of the storage compartment 22 to face the lower portion of the storage compartment 22.

In addition, a drawer 50 may be installed in the storage compartment 22 to store articles and to be able to withdraw and unload.

Other details are omitted because they are the same as the refrigerator described above.

62 is a screen provided to a user from the refrigerator according to FIG. 61. Hereinafter, it demonstrates with reference to FIG. 62.

The user may be provided with an image of an article stored in the drawers 50 and 51 and an image of an article stored on the upper side of the shelf 40. One drawer 50 may be provided in the refrigerator. Accordingly, an image of one drawer may be provided in FIG. 62 as well. However, the two drawers 50 and 51 may be provided vertically in the Bawak kx shown in FIG. 61. In this case, it is also possible to provide an image of the interior of each drawer.

In the example described with reference to FIG. 2, drawers are provided on the left and right sides, respectively. However, in FIG. 62, drawers may be provided in a vertical direction. Accordingly, there may be two drawer regions, and the lower drawer 50 may protrude forward compared to the upper drawer 51. In this case, a marker may be provided in each of the drawers 50 and 51 as well.

Through this, it is possible to easily grasp food information stored in a plurality of shelf areas and a plurality of drawer areas through a single camera.

A method of upgrading an image provided on a user screen is omitted because it is the same as that of the above-described refrigerator.

63 is a view for explaining a method of adjusting a picture taken by a camera in the refrigerator according to FIG. 61. Hereinafter, it demonstrates with reference to FIG. 63.

In general, since only one drawer is provided or arranged vertically, one horizontal adjustment line 15 may be provided in the refrigerator of FIG. 61. Of course, two horizontal adjustment lines may be provided to separate the two drawers, respectively. In addition, it is possible to adjust the position of the adjustment line by arranging two vertical adjustment lines arranged in parallel. Of course, unlike that shown in FIG. 63, it is possible to select a desired image using only the horizontal adjustment line without using the vertical adjustment line.

The user may be provided with a limit line 16 indicating a limit range in which the adjustment line 15 is moved. In this case, the limit line 16 may be arranged in parallel with a predetermined interval.

The correction of the error through the adjustment line 15 can be said to be the same as or similar to the above-described embodiment.

The present invention is not limited to the above-described embodiments, and as can be seen from the appended claims, modifications may be made by those of ordinary skill in the field to which the present invention belongs, and such modifications are within the scope of the present invention.

10: outer case 12: inner case
20: door 40: shelf
42: first area 50: drawer
52: second area 60: barrier
62: third area 70: camera
100: control unit

Claims (39)

A main body having an opening in front and a heat insulating wall forming a storage space therein;
A door rotatably provided with respect to the main body to open and close the opening so that the storage space forms a storage compartment capable of storing food at or below room temperature;
A drawer provided in the storage room;
A marker provided on the drawer;
At least one shelf provided in the storage compartment and positioned above the drawer;
A camera disposed outside the drawer to photograph the drawer; And
When the drawer is pulled out from under the shelf to the front of the shelf, the camera continuously photographs the drawer. The method of claim 1,
And a controller configured to determine a position or a moving state of a drawer through a marker included in an image of a photograph or video obtained by continuous shooting by the camera. The method of claim 2,
The control unit divides an image of a photo or video sent from the camera into predetermined pixels, assigns coordinate values at regular intervals, and determines the location of the marker on the coordinates of the image. The method of claim 3,
When the control unit recognizes that the coordinate value of the marker increases from 0, the control unit determines only a change in coordinates for a picture or image captured by the camera and sent to the control unit. The method of claim 4,
A refrigerator for partitioning and deleting an image of a portion other than a moving path of a marker for determining a change in coordinates. The method of claim 3,
When the control unit recognizes that the coordinate value of the marker has not changed, the control unit moves and stores a portion corresponding to the draw area from the picture or image image captured by the camera and sent to the control unit. Refrigerator. The method of claim 3,
When the control unit recognizes that the coordinate value of the marker changes to a smaller value, the control unit selects the most recent image from temporarily buffered images in a temporary buffer, moves to the storage unit, and stores it, displays it on the display, or Refrigerator sent to the terminal. The method of claim 3,
The control unit determines that the drawer is not open when the position of the coordinate value of the marker is within the set first section, and when the drawer is in the second section beyond the set section, the drawer is opened. Refrigerator judged by condition. The method of claim 8,
A refrigerator in which the drawer is within 30% of the withdrawal distance of the drawer as a first section, and when the withdrawal distance is 30% or more, as a second section. The method of claim 8,
The control unit is configured to change a method of determining the position of the drawer with respect to a picture or image transmitted from the camera according to a state in which the drawer is not opened and a state in which the drawer is opened. The method of claim 10,
In the first section in which the drawer is determined to be in a closed state, the control unit determines only whether the position of the marker in the transmitted current image reaches a position corresponding to a distance corresponding to a set drawing distance. The method of claim 11,
The control unit partitions and deletes or temporarily buffers a portion of the image except for a position of a marker. The method of claim 10,
When the marker reaches a position of a specific pixel or higher in the second section in the image processed by the controller, two pictures (the image of the picture taken immediately before and the picture taken after that) are taken to determine the stop state of the drawer. Image) of the refrigerator to determine the coordinate values of the markers. The method of claim 13,
The control unit divides a drawer portion excluding a position of a marker in the image, moves to a temporary storage unit, and stores it. The method of claim 13.
The control unit compares the coordinate values of the markers shown in two consecutive images in the second section, and determines whether the drawer is in a moving direction, the drawer is stopped, or the drawer is in a completely open state. Refrigerator. The method of claim 8,
A refrigerator configured to match the speed at which the camera captures the interior of the container (a time interval at which a photo is taken) with the speed at which the control unit determines and processes a transmitted photo or image from an image. The method of claim 3,
When the marker is not detected from the image, the controller determines that the drawer is closed as a coordinate value of 0. The method of claim 16,
The controller determines that the drool moves when the coordinate value of the marker changes from 0. The method of claim 3,
Further comprising a door switch for sensing the opening and closing of the door,
When only the door is open, the controller moves and stores the image corresponding to the shelf area to a temporary storage unit. The method of claim 18,
When the door is closed, the controller selects the most recent image from the images stored in the temporary storage unit and stores it in the storage unit, displays it on a display, or transmits it to a terminal. The method of claim 3,
Further comprising a door switch for sensing the opening and closing of the door,
When the door is open and the drawer is detected to be open, the control unit moves and stores an image corresponding to the drawer part from the picture or image transmitted from the camera to a temporary storage unit. The method of claim 21,
The controller selects the most recent image from the images stored in the temporary storage unit and stores it in the storage unit, displays it on a display, or transmits it to a terminal when it is determined that the droopy has started to close, The method of claim 21,
The control unit also separates the image portion corresponding to the shelf area and moves to the temporary storage unit for storage. The method of claim 23,
When the door is closed, the controller selects the most recent image from the images stored in the temporary storage unit and stores it in the storage unit, displays it on a display, or transmits it to a terminal. The method of claim 1,
Refrigerators each including a marker in the image of the photograph or image continuously taken by the camera. The method of claim 25,
Refrigerator comprising a storage unit for storing the image obtained by the continuous photographing by the camera. The method of claim 25,
The refrigerator including images having different marker positions in the image. The method of claim 1,
The camera is a refrigerator disposed above the drawer so as to photograph a plane of the drawer when the drawer is withdrawn. The method of claim 25,
A refrigerator including the inside of the drawer and the marker while the drawer is withdrawn, the image of the picture or image taken by the camera. The method of claim 1,
Markers are refrigerators in which shapes of the same shape or color are repeated at regular intervals. The method of claim 1,
The marker is a refrigerator having a color or shape that contrasts with the drawer around the drawer and the border part. The method of claim 31,
The inner portion of the marker has a color or shape that contrasts with the outer periphery of the marker. The method of claim 1,
The marker is a refrigerator in which a length in a moving direction of the drawer is smaller than a length in a direction perpendicular to a length in a moving direction of the drawer. The method of claim 1,
Further comprising a display for displaying an image of a picture or video in the storage room taken by the camera,
The display separates and displays the image for each storage area. The method of claim 34,
A refrigerator including an image of an inner area of the drawer in the image displayed by the display. The method of claim 35,
A refrigerator in which the image of the inner area of the drawer is an image of a photograph or image taken when a closing motion of the drawer is detected. The method of claim 36,
A refrigerator that displays a user input unit on the display to allow a user to check the picture or image when the drawer closed state is sensed and a picture or image for a storage area inside the drawer is secured. The method of claim 1,
A refrigerator configured to obtain an image of the drawer by continuously photographing the camera even in a state in which the drawer is stopped or completely withdrawn. The method of claim 38,
The drawer withdrawal length in a state in which the drawer is completely withdrawn is 50% of the length before and after the drawer.

Images