KR102512750B1 - Refrigerator and Controlling Method for the same - Google Patents

Abstract

The present invention relates to a refrigerator and a control method thereof, and more particularly, to a refrigerator and a control method thereof capable of providing information about food stored in the refrigerator without a user opening the door.
According to one embodiment of the present invention, a storage compartment formed by a heat insulation wall fixed to a main body of a refrigerator and having an opening; a door rotatably provided on the main body to open and close the opening; a drawer provided in the storage compartment; The area (region 1) in which food is stored in the outer space of the drawer in the storage compartment and the area (region 2) in which food is stored in the inner space of the drawer in the storage chamber are photographed together from the ceiling of the storage compartment. a camera provided to be fixed to; In addition, through the time of taking the picture in which the first area and the second area are taken together, the part for the first area (picture of the first area) and the part for the second area (picture of the second area) in the picture A refrigerator characterized in that it includes a control unit for separating, individually classifying and storing the refrigerator may be provided.

Description

Refrigerator and its control method {Refrigerator and Controlling Method for the same}

The present invention relates to a refrigerator and a control method thereof, and more particularly, to a refrigerator and a control method thereof capable of providing information about food stored in the refrigerator without a user opening the door.

In general, a refrigerator is a device for supplying cold air generated according to driving of a refrigerating cycle, and is a device for storing food at a low temperature. In the case of a conventional refrigerator, it can perform only a function of simply storing food in a low temperature state. However, in recent years, the need for additional functions in addition to the food storage function has increased.

A refrigerator is a device for storing and storing certain items, and in order to check the inside, there is no choice but to open the refrigerator door and check. In addition, when a user tries to buy a product at a market or a mart, if the user does not know the amount and type of food stored in the refrigerator, there is inconvenience in purchasing the same food repeatedly or not purchasing necessary food.

According to Japanese Patent Publication No. 3450907 and Japanese Unexamined Patent Publication No. 2004-183987, which are prior art, a camera is installed on the door to photograph the inside of the refrigerator. In addition, according to Japanese Patent Laid-open Publication No. 2001-294308, a technology is disclosed in which cameras are respectively installed in a refrigerator, a drawer, and a door.

However, according to the prior art reviewed, there is a problem in that a plurality of cameras must be used to photograph various storage areas because the photographing range of the camera installed in the refrigerator is limited.

Therefore, while reducing the number of cameras installed in the building, it is necessary to improve the shooting efficiency of the camera by increasing the range of the area captured by the camera.

In addition, according to the prior art, there were various problems in obtaining an image of the inside of the drawer. For example, a specific method for acquiring a photograph taken at a desired time point, a concern about dew condensation occurring in a camera, and a concern for improving power consumption by the camera have not been made in detail.

Specifically, according to the prior art, when there are a plurality of storage areas including drawers in the refrigerator, there is a problem in that specificity for providing optimal information to the user is lacking in consideration of the characteristics and positional relationship of each area.

The present invention basically aims to solve the above problems.

In order to solve the above problems through an embodiment of the present invention, the present invention provides a storage compartment display device of a refrigerator that provides information about food stored in the refrigerator even if the user does not open the refrigerator door and a control method thereof. is to provide

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof in which a user can intuitively recognize information about food stored in each of a plurality of storage areas in a storage compartment.

In addition, according to an embodiment of the present invention, a refrigerator capable of intuitively recognizing a positional relationship between a plurality of storage areas and providing stored food information substantially consistent with a user's visual point of view, and a refrigerator thereof. We want to provide a control method.

In addition, through an embodiment of the present invention, the driving time of the camera and the shooting time of the camera are interlocked with the opening of the door and / or the rotation angle of the door to reduce power consumption due to the camera and food through the camera at the optimal time. It is intended to provide a refrigerator capable of obtaining storage information and a control method thereof.

In addition, through an embodiment of the present invention, it is intended to prevent deterioration of the quality of pictures taken by a camera due to dew condensation.

In addition, through an embodiment of the present invention, it is possible to provide the user with up-to-date information on food stored in the refrigerator.

In addition, through an embodiment of the present invention, it is intended to provide the user with food storage information in an optimal state by allowing the manager or the user to directly correct the area of the picture displayed to the user.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof capable of providing a user with optimal food storage information by installing a camera on a fixed main body to prevent shaking of the camera.

In addition, according to an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof capable of effectively grasping and providing food storage information in a drawer while simplifying a configuration by grasping state information of a drawer in a software manner.

In addition, through an embodiment of the present invention, a refrigerator capable of providing the latest food storage information for each of a plurality of storage areas inside a storage compartment including a drawer by continuous shooting through a single camera and a control method thereof are provided. want to do

In addition, through an embodiment of the present invention, food stored in another storage area vertically overlapping at least a part of the inner space of the drawer or another drawer inner space overlapping at least a part of the inner space of the drawer vertically. It is intended to provide a refrigerator capable of providing stored information using a single camera and a control method thereof.

In addition, through an embodiment of the present invention, it is possible to prevent deterioration of the quality of a picture taken by a camera due to dew condensation.

In addition, through an embodiment of the present invention, it is possible to provide a refrigerator capable of reducing the cost required for a refrigerator by limiting the number of cameras installed in the refrigerator to one.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof capable of effectively preventing an increase in power consumption due to a camera.

In addition, through an embodiment of the present invention, it is intended to provide a refrigerator and a control method thereof capable of minimizing the load on a controller and a memory unit and effectively and efficiently processing pictures taken continuously.

In order to achieve the above object, according to one embodiment of the present invention, a storage compartment formed by a heat insulating wall fixed to a main body of a refrigerator and having an opening; a door rotatably provided on the main body to open and close the opening; a drawer provided in the storage compartment; The area (region 1) in which food is stored in the outer space of the drawer in the storage compartment and the area (region 2) in which food is stored in the inner space of the drawer in the storage chamber are photographed together from the ceiling of the storage compartment. a camera provided to be fixed to; In addition, through the time of taking the picture in which the first area and the second area are taken together, the part for the first area (picture of the first area) and the part for the second area (picture of the second area) in the picture A refrigerator characterized in that it includes a control unit for separating, individually classifying and storing the refrigerator may be provided.

Preferably, the control unit determines a photo that satisfies update conditions for the first region photo and the second region photo among the captured photos, and the update condition is different according to a capturing time point.

Preferably, the control unit individually and independently updates the first area picture and the second area picture.

Preferably, the control unit determines a picture taken at a specific point in time when the door is opened and then closed, and extracts and stores an image for updating the first area.

Preferably, the control unit determines a photo taken at a specific point in time when the drawer is drawn in after being drawn out, and extracts and stores an image for updating the second area.

Preferably, the control unit separates the first area picture and the second area picture from each of the plurality of pictures taken continuously, and separately classifies and stores the picture.

The first area is an area where food is placed by at least one shelf, and the first area may be located above the second area.

A display may be provided in front of the door to display the first area picture and the second area picture separately from each other and independently update the first area picture and the second area picture.

In the storage compartment, a third area provided under the drawer so as to overlap vertically with the drawer in a state in which the drawer is drawn out, and separated from the first area and the second area to store food may be provided. can

When the drawer is drawn out, the first area and the second area may be photographed together, and when the drawer is drawn in, the first area and the third area may be photographed together.

Preferably, the control unit separates the first region photo and the third region (third region photo) from the photo, and separately classifies and stores the photo.

Preferably, the control unit individually updates the first area photo, the second area photo, and the third area photo.

In order to achieve the above object, according to one embodiment of the present invention, a storage compartment formed by a heat insulating wall fixed to a main body of a refrigerator and having an opening; a door rotatably provided on the main body to open and close the opening; a drawer provided in the storage compartment; The area (region 1) in which food is stored in the outer space of the drawer in the storage compartment and the area (region 2) in which food is stored in the inner space of the drawer in the storage chamber are photographed together from the ceiling of the storage compartment. a camera provided to be fixed to; Separating the part for the first area (picture of the first area) and the part for the second area (picture of the second area) in the picture through the timing of taking the picture in which the first area and the second area were taken together. So, a control unit that individually classifies and stores; Also, a refrigerator may be provided that includes a display displaying a picture of the first area and a picture of the second area to be distinguished from each other.

Preferably, the display displays a first area frame assigned to the first area picture and displayed and a second area frame to which the second area picture is assigned and displayed so as to be divided from each other.

The first area may be an area where food is stored by a shelf in an upper space outside the drawer.

Preferably, at least one of the first area picture and the second area picture stored in the controller is displayed after being corrected in size to match the size of the first area frame and the second area frame.

In the storage compartment, a third area provided under the drawer so as to overlap vertically with the drawer in a state in which the drawer is withdrawn and separated from the first area and the second area to store food is provided this is preferable

The control unit separates the first region photo and the part (third region photo) from the photo through the photographing time point of the photo at which the first region and the third region are taken together, and separately can be identified and stored.

It is preferable that frames of the third region, to which the pictures of the third region are allocated and displayed, are separately displayed below the frames of the first region on the display.

It is preferable that the left and right widths of the first area frame and the third area frame are the same, and the third area picture is displayed after being corrected to match the left and right widths of the third area frame.

The drawer includes two drawers provided to be symmetrical on the left and right, and the photos of the second area can be divided into left and right, individually classified and stored.

Preferably, the display displays an image captured by the camera and an adjustment line for adjusting a position of a dividing line for separating the image into a plurality of areas.

Preferably, the adjustment line includes at least one horizontal adjustment line and at least one vertical adjustment line, and the adjustment line is movable through a user's manipulation of the display.

In order to achieve the above object, according to one embodiment of the present invention, a storage compartment formed by a heat insulating wall fixed to a main body of a refrigerator and having an opening; a door rotatably provided on the main body to open and close the opening; a drawer provided in the storage compartment; The area (region 1) in which food is stored in the outer space of the drawer in the storage compartment and the area (region 2) in which food is stored in the inner space of the drawer in the storage chamber are photographed together from the ceiling of the storage compartment. a camera provided to be fixed to; In addition, a part (first area photo) of the first area is separated from a photo taken at a specific time point when the door is opened and closed, and the second area is taken from a photo taken at a specific time point when the drawer is pulled out and retracted. A refrigerator characterized in that it includes a control unit that separates, classifies, stores, and updates the first area picture and the second area picture separately from each other by separating the part (second area picture) for .

The door includes a left door and a right door, the drawers are provided on left and right sides of the storage compartment, and the control unit divides a portion for the left drawer and a portion for the right drawer in the picture of the second area to each other. It is desirable to separate, classify, store and update individually.

A door sensor for detecting an opening angle of each of the left door and the right door may be provided.

It is preferable that the door sensor is provided in a connecting portion of a hinge portion that connects between an upper surface of the main body and an upper surface of the door to enable rotation of the door relative to the main body.

In order to achieve the above object, according to one embodiment of the present invention, a storage compartment formed by a heat insulating wall fixed to a main body of a refrigerator and having an opening; a door rotatably provided on the main body to open and close the opening; a camera fixed to the storage compartment and photographing the inside of the storage compartment in order to update food information in the storage compartment; a door switch that senses opening and closing of the door; a door sensor that detects an opening angle of the door; And, according to the detection result of the door switch, a transition point between the standby mode of the camera and the driving mode for starting shooting is determined, and according to the detection result of the door sensor, a valid picture for update among the captured pictures is determined. A refrigerator including a controller for determining a reference time point for selecting may be provided.

In order to implement the above object, according to an embodiment of the present invention, a storage compartment; a drawer movably provided in the storage compartment and equipped with a marker; a camera photographing the drawer from the outside of the drawer; In addition, by detecting the position of the marker in pictures continuously taken by the camera, state information of the drawer including at least one of the degree of withdrawal, whether or not the drawer has been withdrawn, the movement direction, the stationary state, or the movement state of the drawer is determined. A refrigerator characterized in that it includes a; may be provided. Since the marker moves integrally with the drawer, state information of the drawer can be grasped through the position of the marker.

In order to implement the above object, according to an embodiment of the present invention, a storage compartment; a drawer movably provided under the storage compartment and equipped with a marker; a camera fixed to the ceiling of the storage compartment and photographing the inside of the drawer from the outside of the drawer; In addition, by detecting the position of the marker in pictures continuously taken by the camera, state information of the drawer including at least one of the degree of withdrawal, whether or not the drawer has been withdrawn, the movement direction, the stationary state, or the movement state of the drawer is determined. A refrigerator characterized in that it includes a; may be provided.

In order to achieve the above object, according to one embodiment of the present invention, a storage compartment formed by a heat insulating wall fixed to a main body of a refrigerator and having an opening; a door rotatably provided on the main body to open and close the opening; a drawer provided in the storage compartment; at least one shelf provided in the storage compartment and positioned above the drawer; And the area where food is stored by the shelf in the outer upper space of the drawer (first area) and the area where food is stored in the inner space of the drawer (second area) are photographed together in the storage compartment, A refrigerator including a camera provided to be fixed to the ceiling of the storage compartment between the front end of the shelf and the opening may be provided.

As long as the above-described embodiments of the present invention are not contradictory or exclusive to each other, each feature may be implemented in combination with each other.

According to the present invention, since the user does not need to open the door to obtain information about the food stored in the refrigerator, leakage of cold air contained in the storage compartment can be prevented. Therefore, unnecessary loss of cold air can be prevented and energy efficiency of the refrigerator can be improved.

In addition, since the present invention can provide the user with up-to-date information on food stored in the refrigerator, reliability of the food information provided to the user can be improved.

In addition, in the present invention, since information on food stored in various locations can be provided by a single camera, only a structure for installing a single camera is added, and thus the design can be made easily. In particular, the cost incurred due to the use of the camera can be reduced.

According to the present invention, since condensation is prevented on the camera installed in the refrigerator, pictures taken by the camera can be stably provided to the user.

According to the present invention, an individual can remotely grasp the current internal situation of the refrigerator and receive necessary food-related information from an external operator.

Further, according to the present invention, an image captured by a camera of the inside of the drawer may provide a screen similar to a user actually seeing when using the refrigerator.

Also, according to the present invention, locations that cannot be seen at once because they overlap in terms of spatial arrangement can be provided to the user as flat images on one screen.

1 is a front view of a refrigerator according to the present invention;
FIG. 2 is a view in which a door is opened in FIG. 1;
3 is a view showing a third area disposed on the floor of the storage compartment;
4 is a control block diagram of an embodiment according to the present invention;
5 is a diagram illustrating an angle of view of a camera;
Figure 6 is a cross-sectional side view of Figure 5;
7 is a screen captured by a camera in the state of FIG. 6;
8 is a view for explaining position selection of a camera;
Fig. 9 is a cross-sectional view showing a main part of the refrigerator;
10 is a diagram explaining an operation method of a door sensor;
11 is a view showing a left hinge part and a right hinge part in detail;
12 is a screen provided to a user;
13 is a diagram explaining a method of adjusting a picture taken by a camera;
14 is a perspective view of a camera;
Fig. 15 is a diagram showing a main part of a camera;
Fig. 16 is a sectional view of Fig. 14;
17 is a view explaining the arrangement of heaters;
18 is a view for explaining pictures taken in a state in which a camera is installed in a refrigerator in an up-and-down direction;
19 is a diagram for explaining a picture taken in a state in which a camera is turned horizontally on a refrigerator;
20 is a view showing an assembled state of the camera housing;
21 is a front view of the first housing;
Fig. 22 is a front view of the second housing;
23 is a front view illustrating a state in which a camera is installed in a first housing;
Fig. 24 is a cross-sectional side view of Fig. 23;
25 is a cross-sectional view of a state in which a camera housing is installed in an inner case;
26 is a table comparing power consumption of cameras in the present invention.
27 is a diagram comparing supplied standby current and driving current;
28 and 29 are diagrams for explaining the camera's picture-taking start point and the camera's continuous picture-taking;
30 is a diagram illustrating an embodiment of a drawer sensing unit;
31 is a diagram explaining a method of detecting movement of a drawer in the drawer detector of FIG. 30;
Fig. 32 is a diagram showing a state where a marker is displayed on the drawer;
33(a) is a view showing a photograph taken in a state in which the left and right drawers are withdrawn;
33(b) is a view showing a state in which the left drawer is retracted and the right drawer is withdrawn;
34 is a flowchart for explaining a refrigerator operating method according to an embodiment of the present invention;
35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.
36 is a diagram for explaining a marker for detecting a draw-in/out time point of a drawer according to an embodiment of the present invention;
37 is a view for explaining the shape of a marker according to an embodiment of the present invention;
38 is a flowchart for explaining a method of detecting a drawer closing operation according to an embodiment of the present invention;
39 is a flowchart for explaining an operation of a refrigerator performed upon detection of completion of a drawer closing operation according to an embodiment of the present invention;
40 is a flowchart illustrating a method of controlling an image of a specific region inside a refrigerator captured at a specific point in time according to an embodiment of the present invention;
41 illustrates various types of markers;
42 is a view for explaining a marker position recognition method;
43 is a view showing the extent to which a drawer is opened;
Fig. 44 is a diagram explaining movement of markers;
45 is a control flow diagram according to an embodiment of the present invention;
Fig. 46 is a control flowchart for explaining a modified example of Fig. 45;
Fig. 47 is a control flowchart for explaining another modified example of Fig. 46;
Fig. 48 is a control flowchart for explaining another modified example of Fig. 45;
Fig. 49 is a modified example of Fig. 45 differently;
50 is a control flow chart according to another embodiment of the present invention;
51 is a view explaining a process of updating an image in a state in which two doors and two drawers are provided to open and close the storage compartment;
52 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
53 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
54 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
55 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.
Fig. 56 is a view explaining the operation of the heater of the camera;
57 is a diagram explaining experimental results on condensation generated by temperature on a transparent window of a camera.
58 is a cross-sectional view of a transparent window;
59 and 60 are diagrams schematically illustrating a structure in which a camera is installed in an inner case.
61 is a view showing another type of refrigerator to which the present invention can be applied;
62 is a screen provided to a user from the refrigerator according to FIG. 61;
63 is a diagram explaining a method of adjusting a picture taken by a camera in the refrigerator according to FIG. 61;

Hereinafter, a preferred embodiment of the present invention that can specifically realize the above object will be described with reference to the accompanying drawings.

1 is a front view of a refrigerator according to the present invention, and FIG. 2 is a view of FIG. 1 with a door opened. It will be described with reference to FIGS. 1 and 2 below.

Refrigerators according to the present invention include a top-mounted type in which a freezing chamber and a refrigerating chamber, which are storage chambers for storing food, are partitioned up and down and the freezing chamber is disposed above the refrigerating chamber; The same can be applied to side-by-side-type refrigerators.

However, in this embodiment, for convenience of description, the bottom freezer type in which the freezing compartment and the refrigerating compartment are divided into upper and lower compartments and the freezing compartment is disposed below the refrigerating compartment will be mainly described.

The body of the refrigerator includes an outer case 10 forming an overall appearance when viewed by a user from the outside and an inner case 12 forming a storage compartment 22 in which food is stored. A predetermined space may be formed between the outer case 10 and the inner case 12 to form a passage through which cold air circulates. Meanwhile, an insulating material is filled between the outer case 10 and the inner case 12 so that the inside of the storage compartment 22 can maintain a relatively low temperature compared to the outside.

In addition, a refrigerant cycle device for generating cold air by circulating a refrigerant is installed in a machine room (not shown) formed in a space between the outer case 10 and the inner case 12 . The freshness of stored foods can be maintained by maintaining the inside of the refrigerator at a low temperature using the refrigerant cycle device. The refrigerant cycle device includes a compressor that compresses the refrigerant, an evaporator (not shown) that phase-converts the liquid refrigerant into a gaseous state, and exchanges heat with the outside.

The refrigerator is provided with doors 20 and 30 that open and close the storage compartment. At this time, the doors may include the freezing compartment door 30 and the refrigerating compartment door 20, respectively, and one end of each door is rotatably installed in the main body of the refrigerator by a hinge. The freezing compartment door 30 and the refrigerating compartment door 20 may be formed in plurality. That is, as shown in FIG. 2 , the refrigerating compartment door 20 and the freezing compartment door 30 may be installed in a form in which both corners of the refrigerator are opened toward the front.

A foaming agent may be filled between the outer case 10 and the inner case 12 so that insulation between the outside and the storage compartment 22 may be insulated.

The storage compartment 22 forms a space insulated from the outside by the inner case 12 and the door 20 . When the door 20 seals the storage compartment 22, the storage compartment 22 can form a space that is isolated from the outside and insulated. In other words, the storage compartment 22 is formed by the door 20. It can be said to be a space isolated from the outside through the insulating wall and the insulating wall by the cases 10 and 12 .

Cool air supplied from the machine room can flow throughout the storage compartment 22, so that food stored in the storage compartment 22 can be maintained at a low temperature.

The inner case 12 may form a barrier 60 on the bottom of the storage compartment 22 . A barrier 60 is installed at the lower end of the storage compartment 22 to distinguish the freezing compartment from the refrigerating compartment. The barrier 60 may have a predetermined thickness and be formed on the inner case 12 . The barrier 60 may extend in a horizontal direction.

The storage compartment 22 may include a shelf 40 on which food is placed. At this time, a plurality of shelves 40 are provided, and food can be placed on each shelf 40 . The shelf 40 may partition the inside of the storage compartment in a horizontal direction.

A drawer 50 capable of being pulled in or out is installed in the storage compartment 22 . In the drawer 50, food and the like are accommodated and stored. Two drawers 40 may be disposed left and right in the storage compartment 22 . A user may open the left door of the storage compartment 22 to access the drawer disposed on the left side. On the other hand, the user may open the right door of the storage compartment 22 to access the drawer disposed on the right side.

A space for storing food may be formed in the barrier 60 . Since the barrier 60 is not provided on the door but is provided on the inner case 12, it does not move according to the rotation of the door and remains fixed. Therefore, the user can stably store or withdraw food.

The storage compartment 22 is divided into a space located above the shelf 40, a space formed by the drawer 50, a space formed in the barrier 60, and the like, so that food is stored. It can be divided into a plurality of these.

At this time, each space is a storage space disposed inside one storage compartment 22, and cold air supplied to the storage compartment 22 can be moved to each space. That is, since each space is partitioned to the extent that cold air can move to each other, it has a different meaning from the storage room described above.

In particular, unlike a storage compartment in which each space is an insulated space, a temperature difference may exist within one storage compartment, but it does not form a space insulated from each other.

Cold air supplied to one storage compartment does not freely move to another storage compartment, but cold air supplied to one storage compartment can freely move to each partitioned space installed inside one storage compartment. That is, the cold air positioned above the shelf 40 can move to the space formed by the drawer 50 .

In addition, the present invention may include one camera 70 for taking pictures of the inside of the storage compartment 22 . At this time, the camera 70 is provided to maintain a fixed position, so that it is possible to photograph the same part.

In particular, the camera 70 is installed on the upper side wall of the inner case 12 facing downward, so that a picture of the food stored in the storage compartment 22 can be taken. The captured picture may have a form in which the user looks down, that is, a form in which the user looks when actually using the refrigerator.

In particular, the camera 70 is installed at a position corresponding to the inside of the drawer when the drawer is in a fully opened state, and a photographed picture may provide a screen similar to a user viewing the inside of the drawer from above.

At this time, the storage compartment 22 may include a first area 42 where food is placed on the shelf 40 and a second area 52 defined by an inner space of the drawer 50 . In addition, a third area 62 overlapping the second area 52 when the drawer 50 is withdrawn, which is a storage space different from the first area and the second area, may be included. This part will be described in more detail later.

The first area 42 and the second area 52 may have different heights within the storage compartment 22 as a whole. That is, the first area 42 and the second area 52 may be lowered in order.

Photos taken by the camera 70 or various information about the refrigerator may be provided to the user through the display 14 provided on the front of the refrigerator. Also, the user can control the refrigerator through the display 14 .

The cold air supplied to the storage chamber 22 can be moved to the first area 42 and the second area 52, and the cold air located in each area can be moved to other areas as well.

3 is a view showing a third area disposed on the floor of the storage compartment. It will be described with reference to FIG. 3 below.

A recessed portion is provided in the barrier 60 so that food can be stored in the corresponding portion, and this area may be referred to as the third area 62 . The third area 62 is a space formed in the barrier 60 .

In this case, the third area is a storage space disposed at a location different from that of the first area and the second area, and may provide another space for storing food to the user.

In the third region 62, frequently used foods such as eggs can be stored. For this purpose, it is possible to provide a cover 68 for opening and closing the third region 62. At this time, it is preferable that the cover 68 is made of a transparent material so that the food stored in the third area 62 can be photographed even if the cover 68 is covered when the camera 70 takes a photograph from the upper side. .

Meanwhile, the drawer 50 is disposed above the barrier 60 . Therefore, when the drawer 50 is drawn into the lower space of the shelf, the user can access the third area 62 .

On the other hand, when the drawer 50 is withdrawn from the lower space of the shelf, the drawer 50 is disposed above the third area 62, and the user approaches the third area 62. Can not.

That is, when the drawer 50 is withdrawn toward the user, the drawer 50 and the third area 62 overlap each other. At this time, when the user looks down at the drawer 50, the third area 62 is not visible to the user, and the inside of the drawer 50 can be seen.

On the other hand, when the drawer 50 is not withdrawn toward the user, the drawer 50 and the third area 62 are disposed so as not to overlap each other. When the user looks down at the third area 62, the inside of the drawer 50 is not visible to the user, and only the third area 62 may be shown.

The first region 42 , the second region 52 , and the third region 62 may have different heights within the storage compartment 22 as a whole. That is, the first area 42, the second area 52, and the third area 62 may be lowered in order.

The cold air supplied to the storage chamber 22 can be moved to the first area 42, the second area 52, and the third area 62, and the cold air located in each area is transferred to other areas. can also be moved.

That is, the area photographed by the camera 70 may be a plurality of storage spaces inside the storage compartment 22 . The camera 70 takes a picture of the inside of one storage compartment 22, but information on a plurality of partitioned storage spaces may be included in one picture.

At this time, the camera 70 may take a picture so that a plurality of storage spaces are included in one picture. In particular, it is possible for one picture to be taken in different storage spaces depending on the time it is taken.

For example, a camera disposed in a fixed form may take pictures of the first area and the second area or selectively select the second area and the third area according to the relationship between the time when the picture is taken and the position of the door or the drawer. It is possible to shoot with Details about this will be described later.

4A is a control block diagram of an embodiment according to the present invention. Hereinafter, it will be described with reference to FIG. 4A.

The present invention may include a controller 100 that divides and manages one picture taken by the camera 70 into a plurality of images. At this time, the image may mean a part of a photo processed or corrected by the control unit 100 .

The controller 100 may command the camera 70 to take a picture, and may receive a picture taken by the camera 70 .

In addition, the controller 100 may provide a portion of the divided image to the display 14 to provide the user with up-to-date information on the food stored in the storage compartment 22 . At this time, the display 14 may be installed on the front of the refrigerator, or may form a separate device separated from the refrigerator. That is, the user can receive an image of the storage room 22 and acquire information through an external communication terminal such as a mobile phone.

At this time, the control unit 100 may divide the picture taken by the camera 70 into each area, divide the picture into a plurality of independent images, and provide the divided images to the display 14 . At this time, the photo selected by the control unit 100 may include information about a recent storage room state in which the user last approached the storage room 22 and took out food from the storage room or stored food therein.

An embodiment of the present invention may include a door switch 110 capable of detecting whether the door 20 opens or closes the storage compartment 22 . At this time, the door switch 110 is provided in the outer case 10, and detects that the door 20 closes the storage compartment 22 when the door 20 contacts the outer case 10. can do. In addition, when the door 20 does not contact the outer case 10, it can be sensed that the door 20 opens the storage compartment 22.

In addition, the embodiment of the present invention may include a door sensor 120 for detecting a rotation angle of the door 20 . At this time, the door sensor 120 may detect the rotation direction of the door 20 and the rotation angle of the door 20 . For example, when the door 20 is rotated at a predetermined angle or more, a change occurs in the door sensor 120, and the door sensor 120 can detect that the door 20 is rotated at a predetermined angle or more. there is. In addition, when the door 20 is rotated in a specific direction, it is possible to detect the rotation direction of the door 20 by detecting a change in a pulse generated according to the direction. Of course, the door sensor 120 may be modified in various forms other than the above-described method.

In addition to this, the door sensor 120 may include a light emitting unit and a light receiving unit to detect the rotation angle of the door 20 by determining whether light emitted from the light emitting unit is transmitted to the light receiving unit.

In particular, the door sensor 120 can be driven only when the door switch 110 determines that the door 20 has opened the storage compartment 22 .

Meanwhile, when the door sensor 120 determines that the door 20 is rotated at a specific angle, the camera 70 may take a picture. In this case, even if the opening of the door 20 is detected by the door switch 110, the camera 70 does not immediately start taking a picture, but takes a picture when the door sensor 120 reaches a specific angle. will be filming

When the camera 70 starts taking pictures, it is also possible to continuously take pictures at predetermined time intervals. Of course, if the door sensor 120 determines that the door 20 is opened at a certain angle, the camera 70 can continue to take pictures until a command to stop taking pictures is provided, not just one picture. do.

The control unit 100 may include a drawer detecting unit 130 that detects whether the drawer 50 is pulled out or retracted. At this time, the drawer detecting unit 130 may refer to a part in which a picture taken by the camera 70 reads an image in the controller 100 . The drawer detecting unit 130 can detect movement of the drawer 50 as well as a movement direction of the drawer 50 .

In particular, the drawer detection unit 130 may be implemented by software. The drawer detecting unit 130 may detect the position of the drawer 50 using information captured from a picture taken by the camera 70 .

The drawer 50 is provided with an identification mark called a marker, and through a photograph of the marker, whether the drawer 50 entered a specific section, stopped if entered, and changed its moving direction after stopping. information about it can be found. The marker will be described in detail later.

Since the marker is marked on the drawer, it has the same movement as the drawer 50 . Therefore, various information about the movement of the drawer 50 can be obtained using the information of the picture taken by the camera 70 without having a separate sensor for detecting the movement of the drawer.

A picture taken by the camera 70 may be stored in the storage unit 18 . At this time, the storage unit 18 may be provided in the refrigerator or may be provided in another device separate from the refrigerator. At this time, the storage unit 18 may be disposed in the refrigerator together with the display 14, or may be provided together with the display 14 in a device other than the refrigerator, a server connected to the refrigerator and the network, or a terminal connected to the refrigerator and the network. there is.

At this time, the storage unit 18 may not store all pictures taken by the camera 70 . For example, the storage unit 18 may delete a previously taken and stored picture and store the currently taken picture when a command to select a specific picture is not generated from the control unit 100 (First In , First Out). The storage unit 18 stores only some of the photos taken by the camera 70, thereby reducing storage capacity.

4B is a control block diagram of a modified form of one embodiment according to the present invention. Hereinafter, it will be described with reference to FIG. 4B.

In FIG. 4B , unlike FIG. 4A , the drawer detecting unit 130 may separately detect the movement of the drawer 50 without using information obtained from a picture taken by the camera 70 .

For example, the drawer detector 130 may be formed in the form of a plurality of hall sensors. A plurality of hall sensors may be installed in the moving path of the drawer 50 . When the drawer 50 is moved, the position and movement direction of the drawer 50 may be sensed by detecting changes in each of a plurality of hall sensors.

The movement of the drawer is sensed by the drawer detector 130, so that when taking a picture with the camera 70, a point in time at which the picture taken by the camera 70 contains the inside of the drawer can be determined. . That is, the drawer detecting unit 130 may detect a time point at which information about items stored in the drawer may be expressed in a picture taken by the camera.

The drawer detecting unit 130 may detect movement of the drawer and detect a point in time when it is determined that the drawer is up-to-date after the user uses the drawer. That is, the drawer detecting unit 130 can detect the time when the picture taken by the camera includes the latest information that the user has finished using the drawer. That is, it is possible to indirectly grasp the completion time of goods storage and warehousing or the end time of product update.

5 is a diagram illustrating an angle of view of a camera. It will be described with reference to FIG. 5 below.

In FIG. 5 , for convenience of explanation, the drawer 50 is drawn out so that the camera 70 can photograph the second area 52 and the shelf is omitted.

The camera 70 may be disposed on an upper wall of the inner case 12 so as to be inclined toward a rear wall of the inner case 12 .

Meanwhile, a picture taken by the camera 70 may have a horizontal angle of view along the width direction of the storage compartment 22 and a vertical angle of view along the front-back direction.

At this time, the horizontal angle of view according to the camera 70 is provided so that the food stored in the inner space of the two drawers 50, that is, the second area 52, can be seen when the two drawers 50 are withdrawn. it is desirable

In addition, the range of the horizontal angle of view according to the camera 70 may include at least a portion of both left and right ends of the shelf 40 . This is because it is preferable to photograph both ends of the shelf 40 with the camera 70 in order to obtain information on food placed on the upper side of the shelf 40 .

As the angle of view of the camera 70 changes, the range of the inside of the storage compartment shown in the picture changes.

FIG. 6 is a cross-sectional side view of FIG. 5 , and FIG. 7 shows a screen captured by a camera in the state of FIG. 6 . It will be described with reference to FIGS. 6 to 7 below.

An opening 14 is formed in the front of the inner case 12 so that a user can access the storage compartment 22 . Through the opening 14 , a user may put food into the storage compartment 22 or take out food from the storage compartment 22 . The door 20 may open and close the opening 14 .

The camera 70 is installed outside the drawer, and takes pictures while the drawer is moving or stopped, thereby taking pictures of an open state, a closed state, and a movement state of the drawer. That is, when the camera 70 continuously takes pictures, a state in which the position of the drawer is changed according to a picture taking time may be expressed in the pictures taken by the camera 70 .

In addition, the camera 70 may provide the user with a screen including a recent storage/receiving/receiving state of items inside the storage compartment 22 .

The camera 70 is preferably installed within a range l between the opening 14 and the front end of the shelf 40 . When there are a plurality of shelves 40, they are preferably installed in a region (l, see FIG. 8) between the frontmost front end of the front ends and the opening 14. That is, it is preferable to be installed on the ceiling of the storage room corresponding to this area. Since the camera 70 needs to selectively take pictures of the first area and the second area 52 or the first area and the third area 62, it is better than the front end of the shelf 40. It cannot be moved to the rear wall of the inner case 12 .

6 shows the vertical angle of view of the camera 70, it is preferable that one end (front end) of the vertical angle of view is arranged so that a picture can be taken up to the front end of the drawer 50 in an open state. do. In addition, the other end (rear end) of the vertical angle of view is preferably arranged so as to be able to photograph up to the rear end of the uppermost shelf.

Since the range of the vertical angle of view of the camera 70 includes the second area 52 by the drawer 50, the camera 70 allows the user to obtain information about the second area 52. can be obtained.

As shown in FIG. 7 , the camera 70 can take pictures of the first area 42 and the second area 52 with one picture. At this time, the camera 70 may take pictures of the two drawers 50 disposed on the left and right sides. In addition, when a plurality of shelves are provided, it can be seen that the first region 42 and the second region 52 divided by the plurality of shelves can be photographed together.

The camera 70 may be installed inside the storage compartment 22 to take a picture including information about the inside of the storage compartment 22 . Since the camera 70 is installed inside the storage compartment, it is not moved. That is, the camera itself does not move because it is fixedly installed in the storage compartment of the main body, which is generally always fixed. Therefore, when a picture is taken by the camera 70, the camera 70 is always in a stopped state.

A picture taken by the camera 70 includes the first area 42 and the second area 52 or includes the first area 42 and the third area 62 .

When the camera 70 takes a picture including the first area 42 and the second area 52, the user can view the food stored on the upper side of the shelf 40 and the food through the corresponding picture. Information on food stored in the lower 50 may be obtained.

On the other hand, when the camera 70 takes a picture including the first area 42 and the third area 62, the user can view the food stored on the upper side of the shelf 40 and the food through the corresponding picture. Information on food stored in the barrier 60 may be obtained.

The camera 70 takes a picture of the inside of one storage compartment 22 and transmits the photographed picture, and the controller 100 can obtain information on a plurality of storage spaces.

The camera 70 is disposed above the first area 42, the second area 52 and the third area 62, and takes pictures while looking down.

8 is a diagram explaining position selection of a camera. It will be described with reference to FIG. 8 below.

As shown in FIG. 8, the camera can be mounted at positions L1, L2, L3, and L4.

L1 is an external space partitioned by the outer case 10 to the outside of the storage compartment 22 . When the camera is installed at the position of L1, the user uses the refrigerator while approaching the storage compartment 22 with the door open, but a problem may occur that the camera is covered by the user.

As disclosed in the above-disclosed Japanese patent, when the camera is installed outside the refrigerator to take a picture of the inside of the storage room, the image interference by the user is too great, and the movement of the drawer cannot be tracked because the drawer is covered. In the embodiment of the present invention, it is impossible to grasp both the internal image of the drawer and the movement state of the drawer. In addition, the location of L1 may mean that the camera is located on the door. In this case, since the door is configured to be moved, the camera may be moved during shooting. Therefore, it can be said that it is very difficult to obtain a stable and clear picture.

L2 is the inside of the storage compartment 22, and means an upper wall of the storage compartment 22, that is, a ceiling. In the embodiment of the present invention, it corresponds to the location where the camera is installed, but the problem of being covered by the user does not occur. In addition, since the upper wall of the storage compartment 22 corresponds to the inner case 12, it is convenient to arrange wires for supplying electricity to the camera. In addition, since the inner case 12 is generally always fixed, the camera can be stably mounted on the fixed structure.

L3 is the inside of the storage compartment 22 and is a side wall of the storage compartment 22 . At this time, the camera is installed so as to tilt from one side of the storage compartment 22 toward the other side of the storage compartment 22 to take a picture. If the camera is to be placed in the middle of the storage compartment 22 based on the height of the storage compartment 22 instead of the uppermost end, it must be installed on the sidewall of the inner case 12 .

When the camera is installed on one side wall, the camera lens cannot but be installed to face the other side wall, and thus, there is a problem in that an asymmetrical image of the inside of the storage compartment 22 is obtained due to a difference in left and right distances.

L4 is the inside of the storage compartment 22 and means one end of the shelf 40 . In order for the camera to be placed based on the height of the storage compartment 22, it must be supported by a specific structure. In this case, the camera may be fixed to one end of the shelf 40.

In this case, since the distance between the drawer 50 and the camera is shorter than that of L2, there is a problem in that a camera having a larger angle of view than that of L2 must be used. In addition, there is also a problem that the degree of distortion of the obtained photograph becomes severe. In addition, various external problems such as installing wires supplied to the camera on the shelf 40 may cause inconvenience to the user. Also, the shelf 40 is generally movable. Therefore, when food is placed on the shelf, a camera itself may shake.

The camera applied to the embodiment of the present invention has an angle of view of 120 degrees, is a VGA level, and can provide performance of 20 fps.

9 is a cross-sectional view showing a main part of the refrigerator. It will be described with reference to FIG. 9 below.

A basket 21 for storing food may be provided inside the door 20 . The basket 21 may be installed on a surface where the door 20 faces the storage compartment 22 .

There may be a problem that the basket 21 is shown in the picture taken by the camera 70 . Therefore, in the embodiment of the present invention, it is possible to detect whether the door 20 is rotated more than a certain angle θ through the door sensor 120 and take a picture through the camera 70 . Of course, it is also possible to transmit information about a corresponding point in time from the door sensor 120 to the control unit 100 so that a photo taken at a point in time adjacent to the point in time detected by the door sensor 120 is selected.

For example, it is possible that the predetermined angle (θ) is 60 degrees to 80 degrees. Of course, the angle may be variously changed according to the capacity of the refrigerator, the size of the door, or the front and rear length of the basket 21 .

At this time, the certain angle θ may mean an angle at which the basket 21 may not appear in the corresponding photo when the camera 70 takes a picture, and this certain angle θ is the angle of the basket 21 It can vary depending on the width of the left and right and the length of the front and back.

The door switch 110 for detecting whether the door 20 is opened or closed may be installed on the upper side of the refrigerator. At this time, when the door switch 110 is pressed by the door 20, it detects that the door 20 closes the storage compartment, and when it is not pressed by the door 20, the door 20 closes the storage compartment. You can sense that it is open.

A left hinge part 300 rotatably installs the left door 20 on the refrigerator, and a right hinge part 320 rotatably installs the right door 20 on the refrigerator. The left hinge part 300 and the right hinge part 320 are provided with a rotating shaft capable of rotating the door 20 and coupled to the door 20, and a part installed on the outer case 10, It can be divided into a connection part connecting the two parts. That is, it may be divided into door coupling parts 300c and 302c, outer case coupling parts 300a and 320a, and connection parts 300b and 320b connecting them.

A left door sensor 120L for detecting rotation of the left door 20 may be installed on the left hinge part 300 .

A right door sensor 120R for detecting rotation of the right door 20 may be installed on the right hinge part 320 .

The left door sensor 120L and the right door sensor 120R can independently detect the rotation angle of each door.

10 is a diagram explaining an operating method of a door sensor. It will be described with reference to FIG. 10 below.

The door sensor 120 may include a light emitting unit 122 that emits light and a light receiving unit 124 that receives the light emitted from the light emitting unit 122 .

Light emitted from the light emitting part 122 may be reflected through the upper surface of the door 20 and transmitted to the light receiving part 124 . In this case, it is also possible to be composed of a well-reflective material so that the reflection of light can be made stably.

When the light irradiated from the light emitting unit 122 is transmitted to the light receiving unit 124, it may be detected that the door 20 has been rotated by an angle smaller than the angle at which the door sensor 120 is installed. That is, it may mean that the door 20 is rotated at an angle smaller than a certain angle θ or that the storage compartment 22 is not rotated in a closed state.

On the other hand, if the light irradiated from the light emitting unit 122 is not transmitted to the light receiving unit 124, it may be detected that the door 20 has been rotated by an angle greater than the angle at which the door sensor 120 is installed. That is, it may mean a state in which the user can access the storage compartment 22 as the door 20 is rotated at a greater angle than a predetermined angle θ.

To this end, as shown in FIG. 9 , the left hinge part 300 and the right hinge part 320 may be mounted across the top surface of the outer case 10 and the top surface of the door 20 of the main body. Accordingly, as the door 20 rotates about the rotation centers 302 and 322, the uppermost surface of the door or a portion of the upper surface of the door (for example, a stepped portion) may function as a light receiver. Of course, the left door sensor 120L and the right door sensor 120R may be installed in each of the connecting parts 300b and 320b to function as light emitting units. Due to the positional relationship between the sensors 120L and 120R and the door, the sensors 120L and 120R determine when the door is opened at a certain angle θ or more and when the door closes at a certain angle θ while being closed. can

11 is a view showing a left hinge part and a right hinge part in detail. In FIG. 11, other components of the refrigerator are omitted for convenience of explanation. It will be described with reference to FIG. 11 below.

The door sensor may be disposed in front of the inner case, that is, in front of the opening (ie, at a position where a user is standing) in order to detect rotation of the door.

A left door sensor 120L for detecting rotation of the left door 20 may be installed in the left hinge part 300 .

A right door sensor 120R for detecting rotation of the right door 20 may be installed on the right hinge part 320 .

The left door sensor or the right door sensor may be disposed in a region between the opening and a rotation center of each of the hinge parts. That is, it may be disposed at the connection parts 300b and 320b of each hinge part.

The door sensor 120 may be connected to other components through a connector (c) and a wire in order to receive electricity or transmit a signal to the outside. At this time, the door sensor 120 may be connected to the above-described control unit 100 to provide acquired signal information.

The left hinge part 300 may have a rotation center 302 around which the left door 20 rotates. In this case, the left door sensor 120L may be disposed on the right side of the rotation center 302 . Depending on the rotation angle of the left door 20, the right space from the center of rotation 302 may or may not be located below the left door sensor 120L, in particular, the upper surface of the left door 20. It is a space where you can Therefore, it is preferable that the left door sensor 120L is located at the connection part 300b of the left hinge part 300. Through this, when the size of the door or the size of the basket is changed, the predetermined angle θ can be flexibly changed. That is, this is because the space on the right side of the connection part 300b can be easily changed, and accordingly, the position of the left door sensor 120L in the connection part 300b can be easily changed.

The wire w connected to the left door sensor 120L may be connected through the rotation center 302 . Also, the wire w may be connected to the refrigerator body through the outer case 10 .

The right hinge part 320 may have a rotation center 322 around which the right door 20 rotates. In this case, the right door sensor 120R may be disposed on the left side of the rotation center 322 . A space left of the rotation center 322 is a space in which the right door 20 may or may not be located below the right door sensor 120R according to the rotation angle of the right door 20 . Therefore, for the same reason, it is preferable that the right door sensor 120R is installed on the connection part 320b of the right hinge part 320.

The point in time detected by the door sensor 120 may be a point in time for capturing a picture of each area inside the storage compartment 22, specifically, the first area 42 or the third area 62. . That is, the time point detected by the door sensor 120 may mean the time point of capturing a photo of the external storage space of the drawer 50 .

Meanwhile, since the door sensor 120 emits and receives light, the door sensor 120 emits light downward. Light irradiated downward is reflected upward again by the door 20 .

A short distance between the door sensor 120 and the door 20 may be maintained so that a sufficient amount of light may be received by the door sensor 120 . At this time, the vertical distance between the door sensor 120 and the door 20 may be maintained at a maximum of 20 mm.

12 are examples of screens provided to the user. It will be described with reference to FIG. 12 below.

In FIG. 12A, the first area 42, the second area 52, and the third area 62 are all provided to the user, and in FIG. 12B, the second area 52 and the third area 62 are provided to the user. 12c is a form in which the first area 42 and the second area 52 are provided to the user.

First, it will be described with reference to FIG. 12A.

One picture taken by the camera 70 shows a first image showing the first area 42, a second image showing the second area 52, and the third area 62. It can be divided into a third image that does. That is, one picture or a plurality of pictures may be classified for each area. In this case, the first image, the second image, and the third image may refer to shapes cut from a single photograph taken to a range showing a corresponding area.

The classified photos may be independently provided to the user as shown in FIG. 12A. That is, since pictures of the corresponding area are individually provided to the user, the user can easily obtain information about food stored in the corresponding area.

The display 14 can be arranged so that a plurality of storage areas can be displayed within a frame.

That is, on the upper side of one square frame, the second image showing the second area 52 is disposed on the left and right sides, respectively, and on the lower side of the square frame, the third image showing the third area 62 is disposed on the left side. and are placed on the right side, respectively.

A plurality of regions may be arranged as images having the same width in one frame on the display 14 . To this end, the control unit may crop the captured picture, correct the size, and provide it to the display 14 . At this time, the corrected image is displayed at the corresponding position of the frame.

At least some of the images provided on the display are calibrated to have the same width, so that the user feels similar to looking at the storage compartment with the refrigerator door open, which can have a planar arrangement effect.

The display 14 may provide two overlapping areas through one screen, thereby providing food information in a state that an actual user cannot see at once.

In particular, since the second image and the third image are overlapping spaces when the drawer is withdrawn, the user cannot simultaneously obtain information on the two storage spaces when the drawer is withdrawn. However, the display 14 can simultaneously provide two storage space information.

The controller 100 may individually update (replace) each image to provide information about the type of food accommodated in each region to the user.

For example, if a part corresponding to the second image is upgraded through a picture taken by the camera and a part corresponding to the third image does not need to be upgraded, only the second image may be replaced.

Conversely, if the part corresponding to the third image is upgraded through the picture taken by the camera and the part corresponding to the second image does not need to be upgraded, only the third image may be replaced.

In one frame, it is possible to upgrade the second image and the third image independently of each other and regardless of other images.

Also, among the two second images disposed on the left and right sides, only the second image disposed on the left side may be upgraded or only the second image disposed on the right side may be upgraded. At this time, it is also possible to separately upgrade the third image to the left and right sides.

That is, the display 14 provides a frame in which a plurality of images are provided to provide information about items stored in the storage room to the user, but it is possible for the plurality of images to be individually upgraded regardless of other images. In particular, since the second image and the third image are images selected from different photos, the control unit can upgrade each image by determining which area the corresponding photo represents.

In this case, the second image and the third image may have the same or substantially similar widths. That is, the portions corresponding to the second image and the third image obtained from the photograph may be converted to have the same width by correction of the controller and provided to the display.

When the sizes of the second image and the third image are different, the number of pixels of the second image and the third image in the corresponding picture, that is, the size according to the pixels may be selected differently. Otherwise, it is also possible to select the second image and the third image to have the same number of pixels and correct them in the control unit so that they have different sizes when displayed on the display 14 .

It is possible to arrange the second image and the third image vertically so as to have a plane feeling of a vertical arrangement. The area originally shown by the second image is disposed above the area shown by the third image. Therefore, the user can intuitively recognize the image provided by the display, and can easily grasp the information provided by the image.

Of course, the display 14 can provide the user with a screen in which the first image and the third image are simultaneously replaced. Unlike this, a screen in which only the second image is replaced may be provided to the user while the first image and the third image remain the same.

The refrigerator compartment shelf shown in FIG. 12A may mean the first area 42, the vegetable compartment may mean the second area 52, and the multi-storage corner may mean the third area 62. Of course, it is also possible to transform into various arrangements and various shapes according to the user's preference or the convenience of the image provider.

Meanwhile, the screen of FIG. 12A may be provided to the display 14 installed in the refrigerator, but may also be provided to a separate external device other than the refrigerator, for example, an external terminal device such as a smart phone. Therefore, the user can obtain information on food stored in the refrigerator while going out of the house, and can use it for shopping. Accordingly, the user can determine the type of food stored in the refrigerator without opening the door. The screen provided to the user may be displayed by dividing one storage room into a plurality of storage areas. Accordingly, the user can easily obtain information classified by location of foods stored in each location in one storage compartment.

At this time, the image shown in the vegetable compartment shows the area from the front end of the drawer 50 to the front end of the shelf 40 when the drawer 50 is pulled out, vertically. The image shown in the multi-storage corner shows an area of an overlapping portion when the drawer 50 is retracted in a state not covered by the door 20 . The shelf shows the area above the shelf 40 to the user.

In one embodiment of the present invention, since the screen provided to the user does not show one storage room as it is, but separate images are arranged in each part separated from each other, the user can easily obtain information. In other words, rather than using the picture shown in FIG. 7 as it is, it can be said that each area is separated from one picture and used independently and individually.

As shown in FIG. 12B, a screen of a form different from that of FIG. 12A may be provided. In this case, it is possible to take a picture including the second area 52 and the third area 62 with the camera 70 . Otherwise, even if the camera 70 takes pictures including all of the first area 42, the second area 52, and the third area 62, the user sees images corresponding to the two areas. It is possible to provide only

As shown in FIG. 12c, another type of screen may be provided. In this case, it is possible to take a picture including the first area 42 and the second area 52 with the camera 70 . Otherwise, even if the camera 70 takes pictures including all of the first area 42, the second area 52, and the third area 62, the user sees images corresponding to the two areas. It is possible to provide only

That is, according to one embodiment of the present invention, it is also possible to provide related images to the user after capturing all three regions. However, as shown in FIGS. 12B and 12C , it is possible to provide the user with at least an image corresponding to the internal storage area of the drawer, ie, the second area, and an image corresponding to an area other than the second area.

13 is a diagram explaining a method of adjusting a picture taken by a camera. It will be described with reference to FIG. 13 below.

13 may be an example of a screen displayed on the display 14. Of course, if the refrigerator is not equipped with a display, FIG. 13 may refer to a screen displayed on a separate display device of a worker when manufacturing the refrigerator. It may also be a screen displayed on a user's mobile phone.

In a state where the camera 70 is installed, the camera 70 may not be accurately installed in a desired position due to assembly tolerance.

Therefore, in the embodiment of the present invention, it is possible for a worker or a user to change the adjustment line 15 dividing each region through a picture or video taken by the camera 70 .

The adjustment line 15 may include a vertical adjustment line and a horizontal adjustment line. Vertical adjustment lines may be placed in the center or at both ends of the image. The user can move the vertical adjustment line up and down and the horizontal adjustment line left and right through the buttons shown on the screen to select a portion where the image provided to the display 14 is cut off.

Meanwhile, a limit line 16 limiting a movable range of the control line 15 may also be shown. At this time, it is possible that the limit line 16 is arranged to have a predetermined interval from side to side around the vertical control line. In addition, the limit line 16 may be arranged to have a predetermined interval up and down around the horizontal adjustment line.

With the limit line 16, if a desired image is not obtained despite the movement of the control line 15, it is determined as a defect and the installation position of the camera 70 can be adjusted.

As shown in FIG. 13, in the form in which two drawers are provided in the storage compartment, the vertical adjustment line may be disposed between the two drawers.

And the leveling line can be arranged at the front end of the shelf.

In a state in which the movement of the horizontal adjustment line and the vertical adjustment line is completed, the lower left corner of the intersection of the horizontal adjustment line and the vertical adjustment line may constitute an image of the left portion of the second or third area. In this case, the corresponding image may be from the horizontal adjustment line to a specific number of pixels in a horizontal direction, and may be from the vertical adjustment line to a specific number of pixels in a vertical direction.

A lower right corner of an intersection of the horizontal adjustment line and the vertical adjustment line may constitute an image of a right portion of the second area or the third area. In this case, the corresponding image may be from the horizontal adjustment line to a specific number of pixels in a horizontal direction, and may be from the vertical adjustment line to a specific number of pixels in a vertical direction.

An upper side of the horizontal adjustment line may form an image of the first area. In this case, the corresponding image may be from the horizontal adjustment line to a specific number of pixels in the vertical direction.

Meanwhile, as shown in FIG. 13 , it is possible to separately show left and right photos of the third area corresponding to the vegetable compartment to check whether or not the photo of the portion corresponding to the drawer is obtained.

When the adjustment by the user or operator is completed, the completion of the movement of the adjustment line 15 may be entered by clicking the input completion button. Therefore, when the installation error of the camera is small, it is possible to adjust the position of the vertical and/or horizontal lines dividing each area in the photographed picture. Through this, parts corresponding to a specific area can be clearly displayed. Of course, when there are a plurality of areas divided up and down or left and right, a plurality of horizontal adjustment lines or vertical adjustment lines may be provided.

FIG. 14 is a perspective view showing a camera, FIG. 15 is a view showing a main part of the camera, and FIG. 16 is a cross-sectional view of FIG. 14 . It will be described with reference to FIGS. 14 to 16 below.

The camera 70 may include a camera module 71 including a camera lens 71a and a body 71b, and cases 73 and 74 in which the camera module 71 is accommodated.

The camera module 71 may include an image sensor that converts image information passing through the camera lens 71a into a digital signal.

The case may include a first case 73 and a second case 74 coupled to the first case 73 to form a predetermined space therein. The camera body 71b and the camera lens 71a may be accommodated in the first case 73 and the second case 74 . Between the first case 73 and the second case 74, it is possible to maintain the degree of coupling by a separate sealing member so that moisture or dust does not flow into the inside.

Since the camera body 71b and the camera lens 71a are inserted into the space formed by the first case 73 and the second case 74, the inner case 12 and the outer case 10 ) does not come into contact with the foaming agent filled between the In addition, the camera module 71 does not come into direct contact with cold air accommodated in the storage compartment 22 and can be protected.

The first case 73 may include a transparent window 80 installed at a front end of the camera lens 71a and a heater 84 providing heat to the transparent window 80 . The transparent window 80 is preferably made of a transparent material.

It is possible to prevent dew from forming on the transparent window 80 by the heat provided by the heater 84 .

If the heater 84 is installed on the camera lens 71a to prevent dew formation, the camera lens 71a may be deformed. Since the possibility of aberration and permanent damage due to such deformation increases, the transparent window 80 does not directly contact the camera lens 71a in the embodiment of the present invention.

As the material of the transparent window 80, both plastic and glass can be applied in terms of camera image acquisition, but plastic has poor thermal conductivity. Therefore, if plastic is used as the transparent window 80, power consumption may be increased compared to using glass. Therefore, it is preferable that the transparent window 80 is made of a glass material to reduce power consumption.

If the thermal conductivity is increased, the heat supplied by the heater 84 can be easily conducted from the transparent window 80 to a portion not in direct contact with the heater 84 . For example, when the thermal conductivity is lowered, it is difficult to prevent dew condensation because the temperature is lowered in a portion away from the heater 84 while the temperature is high in a portion adjacent to the heater 84 . Also, since more heat energy has to be supplied by the heater 84, energy efficiency is lowered.

Meanwhile, since the heater 84 makes direct surface contact with the transparent window 80, heat transfer efficiency can be improved.

The heater 84 is disposed on the opposite side of the surface where the transparent window 80 is exposed to the storage compartment 22, so that the heat generated from the heater 84 does not pass through the transparent window 80, and the storage compartment It can be prevented from being supplied to (22).

In particular, the heater 84 may include a hot wire coil that generates heat, and the hot wire coil may make surface contact with the transparent window 80 .

In addition, it is preferable that the transparent window 80 and the camera lens 71a are provided to have a predetermined gap (g). This is to consider deformation or assembly tolerance that may occur when heat is supplied to the transparent window 80 .

In addition, the camera 70 may include a cable 79 that receives electricity supplied from the outside and transmits a signal related to an acquired picture. The cable 79 may electrically connect the camera module 71 to external components.

Meanwhile, the case is provided with a first mounting piece 75 capable of fixing the camera 70 to other components. At this time, the first seating piece 75 is provided in either the first case 73 or the second case 74, so it doesn't matter.

The first mounting piece 75 includes a surface having a predetermined area, and the camera 70 can be placed at a desired installation position with a desired angle of inclination.

The first mounting pieces 75 are provided on both sides of the camera 70, so that the left and right sides of the camera 70 can be fixed by external components.

A second seating piece 76 formed to have a predetermined area and be stepped may be formed in the second case 74 . The second seating piece 76 is provided at a different part from the first seating piece 75 so that, like the first seating piece 75, the camera 70 is positioned at a specific position and at a predetermined angle to other components. to ensure a stable connection. Accordingly, the camera 70 may be fixed at a predetermined angle with three or more supporting points with respect to other components through the first mounting piece 75 and the second mounting piece 76 . Details of the support structure of the camera 70 will be described later.

The camera 70 may apply a resolution of VGA resolution, and a resolution of VGA (640x480) or 30 megapixels may be used in consideration of image data transmission speed and processing time.

The interface of the camera 70 is a USB type and can be driven using a 5V, 500mA USB power supply.

In addition, the current value supplied to the camera 70 may be 87 mA in standby mode and 187 mA (heater: 67 mA) in operating mode. Accordingly, power consumption proportional to the supplied current value can be consumed. At this time, the operation of the camera 70 is divided into a standby mode and an operation mode, and power may be applied to the heater 84 at all times to prevent dew condensation on the surface of the transparent window 80 .

Meanwhile, the camera 70 operates in a standby mode when a rated voltage is applied, and when the door switch 110 detects that the door 20 is open, the camera 70 is switched to an operation mode to take a picture. That is, when power is applied to the refrigerator, the camera operates in a standby mode, and when the door switch 110 detects that the door 20 is open, it can be switched to an operation mode for taking a picture.

17 is a diagram explaining arrangement of heaters. It will be described with reference to FIG. 17 below.

The camera 70 may have a horizontal angle of view of 115°, a vertical angle of view of 95°, and a diagonal angle of view of 145°. That is, the camera 70 can be disposed with a horizontal angle of view larger than a vertical angle of view. Through this arrangement, when the length of the storage compartment 22 in the width direction (horizontal direction in FIG. 17) is longer than the length in the front-back direction of the storage compartment 22 (vertical direction in FIG. 17), food information about the storage compartment 22 It is possible to obtain a photo that can provide.

Meanwhile, the heater 84 may be formed along the outer line of the transparent window 80 . At this time, the heater 84 may be arranged to form a rectangle so as not to block the angle of view of the camera 70 .

In addition, when the heater 84 has a rectangular shape, since heat can be supplied to the transparent window 80 over a relatively large area, energy efficiency of the heater 84 can be improved.

In particular, when a gap g is provided between the camera lens 71a and the transparent window 80 as shown in FIG. 16 , the angle of view of the transparent window 80 forms a rectangular shape as shown in FIG. 17 . If there is no gap g and the transparent window 80 and the camera lens 71a are in contact with each other, since the heater 84 contacts one surface of the transparent window 80, the camera lens 71a ) may be provided in the form of a square having sides equal to or greater than the diameter of ).

However, as described above, since the gap g exists and the vertical angle of view and the horizontal angle of view are different from each other, the heater 84 preferably has a rectangular shape with one side longer than the other side.

FIG. 18 is a view for explaining a picture taken while the camera is mounted on the refrigerator in a vertical direction (front and rear direction), and FIG. 19 is a view for explaining a picture taken when the camera is tilted horizontally (left and right) on the refrigerator. it is a drawing It will be described with reference to FIGS. 18 and 19 below.

FIG. 18A is a state in which the camera is turned upward by 5 degrees, FIG. 18B is installed in a desired position, and FIG. 18C is a state in which the camera is turned downward by 5 degrees. A photo of the degree shown in FIG. 18 can provide information about the food stored in the storage compartment to the user, but cannot provide desired information to the user if it is distorted at a larger angle.

* Figure 19a is a state in which the camera is turned to the left by 4 degrees, Figure 19b is a state in which the camera is turned to the left by 2 degrees, and Figure 19c is a state in which the camera is installed at a desired position.

Compared to a picture in which the camera is installed to rotate vertically as in FIG. 18, a picture in which the camera is not installed at a desired angle in a horizontal direction as in FIG. 19 provides a relatively greatly distorted screen.

As can be seen in FIGS. 18 and 19 , even a slight difference in the installation angle of the camera can greatly distort the captured picture, so it is necessary to manage the installation angle of the camera so that the assembly tolerance is as small as possible. Of course, fine errors can be adjusted through the adjustment line described with reference to FIG. 13, so that an optimal picture can be provided.

20 is a view showing an assembled camera housing, FIG. 21 is a front view of the first housing, FIG. 22 is a front view of the second housing, and FIG. 23 is a front view showing a camera installed in the first housing. 24 is a cross-sectional side view of FIG. 23, and FIG. 25 is a cross-sectional view of the camera housing installed in the inner case. It will be described with reference to FIGS. 20 to 25 below.

A camera housing may be provided to install the camera 70 to the inner case 12 . That is, a camera housing for accommodating the camera shown in FIG. 14 and fixing the camera to the ceiling of the storage compartment may be provided.

The camera housing includes a first housing 400 coupled to the lower surface of the ceiling of the inner case 12, a second housing 410 coupled to the upper surface of the ceiling of the inner case 12, and the first housing 400. ) It may include a third housing 420 and a fourth housing 430 coupled to. Here, it can be said that the first housing 400 is coupled to the upper surface of the ceiling of the storage compartment.

The third housing 420 may close the front of the camera 70 so that parts other than the transparent window 80 of the camera 70 are not directly exposed to the storage compartment 22 . That is, the third housing 420 may serve as a cover covering the front of the camera 70 . In this case, the third housing 420 and the fourth housing 430 may be made of one component. Of course, the fourth housing 430 may be a deco housing interposed between the third housing 320 and the first housing 400 .

The camera 70 is assembled in the first housing 400 to manage the degree of left and right tilt (degree of tilt in the horizontal direction) and the degree of tilt (degree of tilt in the vertical direction) of the camera 70 . can

As shown in FIG. 21 , the first housing 400 may include a first seating portion 402 on which the first seating piece 75 of the camera 70 is seated. At this time, two first seating parts 402 are provided so that the left and right first seating pieces 75 can be fastened to each other.

The first mounting piece 75 is provided to be inclined at a predetermined angle on the left and right sides of the camera 70, and may be disposed to be inclined at a desired angle in a state where the camera 70 is installed in the inner case 12. .

A fastening hole 404 is provided in the first seating portion 402, so that the first seating piece 75 and the first seating portion 402 can be fixed with a screw. The first seating part 402 is provided to be disposed at different heights in the horizontal direction, so that the camera 70 can be stably fixed.

The first housing 400 may include a second seating portion 408 on which the second seating piece 76 of the camera 70 can be seated. At this time, the second seating portion 408 is also formed to have a predetermined area, so that the degree of inclination of the camera 70 can be fixed while making surface contact with the second seating piece 76 .

That is, the camera 70 may perform a function of fixing the camera 70 so as to be tilted at a desired angle by being coupled while contacting the first housing 400 by a plurality of surfaces. In other words, it can be fixed to have three or more support points and supported to have a predetermined angle with respect to the first housing 400 at the same time. Of course, the horizontal angle may be fixed to be parallel with the first housing 400, and the vertical angle may be fixed to be a predetermined angle.

The first housing 400 may have a fastening hole 406 to be coupled to the second housing 410 by a screw.

As shown in FIG. 22 , the second housing 410 has a contact surface 416 that can come into surface contact with the upper surface of the ceiling of the inner case 12 . At this time, the contact surface 416 is widely distributed while forming a rectangular shape as a whole, so that the second housing 410 can be stably seated on the upper surface of the ceiling of the inner case 12. Therefore, the second housing ( 410) has a surface parallel to the ceiling of the inner case 12 of the contact surface 416. Through this, the reference line or reference plane for the installation angle of the camera can be the same as the ceiling plane of the inner case 12 .

The second housing 410 may include an accommodation space 412 in which the cable 79 of the camera 70 can be accommodated. A through hole 418 is formed at one side of the accommodating space 412 so that wires connected from other components of the refrigerator can be exposed to the accommodating space 412 . In this case, wires connected from other components of the refrigerator may be connected to the cable 79 in the accommodation space 412 . Therefore, electricity can be supplied to the camera 70 through the cable 79, and a signal for a photograph taken by the camera 70 can be transmitted to other components.

The second housing 410 may include a fastening part 414 coupled to the first housing 400 . The fastening part 414 may be coupled to the fastening hole 406 by a screw. When the fastening part 414 is combined with the fastening hole 406, it is preferable that no error due to assembly occurs. Therefore, it is also possible to protrude by a predetermined height so as to make surface contact with the fastening hole 406 .

As shown in FIGS. 23 and 24 , when the camera 70 is assembled to the first housing 400 , the camera 70 may be disposed inclined at a set angle. As shown in FIG. 24 , one side surface 400a of the first housing 400 has a flat shape so as to be in contact with the inner case 12 . The camera 70 is inclined at a predetermined angle when compared to one side surface 400a of the first housing 400 . In addition, the one side surface 400a is coupled to the lower surface of the ceiling of the inter case 12 so as to be parallel to the contact surface 416 of the second housing 410 . Therefore, as the camera 70 is coupled to the first housing 400 and the first housing 400 is coupled to the second housing 420, the installation angle of the camera 70 with respect to the ceiling of the storage compartment error can be significantly reduced.

Referring to FIG. 25 , the second housing 410 is disposed so as to be exposed to the upper part of the inner case 12, that is, the space between the inner case 12 and the outer case 10 filled with foam liquid. do. The second housing 410 is disposed outside the storage compartment 22 but is disposed in a space partitioned by the outer case 10 . After the second housing 410 is temporarily fixed by being attached to the inner case 12, it is finally fixed by the foam liquid filled between the inner case 12 and the outer case 10. can

The first housing 400 may be coupled to the second housing 410 with the inner case 12 therebetween. Since the first housing 400 is coupled to the second housing 410 while the second housing 410 is fixed to a specific position of the inner case 12, the position of the camera 70 is changed. can be fixed

The third housing 420 and the fourth housing 430 may be coupled to the second housing 410 when the camera 70 is tilted at a specific angle to the first housing 400 . . As described above, the third housing 420 and the fourth housing 430 are composed of one component, and the component that performs the function of the third housing 420 in the first housing 400 It is also possible to combine only

The third housing 420 may allow only the transparent window 80 of the camera 70 to be exposed to the storage compartment 22 and not expose other portions. Accordingly, it is possible to prevent the camera 70 from being affected by moisture, dust, or cold air existing in the storage compartment 22 .

The structure of the camera housing, the installation direction of the camera housing with respect to the inner case, and the installation form also have a technical meaning for reducing dew condensation on the transparent window 80 . Related specific explanations will be made later.

26 is a table comparing power consumption of cameras in the present invention, and FIG. 27 is a diagram comparing supplied standby current and driving current. 26A shows a state in which current is supplied to the camera in the comparative example, and FIG. 27B illustrates a state in which current is supplied to the camera in the present invention. It will be described with reference to FIGS. 26 and 27 below.

In the comparative example, 70 mA of standby current is supplied to the camera 70, and 50 mA is additionally supplied to the camera 70 when the door is opened, so that 120 mA of driving current is supplied to the camera 70. In a comparative example, when the camera 70 is opened 25 times a day, 253 Wh is consumed per month, and when it is opened 50 times a day, 254 Wh is consumed per month.

On the other hand, in the embodiment of the present invention, 20 mA of standby current is supplied to the camera 70, and 100 mA is additionally supplied when the door is opened, so that 120 mA of driving current can be supplied to the camera 70. When the camera 70 is opened 25 times a day, 74 Wh is consumed per month, and when it is opened 50 times a day, 76 Wh is consumed per month.

Comparing the results of the two experiments, it can be seen that the power consumed by the camera 70 can be reduced by approximately 70% by lowering the standby current. Accordingly, it was confirmed that the power consumed by the camera 70 can be greatly reduced by lowering the standby current under the premise that the driving current is the same. This is because, when the refrigerator is operated continuously for 24 hours, the door open time is relatively very short. Therefore, assuming that the current value required for photographing in the operating mode is constant, the lower the current value applied to the camera in the standby mode, the lower the current consumption.

When the standby current is lowered, as shown in FIG. 27 , the timing at which driving current is supplied to the camera 70 is delayed. This is because the delay time inevitably increases due to the difference between the standby current value and the driving current value. In other words, when the standby mode is switched to the drive mode, the photo is not immediately taken, but the photo must be taken after a predetermined delay time has elapsed.

Accordingly, the point at which a picture is actually taken by the camera 70 is delayed. If there is a large difference between the time when the command to take a picture is transmitted to the camera 70 and the time when the camera 70 actually takes a picture, the picture actually taken by the camera 70 provides food information to the user. may not be able to provide

In other words, the lower the standby current, the lower the power consumed by the camera 70. However, the delay time to increase the driving current for driving the camera 70 may increase. Therefore, it is preferable to calculate a normal time that elapses before a user can take a picture at a desired time and select a standby current value so that the delay time is less than the corresponding time. For this reason, according to an embodiment of the present invention, it is preferable to increase the applied current step by step from the standby current value to the driving current value. That is, the applied current value can be increased in stages from a relatively low standby current value to a relatively high driving current value with a sufficient delay time. That is, it is possible to prevent a rapid increase in the current value. Accordingly, the stability of the camera can be achieved while reducing power consumption.

Therefore, in the embodiment of the present invention, as shown in FIGS. 28 and 29, the camera 70 is controlled to start driving at the time when the door 20 is opened. That is, when it is determined that the door is opened through the door switch, the standby mode is switched to the driving mode. However, among the photos taken by the camera, a valid photo is taken after at least a predetermined time has elapsed from the time the door is opened. This is because it physically takes more than a few seconds for the door to open and for the user to update the food. In other words, even if the delay time is long, the picture can be obtained at the desired time. Therefore, in the present embodiment, it can be said that the transition time of the camera from the standby mode to the driving mode is performed according to the detection result of the door switch. And, it can be said that the reference point of time for selecting a valid photo is performed by the detection result of the door sensor.

28 is a diagram explaining a camera's picture-taking start point and continuous picture-taking by the camera. It will be described with reference to FIG. 28 below.

Meanwhile, in the present invention, as shown in FIG. 4 , the door switch may be included. When the opening of the storage compartment 22 is detected by the door switch 110, the control unit 100 may command the camera 70 to start taking pictures. That is, when the door 20 is moved or rotated to open the storage compartment 22 , the camera 70 can start taking pictures.

On the other hand, when the door switch 110 detects that the storage compartment 22 is sealed, the control unit 100 may command the camera 70 to stop taking pictures. That is, when the door 20 is moved or rotated to close the storage compartment 22 , the camera 70 may stop taking pictures.

As shown in FIG. 28, when electricity is supplied to the refrigerator by an external power source, standby current is supplied from the camera 70 (S1). At this time, the camera 70 may be driven in a standby mode. In standby mode, the camera 70 does not take pictures.

Opening of the door 20 may be detected by the door switch 110 (S10).

When it is determined that the door 20 opens the storage compartment 22, the camera 70 may start taking pictures (S14). In particular, the camera 70 may continuously take pictures at predetermined time intervals.

At this time, since a current greater than the current supplied in the standby mode is supplied to the camera 70, the camera 70 can operate in the drive mode. To drive the camera 70, a predetermined amount of time, that is, a delay time, must occur in order to supply a current greater than the standby current and a driving current. That is, it actually takes a certain amount of time for the magnitude of the current supplied to the camera 70 to increase. After a delay time of a predetermined time has elapsed after the current for performing the driving mode is supplied to the camera 70, the driving current is supplied to the camera 70 so that the camera 70 can finally take a picture. .

Since the camera 70 is switched to a driving mode for taking pictures from the time the door 20 is opened, the start time at which pictures are taken is early, reducing the risk of not taking pictures at a specific time. This is because if the standby current is supplied until the point in time at which photography is desired and then the current value is increased at the desired point in time, the time point at which the picture is taken is delayed due to the delay time.

In addition, the number of pictures taken per second by the camera 70 can be reduced. This is because, assuming that the camera 70 takes 30 pictures per second, the possibility of a change in the storage compartment 22 occurring during the time interval at which pictures are taken, that is, 1/30 second, is extremely small. The interior of the storage compartment 22 can be changed by a user's motion, because it is unlikely that a general person will end an action in the storage compartment 22 for 1/30 second. That is, the time interval photographed by the camera 70 can be relatively increased. Therefore, since the relatively inexpensive camera 70 can be used, the cost of the refrigerator can be reduced.

A picture taken by the camera 70 may be stored in the storage unit 18 (S16). In addition, if the number of pictures taken per second by the camera 70 decreases, the number of pictures stored in the storage unit 18 also decreases, so that the storage unit 18 can be operated more efficiently.

Meanwhile, the door sensor 120 can detect whether the door 20 is rotated at a specific angle (S18).

If the door sensor 120 does not detect whether the door 20 is rotated, a photo stored in the storage unit 18 before a predetermined time may be deleted (S90). That is, photos that are not processed to be provided to the user are deleted, thereby preventing an increase in the capacity of the storage unit 18 .

On the other hand, if the door sensor 120 determines that the door 20 has been rotated at a specific angle, a photo taken at that time is selected (S92). At this time, the selected photo may be processed by the control unit 100 as an effective photo and provided to the user. For example, a photo processed by the controller 100 may be transmitted to the display 14 .

In order for the user to access the storage compartment 22, it takes a certain amount of time to rotate the door 20. Since the delay time of the camera 70 elapses during that time, a picture may be taken at the time when the door 20 is rotated at a specific angle. Accordingly, the controller 100 can select a photo at a desired point in time.

If the door sensor 120 is switched from the standby mode to the drive mode so that the camera 70 takes a picture at the time it is detected, there is a problem that the storage room state after the delay time has elapsed from that point in time cannot help but be photographed. can

Meanwhile, since the camera 70 starts shooting when the door 20 is opened, it is possible to supply a relatively small standby current to the camera 70 . If the standby current is low, the delay time until the driving current is supplied increases, but in the control flow described above, since the camera 70 starts shooting early, the problem of not taking a picture at the desired time does not occur. .

FIG. 29 is a diagram explaining a camera's picture-taking start point and continuous picture-taking by the camera. It will be described with reference to FIG. 29 below.

Unlike FIG. 28, in FIG. 29, the photo selection time is not determined according to the rotation state of the door 20, but the photo selection time is determined according to the state of the drawer 50. Other parts are the same, and the same parts are omitted for convenience of description. However, it can be said that the operation of the drawer 50 is based on the opening of the door 20 . A reference point in time for selecting a specific photo may be determined by a door sensor, and a reference point in time for selecting a photo for a specific area within a specific photo may be determined by a determination of the door sensor and a determination of the drawer detector 130. You will be able to. Therefore, in any case, the determination result of the door sensor is required to determine the reference viewpoint of the effective photo.

To store food in the drawer 50 or to take out the food stored in the drawer 50, the user may withdraw the drawer 50. At this time, the user can withdraw the drawer 50 to a sufficient length in order to use the drawer 50 .

After the user finishes using the drawer 50 , the drawer 50 may be pushed in. The time when this state occurs may mean the specific state described in S19.

Since the camera 70 starts taking pictures from the moment when the door 20 is opened in the door switch 110, if the drawer 50 is placed in a specific state, a picture taken at that time is taken. It can be selected and provided to the user (S92).

When the door 20 is opened, a picture is taken by the camera 70, and the movement of the drawer 50 can be monitored from the time the picture is taken, that is, the time the door 20 is opened.

FIG. 30 is a diagram showing an embodiment of the drawer sensing unit, and FIG. 31 is a diagram explaining a method of detecting movement of the drawer in the drawer sensing unit of FIG. 30 . It will be described with reference to FIGS. 30 and 31 below.

Referring to FIG. 30 , the drawer detection unit 130 may be implemented as a separate component instead of being included in the control unit 100 .

The drawer sensing unit 130 may include one light emitting unit and two light receiving units. That is, the moving direction of the drawer 50 can be detected by analyzing the timing when light emitted from one light emitting unit is incident on the two light receiving units.

A plurality of slits 134 may be continuously disposed in the drawer 50 so that light emitted from the light emitting unit of the drawer detecting unit 130 is not reflected when incident on the slits 134 . Accordingly, light may be sequentially incident to the two light receivers.

In FIG. 31, the x-axis means time, and the y-axis means a signal when light is received by the two light receivers (Photo TR1 and Photo TR2). That is, in FIG. 31A , light is received by Photo TR1 before Photo TR2 , and in FIG. 31B , light is received by Photo TR2 before Photo TR2 .

In FIG. 31B, it means a moving state in which the user withdraws the drawer 50 to use it, and in FIG. 31A, the user pulls the drawer 50 back to its original position after finishing using the drawer 50. can mean status.

Accordingly, when the drawer detecting unit 130 detects a signal as shown in FIG. , that is, a picture corresponding to the third area may be selected. That is, the acquisition time of the photo corresponding to the third area may be determined according to the result of detection or determination by the drawer detector 130 . Of course, at this time, the part corresponding to the third area in the acquired picture may be independently and individually selected, separated, and updated from the part corresponding to the other areas.

32 is a diagram illustrating a state in which a marker is displayed on a drawer. It will be described with reference to FIG. 32 below.

30 and 31, the movement of the drawer 50 can be detected using a marker, which is a specific mark on the drawer 50, without using a separate mechanical or electronic device. .

Various types of markers (patterns having black and white colors) may be provided in the drawer 50 . The camera 70 may analyze the movement of the marker through the photographed picture, and read whether the drawer 50 moves and the movement direction. At this time, it is possible to analyze the marker in the drawer detection unit 130 of the control unit 100 . That is, the drawer detecting unit 130 may determine the state of the drawer 50 by analyzing movement of markers in consecutively taken pictures. Accordingly, the acquisition time point of the photo corresponding to the third region may be determined according to a result of detection or determination by the skin detector 130 . Of course, at this time, the part corresponding to the third area in the acquired picture may be independently and individually selected, separated, and updated from the part corresponding to the other areas.

In FIG. 32 , the marker includes a simplified image of a camera shape and the words 'smart sensor'. On the other hand, the marker may include various types of patterns, and may be provided in a form in which black and white intersect each other so that the camera 70 can recognize them.

At this time, the marker may be displayed on the upper front side of the drawer 50 so that the camera 70 can recognize it.

FIG. 33(a) is a view showing a picture taken when the left and right drawers are withdrawn, and FIG. 33(b) is a view showing a state in which the left drawer is retracted and the right drawer is withdrawn. Hereinafter, a description will be made with reference to FIGS. 33(a) and 33(b).

In FIG. 33 , an example in which the marker is provided at the center of the upper side of the drawer 50 and forms a wavy pattern is shown.

The first region 42 may be expressed in one picture taken by the camera 70 . In addition, one of the second area 52 and the third area 62 may be shown in one picture.

That is, as shown in FIG. 33(a), the first area 42 disposed on the left and right sides and the second area 52 disposed on the left and right sides are shown in a picture taken with the left and right drawers drawn out.

On the other hand, as shown in FIG. 33(b), in the picture taken with only the right drawer drawn out, the first area 42 disposed on the left and right, the third area 62 on the left, and the second area 52 on the right ) can be shown.

That is, when the drawer 50 is withdrawn, the third area 62 is not shown and the second area is shown in the picture. On the other hand, when the drawer 50 is drawn in, the second area 52 is not shown and the third area 62 is shown in the picture.

The camera 70 takes a picture at the same location. The drawer 50 can be moved, and therefore, pictures acquired by the camera 70 can take pictures of various areas according to the position of the drawer 50 .

Meanwhile, as shown in FIG. 33(b), when the drawer 50 provided on the left side is completely drawn into the lower space of the shelf 40, the marker is not recognized by the camera 70. That is, the marker is not shown in the picture taken by the camera 70 .

A method of detecting movement of the drawer using a marker will be described with reference to FIGS. 34 to 40 .

34 is a flowchart for explaining a method of operating a refrigerator according to an embodiment of the present invention.

Referring to FIG. 34, when the door switch 110 detects the opening of the door 20, the camera 70 starts taking a picture (S2101), and the controller 100 performs a marker recognition operation and marker tracking. Operation starts (S2103).

Hereinafter, reference numerals S2101 and S2103 will be described with reference to FIG. 35 to describe steps S2101 and S2103 in more detail.

35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.

Referring to FIG. 35 , when the door 20 is opened, the camera 70 starts taking pictures (S2301).

The controller 100 controls the camera 70 to acquire an inside picture of the drawer 50 (S2303).

Subsequently, the controller 100 processes the acquired photo (S2305) and determines whether a predetermined marker is recognized in the acquired photo (S2307).

In order to determine whether a predetermined marker is recognized, the controller 100 may compare whether a marker recognized from an acquired picture is the same as a previously stored marker.

The predetermined marker may be determined by a business operator or a user, and may be previously stored in the storage unit 18 or the like.

A marker may be formed in a searchable area in a picture taken by a camera. Hereinafter, markers that can be recognized and tracked by the controller 100 will be described in detail with reference to FIGS. 36 and 37 .

36 is a diagram for explaining a marker for detecting a draw-in/out time point of a drawer according to an embodiment of the present invention.

Referring to (a) of FIG. 36 , a marker may be formed in an area searchable by the camera 70 from the time the drawer 50 starts to open. For example, when the camera 70 is positioned at the top of the refrigerator and can capture a plane of the drawer 50, the marker may be positioned at the front end of the top of the drawer 50. In addition, the field of view of the camera 70 may be installed so as to take pictures from a state in which the drawer 50 is completely closed to a state in which the drawer 50 is fully opened.

When the marker is located at the top of the drawer 50, there is no obstruction at the top of the drawer 50 at the time when the drawer 50 is closed after the storage inside the drawer 50 is finished. It is planned, but it is not limited thereto.

Referring to (b) of FIG. 36 , when looking at the drawer compartment 54 in which the drawer 50 is accommodated and the drawer 50 from the viewpoint of the camera 70, the camera 70 sees the drawer ( The inside of the drawer 50 including the marker can be photographed in all ranges from the state in which the drawer 50 starts to open to the state in which the drawer 50 is fully opened.

37 is a view for explaining the shape of a marker according to an embodiment of the present invention.

Referring to FIG. 37 , the marker may be formed in a form in which identical shapes of two colors are crossed and repeated.

Referring to (a) of FIG. 37 , the marker may be formed in a form in which a plurality of bands in which white squares and black squares are alternately repeated are intersected and arranged.

Referring to (b) of FIG. 37 , the marker may be formed in a form in which a plurality of bands in which a white slanted rectangle and a black slanted rectangle are alternately repeated are intersected and arranged.

As described above, the repeating marker has an advantage in that the controller 100 can recognize and track the marker even when a part of the marker is covered from the camera shooting range. However, the marker is not limited to a repeating shape, nor is it limited to a plurality of colors. As described above, the marker suffices to have a shape capable of recognizing whether or not movement of a movable internal component of the refrigerator, such as the drawer 50, is sufficient.

Referring again to FIG. 35 , when a predetermined marker is recognized, the controller 100 detects movement of the recognized marker (S2311). The controller 100 continuously or periodically analyzes and processes the acquired picture to determine whether the recognized marker is moving or not. For example, the controller 100 may determine whether the marker is moved or stopped when the opening operation of the drawer 50 continues or ends or when the closing operation of the drawer 50 starts, continues, or completes. .

Alternatively, when the predetermined marker is not recognized, the controller 100 notifies the user that marker recognition has failed (S2309).

For example, when an obstacle is located in a marker, the camera may not acquire a picture including the marker, or the controller 100 may not be able to recognize the marker from the acquired picture. In this case, the controller 100 may notify the user of information indicating failure in marker recognition, information indicating a method for successful marker recognition, such as removing an obstacle, etc. through the display 14 or the like.

Subsequently, when the marker recognition fails, the controller 100 may repeatedly perform a step of recognizing a predetermined marker by processing a photo newly acquired through a camera.

Referring back to FIG. 34 , when a drawer closing operation is detected after recognizing the marker (S2105), the controller 100 determines the inside of the drawer (the third area 62) through a photograph taken at that time. A final picture is acquired (S2107). At this time, the final picture of the inside of the drawer obtained by the controller 100 may be the inside of the drawer immediately before the drawer closing operation is detected or the inside of the drawer immediately after the drawer closing operation is detected, It is not limited to this.

Whether or not the drawer closing operation is detected will be described with reference to FIG. 38 .

38 is a flowchart for explaining a method of detecting a drawer closing operation according to an embodiment of the present invention.

Referring to FIG. 38 , the controller 100 determines whether an end of the opening operation of the drawer 50 has been detected (S2501). For example, the controller 100 recognizes a marker from a previously acquired picture of the drawer 50, tracks the movement of the marker, and detects that the movement of the marker has stopped, the opening operation of the drawer 50 is performed. You can detect that it has ended.

Subsequently, the control unit 100 determines whether or not the start of the closing operation of the drawer 50 has been detected (S2503). For example, the controller 100 may detect that the closing operation of the drawer 50 has started when the marker stops moving and starts moving again in the opposite direction.

Meanwhile, an operation of the controller 100 that detects a drawer closing operation related to another embodiment of the present invention will be described with reference to FIG. 39 .

39 is a flowchart for explaining an operation of a refrigerator performed upon detection of completion of a drawer closing operation according to an embodiment of the present invention.

When the end of the drawer closing operation is detected (S2701), the controller 100 determines whether or not the final photo has been successfully acquired through marker recognition (S2703).

The control unit 100 may detect that the closing operation of the drawer 50 has ended through image processing of the drawer detection unit 130 .

When obtaining the final picture fails as a result of the marker not being normally recognized due to an obstacle or the like, the controller 100 notifies the user that the marker recognition has failed (S2705).

The controller 100 may notify the user of information indicating that marker recognition has failed through the display 14 or the like. This may guide the user to remove the obstruction and lead the drawer to be drawn in and out for normal marker recognition and final image acquisition.

Referring back to FIG. 34 , the controller 100 displays the final image using the display 14 or the like (S2109). At this time, the form provided to the user may be a portion extracted only from the portion corresponding to the third area 62 in the acquired final photo. Next, with reference to FIG. 40, final image control according to another embodiment of the present invention will be described.

40 is a flowchart for explaining a method of controlling an image of a specific region inside a refrigerator captured at a specific point in time according to an embodiment of the present invention.

Referring to FIG. 40 , the controller 100 may obtain a final picture (S2901) and cut out an image corresponding to a desired area from the final picture. Then, the obtained final image is stored in the storage unit 18 (S2903).

Subsequently, the controller 100 determines whether a user input for displaying a final image is received through a user input or the like (S2905).

For example, when acquiring the final photo, the controller 100 may form a user input unit in the form of an icon together with information indicating success in acquiring the final photo on the display 14 composed of a touch screen, and depending on whether the corresponding icon is selected, the user input unit may be displayed. It can determine whether input is received or not. The user input unit may include, but is not limited to, physical keys as described above in addition to icons formed on the touch screen.

Next, when receiving a user input for displaying the final image processed from the final photo, the controller 100 displays the final image (S2907). The controller 100 may display the final image using the display 14 . The display of such a final image may be referred to as displaying by replacing an existing image. Therefore, it can be said to be an update of the final image.

Of course, the update of the final image may be automatically performed regardless of a user's input.

According to this embodiment of the present invention, the refrigerator can detect movement of the drawer or the like by recognizing a pre-formed marker on the drawer or the like. In addition, the refrigerator may acquire a final state of storage stored in a drawer or the like in the form of an image using a camera installed therein. In this case, the refrigerator may determine an optimal time point at which the final storage state of the drawer or the like can be obtained using a marker formed on the drawer or the like.

Hereinafter, the installation position of the camera and the pictures obtainable accordingly will be described.

In an embodiment of the present invention, a camera 70 is installed outside the drawer 50, that is, in the storage compartment 22 to form the inside of the drawer 50 and the drawer 50 (i.e., It is possible to simultaneously recognize the movement state of the drawer as well as the storage state of goods inside the drawer by continuously taking pictures including the moving path of the front and side walls and the top of both side walls. Of course, the picture taken by the camera 70 is also taken up to the upper space of the drawer 50, that is, the portion corresponding to the first area 42.

Basically, the picture taken by the camera should be able to include both the opening and closing motion of the drawer and the moving path of the drawer. Therefore, the camera should be installed at a position where the opening and closing and moving paths of the drawer can be photographed from the outside of the drawer. That is, it is preferable that the camera is installed at a position where a single picture can be taken of at least a closed state and a completely open state of the drawer.

Furthermore, the camera 70 includes not only the drawer area but also another storage area of the storage compartment, for example, a shelf area divided by shelves and storing food or products in one picture with one shot. Thus, a single camera can be configured to cover other storage areas including a drawer area, that is, a plurality of storage areas.

To this end, the drawer 50 is preferably formed on the bottom surface of the refrigerator storage compartment. The drawer 50 may include a front wall, two side walls, a rear wall, a bottom wall, and an upper opening. The camera 70 should be provided on the top of the drawer 50 so that at least a part of the inside of the drawer and an upper portion of the front wall of the drawer in a completely open state may be exposed.

In addition, a marker having a specific pattern is formed or attached at any position on the upper surface of the front wall or both side walls of the drawer exposed to the camera.

At this time, since the camera 70 is fixed at a predetermined position, it is necessary to configure the position of the print unit of the drawer 50 to be relatively changed. That is, when the camera 70 receives a drive command from the control unit, it continuously takes pictures at a constant speed and sequentially transmits the captured images to the control unit 100 . The control unit 100 tracks the position of the printing unit from the picture of storage of goods in the in-room transmitted from the camera 70 to determine the open state of the drawer as well as information on the movement of the drawer.

The range that the camera can capture may be determined by an angle of view of the camera, which includes a vertical angle of view that means a range of capture in a vertical direction and a horizontal angle of view that determines a range of capture in a horizontal direction.

One end of the vertical angle of view of the camera 70 should include at least the front wall of the drawer when the drawer is fully open, and the other end should include a print located on the top surface of the front wall or both side walls of the drawer when the drawer is closed. It should be determined to include a part of the front surface of the upper side of the drawer covering the upper opening of the drawer, for example, up to a portion that can be confirmed that the portion is not exposed.

At this time, it is also possible to simultaneously recognize another storage space along with the inner area of the drawer with a single camera. Another embodiment of the present invention provides a separate storage space (hereinafter Storage compartment floor storage compartment) can be configured. In this case, the vertical angle of view of the camera 70 can be configured to recognize another storage space (floor storage compartment) including the inner area of the drawer even in the shooting range including the fully open and fully closed drawer states. may be

Meanwhile, according to another embodiment of the present invention for the vertical angle of view of the camera 70, the inner region of the drawer and the shelf region may be simultaneously photographed. Accordingly, the range of the other end of the vertical angle of view may be determined to include up to any one part of a plurality of shelves arranged at regular intervals vertically. In making the camera vertical angle include the shelf area, a preferred example may include up to a part of the top shelf among the plurality of shelves, and more preferably, at least the top shelf disposed at the top of the storage compartment. It is to include the rear end.

According to these embodiments, it is possible to check the storage state of goods in at least two storage spaces, including the inner region of the drawer, with a single picture taken by a single camera.

To this end, a preferred installation location of the camera is required for the drawer formed at the bottom of the storage compartment. The inventors of the present invention first reviewed the camera installation location disclosed or known through published data at the time of filing the present invention. It was installed on the top of the door outside the refrigerator storage room to examine whether it could take pictures of the inside of the storage room.

When the camera is installed outside the storage room (door), the camera is always exposed to the outside temperature, so there is no condensation around the camera, so there is no need for a separate anti-condensation structure. However, it is difficult to photograph the inside of the storage room with a camera mounted on the door because the user's operation of storing items in the drawer and the storage space in the storage room is always performed with the door open. Particularly, an object of the present invention is to obtain not only a motion state of the drawer outside the drawer but also a storage state of goods inside the drawer. However, since the user's operation in the drawer is always in an open state and the drawer is also in a closed state when the door is closed, it does not correspond to the purpose of the present invention at all.

Therefore, in order to photograph at least the inside of the drawer and the movement state of the drawer, which is one aspect of the present invention, it is preferable to arrange the storage chamber so that the storage chamber can be photographed inside the refrigerator, that is, in the storage chamber of the main body where the drawer to be photographed is located. do.

On the other hand, in accordance with one aspect of the present invention, when the camera is installed at the bottom of any one of the shelves located on the top of the drawer with the camera facing toward the opening of the storage compartment, it is impossible to take a picture of the shelf area first. It is difficult to install and wire a driving unit capable of driving a camera on the shelf, and since a wide-angle camera must be used to capture all the movement paths of the drawer, there may be a problem of increasing distortion in the captured image.

The inventors of the present invention found that when the camera is installed on the sidewall of the storage compartment in the range where the drawer can be exposed, the image showing the storage state of the goods inside the drawer is distorted due to the asymmetric image due to the difference in distance from the camera in the left and right directions of the refrigerator storage compartment. I knew it was serious. Therefore, it can be seen that the satisfaction of these users can be greatly reduced.

Above all, one object of the present invention is to track the movement state of the drawer without a separate sensor and detect the open/closed state of the drawer and its extent. The marker may be photographed in a relatively enlarged state (image distortion). Therefore, the size (pixel size) occupied by the marker portion relative to the entire size constituting one image may increase, and thus the location change may not be revealed relatively well. In addition, since the range to be processed to determine whether the position of the marker has changed is increased, the image processing time may increase, so that images transmitted continuously may be missed or not be processed.

Therefore, it was found that installing the camera on the sidewall or corner of the storage compartment, which was already known at the time of filing the present invention, does not conform to the purpose of the present invention.

In one embodiment of the present invention, the camera is installed on the upper wall of the storage compartment corresponding to the upper opening of the drawer, and the upper surface of the storage compartment on which the drawer is closed and the drawer is completely open is projected. It was installed within the range, and the direction of the camera was toward the inside of the storage room toward the rear wall of the storage room, but the focus of the camera was tilted at a certain angle so as to face any one point on the rear wall of the storage room. As a result, it was possible to obtain an image with minimized distortion in the above-described embodiments of the vertical angle of the camera of the present invention.

In determining the position of the camera, another embodiment of the present invention is installed on the upper wall of the refrigerator storage compartment, and the front end of the uppermost shelf among the shelves arranged vertically in the storage compartment at the front opening of the storage compartment is attached to the upper wall of the storage compartment. It is to place it between the projected positions. In addition, the direction of the camera is toward the inside of the storage compartment toward the rear wall of the storage compartment, but the focus of the camera is tilted at a predetermined angle so as to face any one point on the rear wall surface of the storage compartment.

Preferably, the camera is located in the central part of the upper part of the storage compartment, and correspondingly, the printing part located on the upper surface of the front wall or both side walls of the drawer is preferably installed on the upper surface of the front wall of the drawer. More preferably, the vicinity of the upper surface of the front wall of the drawer corresponding downward to the position of the camera installed on the upper surface of the storage compartment is proposed.

In a refrigerator that opens and closes the opening of one storage compartment with one or more doors, that is, left and right doors, the camera is installed on the upper surface (ceiling surface) of the storage compartment at the boundary between the left and right doors, and the drawers are present on the left and right respectively. In one case, the printing unit is positioned close to the left and right borders. That is, from the perspective of looking into the inside of the storage compartment, the left drawer is preferably positioned at the upper right end of the front wall or the upper right end of the right side wall, and the right drawer is positioned at the upper left end of the front wall or the upper end of the left side wall. In other words, it is preferable that the printed parts are formed at positions close to the vertical lower part of the camera. This is because the possibility of distortion of the printed part increases as the distance from the vertical bottom to the left and right increases. An example of this is shown in FIG. 41B.

At this time, when the printing unit is installed on both sides of the upper surface of the front wall of the left and right drawers or at the ends of the upper surfaces of both sides, the drawers used in the storage room that are opened and closed by the left and right doors can be used in common regardless of the left and right positions. can increase efficiency.

As described above, according to the location of the camera and the location of the printing unit according to the present invention, all moving paths from the closed state to the open state of the drawer are included within the shooting range of the fixed camera, so there is no need for a separate sensor device. Not only the opening/closing state of the drawer, but also the degree of opening/closing and the stop or movement of the drawer can be detected, and at the same time, the storage state of the article inside the drawer can be detected or confirmed by a single camera.

In addition, in addition to the inner area of the drawer, the storage state of the storage compartment installed on the floor of the storage compartment can be checked while the drawer is closed, so that a plurality of storage areas can be recognized with a single camera.

* In addition, when the angle of view of the camera extends to the shelf area, it is possible to selectively recognize three storage areas with one camera, including the drawer inner area, the storage room floor compartment, and the shelf area.

On the other hand, since the camera installed on the upper surface of the storage chamber and the printed part formed or attached to the upper part of the drawer are aligned in an approximately straight line (alignment), the printed part in the entire image of the storage room captured by the camera and transmitted. There is almost no distortion of the printed part, and since the space occupied by the printing part can be relatively minimized, it is possible to minimize the time to track and determine the location of the printing part through image processing.

A method for recognizing traces of markers formed in the drawer will be described below.

A printing unit (marker or printing paper) that can be recognized by image reading may be installed on a specific part of the drawer body exposed to the camera. By tracking the location of the marker unit, the degree of opening/closing of the drawer and the open/closed state of the drawer can be detected. The marker unit may be configured to be exposed to a camera from a closed position to a fully open position on the movement path of the drawer, and an open/closed state of the drawer may be determined based on a position of the marker unit. That is, various state information of the drawer can be determined through a change in the position of the marker in continuously photographed pictures.

As described above, the structure of the printing unit (Marker) may be formed of a specific pattern in order to track and recognize the moving state or opening/closing state of the drawer. At this time, the printing unit may be disposed at various positions on the upper surface of the drawer exposed to the camera.

The printing unit is preferably installed at an upper end of a front wall of the drawer or at an upper end of both side walls of the drawer, which may be a boundary of a maximum open portion of the drawer.

The shape of the pattern of the printing unit may be formed by repeating patterns having a certain size and a certain shape in contrasting colors, and the shape may be formed in various forms such as a square, a triangle, a circle, and a wavy shape.

There is a concern that such a printing unit may deteriorate the overall aesthetics of the refrigerator, so some users may have a sense of rejection. Therefore, it is necessary to minimize the repetition of patterns so that there is no objection to the aesthetic sense felt by the user, and furthermore, it is possible to consider a printing unit in a form that can appeal to the user's aesthetic sense.

However, this design limitation is not limited to a simple design problem, and a case in which the position of the drawer cannot be accurately recognized occurs. That is, the recognition rate of the printed part of the drawer is affected, and ultimately, the position of the drawer cannot be accurately detected.

That is, considering the design element that requires a significant reduction in the size of the printing unit, it is possible to select the degree of repeating only about 2 to 4 times without repeating several times with a color contrasting with a pattern of a certain shape. In this case, the control unit processing the image captured inside the refrigerator may not accurately recognize the boundary of the printing unit (marker) in the entire image.

Therefore, according to a preferred embodiment of the present invention for solving this problem, since the drawer is usually made of a bright color, transparent or translucent color, it is possible to select a color contrasting thereto. Referring to FIG. 41A , the length L1 in the vertical direction with respect to the moving direction of the drawer (forward and backward movement) may be set to be longer than the length L2 in the drawer moving direction. Then, a first color portion having a certain length in the vertical direction is formed, and a color contrasting with the first color portion or a color identical to that of the drawer is formed with approximately the same thickness in front or rear of the first color portion with respect to the drawer moving direction. A second color portion is formed, and the second color portion is continuously and repeatedly formed in the same color, approximately the same thickness, and the same length as the first color portion. That is, it is possible to arrange the first color part and the second color part sequentially with each other. At this time, it is more preferable that the length L1 in the vertical direction be greater than the length L2 in the drawer direction in the outer sizes of the markers disposed in the first color part, the second color part, and the first color part.

According to another marker shape of the present invention, both ends of the first color portion at the front and rear ends are connected to each other to surround the second color portion, as shown in FIG. 41B. The third embodiment of the marker type shown above can be an example of such a structure.

That is, in an exemplary embodiment of the present invention, the length L1 in the vertical direction with respect to the drawer moving direction is larger than the length L2 in the drawer moving direction, but the part bordering the drawer (especially the drawer) is formed. It is to form a color (dark color) contrasting with the drawer (with respect to the direction of movement of the lower), and to include a bright color part in contrast to the dark color in the middle.

When the printing unit is configured in this way, since it can be configured in a small size, it is possible to satisfy the user's requirements for aesthetic sense, and the size of the printing unit portion is relatively small in one whole image taken by the camera. Accordingly, an image processing area for detecting a change in the position of the printing unit becomes smaller so that the image processing speed can be improved. This works advantageously for the image processing speed of the controller to match the image capturing speed that is continuously photographed and transmitted, thereby further improving the accuracy of acquiring images at the correct time. For example, if the size of a part to be processed is large in pictures that are continuously taken, the load of the control unit is high. Therefore, a very expensive control unit must be used. However, if the size of the part to be processed is small, the load of the controller can be reduced. Therefore, it is possible to effectively process an image even when using a low-cost controller.

On the other hand, as shown in FIG. 41B, it is preferable that the markers include letters, symbols, or patterns so that the user can intuitively understand the purpose of the markers by looking at the markers.

Hereinafter, a method of recognizing marker coordinates of a drawer and processing a camera image will be described.

42 is a diagram explaining a marker position recognition method. Referring to FIG. 42 , a method of recognizing the position of a marker in a drawer will be described below.

When a user wants to newly store (warehouse) or ship goods into the drawer, the user first opens the door. When the door is separated from the front of the storage compartment, the door switch senses it, and the control unit receives the signal from the door switch and drives the camera installed in the refrigerator in the air. When the camera receiving the driving command from the control unit applies a camera driving voltage that is not in a standby voltage state, the camera is ready to drive and starts continuous shooting.

The user will then hold the handle on the front wall of the drawer mounted in the refrigerator storage compartment and pull the drawer to open the drawer. At this time, since the camera is continuously photographing, the open state of the drawer is photographed at regular intervals and the image is sent to the controller.

Meanwhile, a certain type of marker is formed in a portion of the drawer exposed to the camera, and a control unit receiving an image sent from the camera divides the entire image into certain pixels and assigns coordinate values at regular intervals, The coordinates of the marker are analyzed on the coordinates of the image.

Accordingly, if an image captured while the drawer is closed is sent to the controller, the corresponding image marker portion will not be exposed, and the marker coordinates at this time will have the coordinates of 0.

When the user opens the drawer in a closed state, the camera takes a picture while the marker is exposed, and the camera can take a picture including the marker. When this image is sent to the controller, the controller recognizes the coordinate values of the markers on the entire image and compares them with the previous positions. For example, at point 'A' in FIG. 42, when the coordinate value of the marker changes from '0' to '50' as a certain coordinate value, for example, the controller can determine that the drawer is moving from the changed coordinate value. there is. In particular, when it is changed from '0', it can be determined that the drawer has just started to open.

If the user continues to open the drawer, the control unit may determine that the drawer is being opened from the fact that the drawer is moving through a series of processes as described above and the coordinate value changes to a larger value.

When the user opens the drawer to the fully open state, the control unit recognizes that the coordinate values do not change any more and determines that the drawer is fully opened or stopped. For example, the point 'B' in FIG. 42 is the point where the marker coordinate value does not change from '90' to '90', so it is recognized as 'the drawer is stopped' or 'the drawer is completely open'. can judge

After the drawer is normally opened, regardless of whether it is fully opened or partially opened, it can be determined that the user is performing some operation with respect to the drawer when it is in a stopped state. Even when the drawer is stopped, the camera continuously captures images of the inside of the chamber and sends them to the controller, which monitors whether there is a change in the coordinates of the marker.

When the user puts an item in the drawer or removes the item from the drawer, the user closes the drawer again, and at this time, from the image captured by the camera, the control unit confirms that the marker coordinate value for the entire image has changed. As a result, it is judged that the drawer is moving again. For example, point 'C' in FIG. 42 is the point where the coordinate value of the marker changes from '90' to '70', and the size of the coordinate value changes in the direction of shrinking to indicate that the drawer has just started to close. It can be determined immediately after the user completes a task for the drawer.

In the same way, when the coordinate value of the marker is '0', that is, the marker is not recognized from the image, it is determined that the drawer is completely closed.

From the image captured by the camera, the control unit identifies the location of the marker by the coordinates of one whole picture, and determines whether the drawer moves and stops, the drawer closes and opens, the drawer opens, and the drawer starts to open. We explained how to track and determine whether the drawer is starting to close.

Hereinafter, a method of processing a camera image according to a marker position of a drawer and obtaining a desired image will be described. A detailed method of how to process and acquire an internal image of the drawer from the movement state of the drawer, in particular, a method of obtaining an image at a point in time when it is determined that the user finishes working on the drawer will be described.

When it is determined that the drawer's marker coordinates change to '0' or more and it is the process of moving in the drawer opening direction, the controller detects only the change in coordinates of the camera's image and sends it to the controller, and displays the entire picture. It buffers in a temporary buffer and does nothing for each storage area partitioned into pictures.

When the drawer continues to open and reaches a fully open state or is opened to a certain extent and becomes a stop state, a situation in which the user performs work on the drawer proceeds. The controller determines this state from the fact that the coordinates of the marker do not change, and from this time, the image corresponding to the drawer area is moved to the drawer temporary buffer in the storage area partitioned from the entire image for the transmitted image and stored. do. That is, in the section corresponding to point 'D' in the above figure, the control section cuts out the necessary part from the part partitioned into the drawer area among the entire photos in the refrigerator and moves it to the temporary buffer for the drawer area, which is then moved to the 'drawer internal image refresh section'. ' is defined as

In other words, the image processing method is different between the period in which the drawer is open and the period in which the drawer is in a stopped state. The reason is that the user's action on the drawer reflects that the drawer is in a stopped state. As a result, in the section where the drawer is moving, the image processing time for tracking the coordinate change can be increased that much, so that the efficiency of data processing can be ensured.

On the other hand, when the drawer starts to close again from the stopped state, the control unit determines that it is right after the user completes the task from the coordinate value changing to a smaller value, and at that point in the temporary buffer for the drawer's internal image. Among the images temporarily buffered, the most recent image is selected and stored. That is, an image of the inner area of the drawer immediately after the user completes the task is captured.

The captured (stored) image of the storage state of the most recent drawer area replaces the image of the corresponding area of the display online or directly connected to the control unit, or is transmitted to the network server. In addition, the latest image transmitted to and stored in the network server is provided so that the user can check the latest image whenever desired through a display installed in the refrigerator or a mobile terminal connected to the network server.

Next, a method of searching for the position of the marker by selecting a specific section instead of searching for the position of the marker for the entire picture taken by the drawer will be described.

That is, it is possible to set a search range for the drawer area. A method of tracking the marker in each section may be different by classifying the degree of opening of the drawer by section on the movement path of the drawer. In this way, it is possible to shorten the processing (analysis) time of the photo taken by the camera and transmitted, matching the image processing speed with the camera shooting speed.

In general, in a drawer used in a refrigerator, even when the drawer is completely opened, the entire drawer cannot be pulled out to such an extent that the drawer is separated from the storage compartment and falls to the outside. That is, it is common that about 50% of the drawer's total depth (the drawer's front and rear length in the drawer's moving direction) is normally drawn out.

The draw-out distance of the drawer may be determined as an area to which the coordinates of the drawer marker should be assigned in the entire image processed by the controller. In this case, the control unit can determine the location of the drawer from the time the drawer is withdrawn. In particular, in order to detect a change in coordinates of the marker of the drawer, the size corresponding to the draw-out distance of the drawer by the control unit becomes the size of data to be processed. In addition, in order to detect a change in coordinates, the control unit must simultaneously compare and determine two images, and the processing time of this process must be able to correspond to the shooting speed of the camera.

It is preferable that the speed at which the camera captures the inside of the chamber (the time interval between taking pictures) is set to approximately match the speed at which the control unit judges and processes the transmitted image.

If this method is followed, it is possible to determine the position of the marker without delay with respect to the picture taken by the camera, and eventually to determine the position of the drawer. In addition, there is no difficulty in converting and processing new images continuously sent from the camera while processing images for each region divided into each storage space from the entire picture of the inside of the refrigerator. That is, from the entire image, depending on whether the door is open or the drawer is open, the image for each area is cut and moved to a temporary buffer for each area. All images captured by the camera are processed without missing. It will do.

However, since data for two images (previous image and current image) must be processed as much as the size described above, the processing power of the control unit must increase. If the processing capacity of the control unit is increased according to these conditions, the unit cost of the processor may increase rapidly. there is. In addition, for image processing when the drawer is not opened, a problem arises in that efficient part selection is not made because the processing capacity is excessively large.

In order to solve this problem, the inventors of the present invention focused on the fact that it is substantially difficult to sufficiently grasp the inside of the drawer when the draw-out distance of the drawer is not completely open, that is, when the drawer is opened by a certain distance, Provided is a method of different data processing by separating an image corresponding to a drawer area in stages according to a drawing distance of the drawer.

43 shows the extent to which the drawer is opened. 43A shows the drawer fully open, that is, about 50% of the drawer length is open, FIG. 43B shows the drawer about 30% of the total length open, and FIG. 43C shows the drawer fully open. About 10% of its length is open.

For example, as shown in FIG. 43c, in a state in which the user opens the drawer only about 10% of the total length of the drawer in the front-back direction, it is difficult to fully grasp the items stored inside the drawer, and FIG. 43b It is only when about 30% of the drawer is opened that the state of storage of items inside the drawer can be grasped to some extent, and this state is enough to be recognized as an open state.

As shown in FIG. 43A, when the drawer is opened to the maximum, about 50% of the total is withdrawn. In other words, when the drawer is in an open state of about 30% to 50%, the user can store or take out items from the drawer, and this section can be regarded as a substantially open state of the drawer.

Focusing on this point, if the drawer withdrawal distance is less than 30%, it is judged as 'the drawer is not open', and if it is more than 30%, it is classified as 'the drawer is open', and the image processing is different, so that the control unit The data load to be processed can be significantly reduced.

The control unit converts the image captured by the camera into an image to determine the image size corresponding to the drawer movement trajectory, and determines the pixel size from the drawer movement area to the point or distance at which the drawer is pulled out by about 30% to determine the image section to compartmentalize

That is, when the drawer is drawn out from the range corresponding to FIGS. 43A to 43B, the control unit can determine the movement of the coordinates of the marker. In this case, the load of the control unit required is reduced compared to the case of determining the movement of the coordinates of the marker from the range corresponding to FIGS. 43A to 43C.

In the section where the drawer is drawn out by 30% in the closed state, it is determined whether or not the position of the marker has reached the pixel position corresponding to the distance corresponding to 30%. In other words, without comparing the marker coordinates of the previous picture and the marker coordinates of the current picture (a prerequisite for determining the change in marker position), it simply determines whether the marker position is within a certain pixel range and temporarily buffers the entire image. will do

In this section, since only one photo data is processed without the need to compare two photo data, the data processing capability of the controller does not need to be high, and the camera can smoothly respond to the shooting speed.

On the other hand, when the drawer is opened beyond the 30% level, that is, when the marker reaches a position of a specific pixel or more in the picture processed by the controller, two pictures (taken immediately before It is possible to determine the coordinate value of the marker by comparing the photograph taken after the previous image and the photograph taken after that.

If the search section is limited in this way, the size of the photo becomes about 2/5 the size of the case where the search section is not limited. Furthermore, since the size of the picture corresponding to the two pictures to be processed is about 4/5 of that of one picture in the case where it is not, the data to be processed in the control unit is smaller than the entire picture or substantially the same. level can be

Therefore, since the data processing speed for tracking the coordinate change value of the two pictures can correspond to the speed at which the camera shoots, it is possible to process data (coordinate tracking or monitoring) without missing all of the images captured by the camera. Even in a state in which the processing capacity of the image is not increased any more, it is possible to prevent the time lag in the processing speed, so that a desired image can be obtained at an accurate time point.

In conclusion, as shown in FIG. 44 , it is possible to divide a part of the distance from which the drawer can be retrieved from the picture taken by the camera into a first section and a second section.

In the first section, it is possible to determine the position of a marker for a pixel of a photographed picture. At this time, in the first section, it is possible to determine at which pixel position the marker is located in one picture without comparing the coordinate values of the marker.

Meanwhile, in the first section, even if the drawer is actually opened, it is difficult for the user to withdraw or bring in items into the drawer, so it is possible to determine that the drawer is substantially closed.

The second section may be set as a 'search range' for comparing coordinate values of markers shown in two consecutively photographed pictures. In the second section, since the two pictures are compared, the coordinate change of the marker can be detected. For example, if the marker is at the same position in two consecutive pictures, it may be determined that the drawer is stopped during the time when the two pictures are taken. If the position of the marker is changed in two consecutive pictures, it can be determined that the drawer has moved during the time when the two pictures are taken. In this case, it may be determined whether the drawer is moved in the closing direction or the drawer opening direction according to the position change direction of the marker.

In the second section, since an opening through which a user withdraws or carries an article into the drawer is exposed to the user, it is possible to determine that the drawer is substantially open.

45 is a control flow chart according to an embodiment of the present invention. It will be described with reference to FIG. 45 below.

It is determined that the door 20 opens the storage compartment 22 (S10). At this time, the opening and closing of the door 20 may be detected by the door switch 110 .

When the door 20 is opened, the door sensor 120 may be driven. That is, the door sensor 120 can detect whether the door 20 is rotated more than a predetermined angle θ. That is, the door sensor 120 may not operate when the door 20 is not opened, but may start to operate when the door 20 is opened.

Also, when the door 20 is opened, the camera 70 is operated to start taking pictures (S14). At this time, the camera 70 can take pictures at predetermined time intervals, for example, 10 pictures per second.

The picture taken by the camera 70 is stored in the storage unit 18 (S16). At this time, if a plurality of photos are taken by the camera 70, the recently taken photo is stored, and the previously taken photo may be deleted due to the capacity limitation of the storage unit 18.

Meanwhile, it is possible to detect whether the drawer 50 is opened by the drawer detecting unit 130 (S20). The drawer detecting unit 130 may detect that the drawer 50 is pulled out from the lower space of the shelf 40 .

When the drawer 50 is detected by the drawer detector 130 to be closed, a photo taken at that time or a photo taken closest to that time is selected as the final photo (drawer open state) ( S40). Since the camera 70 takes a predetermined number of pictures per second, it is possible that a picture may not be accurately taken at the point when the drawer 50 starts to close. However, if the drawer detecting unit 130 determines the movement of the drawer 50 by analyzing the movement of the marker expressed in the picture taken by the camera 70, the picture is taken at the time when it is determined that the drawer 50 is closed. it is possible to choose

The controller 100 divides the final picture into the first area 42, the second area 52, and the third area 62 (S42). At this time, the control unit 100 can identify each area by specifying the number of pixels based on the control line 15, and obtain a final image for each area.

This is because, as described above, when the drawer 50 is withdrawn, the second area 52 and the third area 62 overlap each other.

Meanwhile, since information that the drawer 50 has been withdrawn is obtained in S20, the controller 100 may determine that the second image has been obtained.

And information on the second area 52 can be updated (S46). The controller 100 may transmit the second image showing the second area 52 to the display 14 and change the corresponding part based on the final picture.

On the other hand, if it is determined in S30 that the drawer 50 does not start to close, a picture can be taken again as in S14.

A state in which the drawer 50 is not closed in S30 can be regarded as that the user has not terminated access to the second area 52 . That is, the user can store new food in the second area 52 or take out the stored food from the second area 52 while the drawer 50 is stopped.

On the other hand, when the drawer 50 is closed, it can be predicted that the user ends access to the second area 52 and draws the drawer 50 into the lower space of the shelf 40 .

Meanwhile, in a state in which the drawer 50 is not opened in S20, it may be determined whether the door 20 is about to be closed (S60).

At this time, whether or not the door 20 is about to be closed may be detected by the door sensor 120 .

When the user ends access to the storage compartment 22 , the door 20 is closed and the storage compartment 22 is sealed. That is, the user can store new food in the storage compartment 22 or take out the stored food from the storage compartment 22 while the door 20 opens the opening 14 .

If the user closes the door 20, it may be determined that the stored food is maintained in that state until the user terminates access to the storage compartment and the door 20 is opened again.

If it is determined in S60 that the door 20 is about to be closed, the control unit 100 selects a photo taken at that time or a photo taken closest to that time as the final photo (the drawer is closed) (S70). That is, the control unit 100 may select a photo taken at a point in time when the photo-taking by the camera 70 ends or a point in time adjacent to the point in time as the final photo.

The controller 100 divides the final picture into the first area 42, the second area 52, and the third area 62 (S72). In this case, the final picture may include a first image showing the first region 42 . This is because the latest state of the food stored in the first area 42 is a state in which the user has terminated access to the storage compartment 22 . Therefore, the user can acquire accurate food information about the first area 42 through the first image obtained at this time.

Meanwhile, since information that the drawer 50 has been drawn in is obtained in S60, the control unit can determine that the first image and the third image have been obtained. As described above, when the drawer 50 is withdrawn, the second area 52 and the third area 62 overlap each other, but the first area and the third area do not overlap each other. It's because it's a location where you can shoot together.

And information on the first area 42 and the third area 62 can be updated (S76). The control unit 100 transmits the first image showing the first area 42 and the third image showing the third area 62 to the display 14, and based on the final picture. can be changed to

Meanwhile, although not shown in the drawings, when the door 20 is rotated to seal the storage compartment 22 and the storage compartment 22 is closed, it is possible to stop driving the camera 70 . At this time, the door switch 110 can detect whether the door 20 closes the storage compartment 22 .

That is, in the present invention, taking into account the time when the user ends use, processing the photo obtained at the time, and providing the user with information about the food stored in the corresponding area. Accordingly, the user can obtain accurate information about the food stored in the storage compartment 22 .

Meanwhile, two of the door sensor 120, the door switch 110, and the drawer detecting unit 130 may be installed on the left and right sides, respectively. In this case, individual information on the left and right doors and left and right drawers can be obtained. Of course, if the drawer detection unit 130 detects the movement of the drawer through the analysis of markers in the photographed picture, the drawer detection unit 130 is only one, whereas the markers are installed in two drawers, respectively. it is possible to become

Therefore, based on this information, when only the left door is opened, it is possible to update only the left part of one picture. On the other hand, if only the right door is opened, it is possible to update only the right part of one picture.

Of course, when the left door and the right door are opened together, it is also possible to update the left and right parts of one picture together and provide them to the display 14, provided that the above conditions are satisfied.

FIG. 46 is a control flow chart explaining a modified example of FIG. 45 . It will be described with reference to FIG. 46 below.

In the modified example of FIG. 46, S60 is specifically divided into S62 and S64, and only the difference will be described in detail.

That is, whether or not the door 20 is about to be closed can be specifically determined by determining whether the door 20 is rotated in the closing direction (S62) and whether the door 20 is rotated at a predetermined angle (θ) or more. there is.

In other words, when the door 20 is rotated in the closing direction (S60) and the door is rotated less than a certain angle (S64), the picture taking by the camera 70 can be ended.

In order to obtain information about the food stored in the storage compartment 22 from a picture taken by the camera 70, interference of the door 20 and a basket installed in the door 20 must not occur. Accordingly, the expression that the door 20 is about to close may also mean a state in which the door 20 and other components are not expressed in a picture taken by the camera 70 .

FIG. 47 is a control flowchart illustrating another modified example of FIG. 46 . Hereinafter, a description will be made with reference to FIG. 47, but only differences from FIG. 46 will be described.

In another modified example according to FIG. 47, unlike FIG. 46, the direction of rotation of the door 20 is not determined.

If it is determined in S12 that the door is rotated at a predetermined angle or more, the door 20 is rotated in the opening direction at the corresponding moment. Therefore, when the user rotates the door 20 again and the door 20 reaches a state where the door 20 is rotated by a predetermined angle, it may correspond to a case where the user closes the door 20 .

Therefore, in another modification, if S62 in FIG. 46 is omitted and it is determined that the drawer is not opened in S20, S64 for determining whether the door has been rotated at a predetermined angle or less is immediately performed.

FIG. 48 is a control flowchart illustrating another modified example of FIG. 45 . It will be described with reference to FIG. 48 below.

48 describes a control flow for detecting the movement of the drawer 50 using a marker. For convenience of description, overlapping parts are omitted, and only differences are described.

Overall, S20 and S30 can be specifically divided into steps S22, S24, and S26.

After the picture is stored in the storage unit 18 as in S16, it is determined whether the marker is recognized by the camera 70 (S22).

Since the camera 70 takes pictures from the upper side of the inner case 12 to the lower side, a case where the marker is not visible in the camera 70 may occur. That is, when the handle portion of the drawer 50 is covered at the front end of the shelf 40, that is, when the drawer 50 is not withdrawn at all, the marker may not be recognized.

If the marker is recognized in the picture taken by the camera 70, it can be determined whether the marker is still (S24). At this time, as described above, it is also possible to determine the movement of the marker only in the second section by identifying the moving path of the marker in one picture and dividing it into the first section and the second section.

Since the camera 70 takes a plurality of pictures at predetermined time intervals, it is possible to determine whether the marker has moved by comparing and analyzing the corresponding pictures. Since the marker is displayed on the drawer 50, it can be determined that the drawer 50 is also moved when the marker is moved.

In addition, it may be determined whether the marker has moved a certain distance or more from the position where the drawer 50 is closed (S26).

If the marker does not move more than a certain distance, an appropriate image of the second area 52 cannot be obtained because the drawer 50 is not sufficiently opened. For example, in a photo taken with only 1/3 of the drawer 50 drawn out, the inner space of the drawer 50 is not sufficiently shown, and the image obtained from the photo shows the user the second area. This is because information on (52) cannot be obtained. At this time, the predetermined distance can be determined by whether the marker has entered a search range for determining the movement of the marker.

When it is determined that the marker has moved by a certain distance or more, it can be considered that the user has terminated access to the second area 52 and has finished using the second area 52 .

Accordingly, a photo obtained at a time when the three conditions are satisfied or at a time adjacent to the time may include information about food stored in the second region 52 .

FIG. 49 is a view showing a modified example of FIG. 45 differently. It will be described with reference to FIG. 49 below.

Unlike FIG. 45 , in FIG. 49 , there is a difference in that the camera is not immediately driven when the door 20 is opened in the door switch 110 .

When the door switch 110 determines that the door 20 is open, the door sensor 120 is driven.

Then, the door sensor 120 determines whether the door 20 is rotated at a predetermined angle or more (S12). At this time, in a state in which the door 20 is not rotated more than a certain angle, the camera 70 does not take a picture.

When the door 20 is rotated over a certain angle, the camera 70 takes a picture (S14). At this time, the camera 70 can take pictures at predetermined time intervals, for example, 10 pictures per second.

The photograph taken at this time includes a first image of the first region 42 . On the other hand, since the door 20 is rotated over a certain angle θ, interference of a basket installed on the door 20 may not occur in the picture. That is, the image of the basket installed on the door 20 may not appear in the photo.

Furthermore, in S60 , the door sensor 120 may detect whether the door 20 is about to be closed. At this time, the meaning that the door 20 is just before closing may mean a state in which the door 20 is rotated at a predetermined angle or less while the door 20 is rotated in the closing direction.

The camera 70 may take pictures until right before the door 20 is closed, and may stop taking pictures (S49). In FIG. 45 , the camera 70 takes pictures until the moment when the door switch 110 is pressed after the door 20 is closed. However, in a modified example, when the door 20 is rotated to a certain angle by the door sensor 120, the camera 70 stops taking pictures. In this case, the constant angle may refer to an angle that prevents interference caused by the door 20 or a basket installed in the door 20 from occurring in the picture taken by the camera 70 .

50 is a control flowchart according to another embodiment of the present invention. It will be described with reference to FIG. 50 below. The control flow of FIG. 50 may continue to be performed at predetermined time intervals when it is determined that the door 20 is opened by the door switch 110 .

First, it is determined by the door sensor 120 whether the door 20 is opened more than a certain angle θ (S100).

When the door 20 is opened at a certain angle or more, it is determined that no interference occurs with the door 20, and the camera 70 takes a picture (S110). At this time, the camera 70 can take one picture.

Then, it is determined whether the drawer 50 has been withdrawn beyond a first set distance (S120). In this case, the first set distance may mean a distance at which food information on the second area 52 can be sufficiently acquired when a picture is taken by the camera 70 . For example, the first set distance may mean 2/3 or 1/2 of the total length of the drawer 50 .

The withdrawal distance of the drawer 50 may be detected by the drawer detection unit 130 .

On the other hand, if it is determined that the drawer 50 has been withdrawn beyond the first set distance, the controller 100 can divide the photo into a plurality of images by dividing the photo into regions (S122).

Since the picture is obtained when the drawer 50 is withdrawn, the first image and the second image may be obtained from the corresponding picture.

The controller 100 may update the first image and the second image (S124).

On the other hand, if it is determined in S120 that the drawer 50 has not been withdrawn beyond the first set distance, it can be determined whether the drawer 50 has been withdrawn below a second set distance (S130).

In this case, the second set distance may mean a distance at which food information on the third area 62 can be sufficiently obtained when a picture is taken by the camera 70 . For example, the second set distance may also mean 1/3 of the total length of the drawer 50 . Of course, it is also possible to select a state in which the drawer 50 is completely retracted into the lower space of the shelf by selecting '0' for the second set distance.

On the other hand, if it is determined that the drawer 50 has been withdrawn below the second set distance, the control unit 100 can divide the picture into a plurality of images by dividing the picture into regions (S132).

Since the picture is obtained when the drawer 50 is withdrawn, the first image and the third image may be obtained from the corresponding picture.

The controller 100 may update the first image and the third image (S134).

In another embodiment of the present invention according to FIG. 50 , the control is repeated when the door 20 is opened, and completely ends when the door 20 closes the storage compartment 22 . In the finished state, since the image is no longer updated, the final image of the food stored in the storage compartment 22 can be obtained.

51 is a diagram explaining a process of updating an image in a state in which two doors and two drawers are provided to open and close the storage compartment. It will be described with reference to FIG. 51 below.

Contents overlapping in the above-described embodiment will be briefly described.

When electricity is supplied to the refrigerator, it may be individually determined whether the left door or the right door is opened by the door switches 110 installed on the left and right sides respectively (S302 and S303).

When the door switch 110 determines that either the left door or the right door is open (S303), the camera 70 starts taking pictures (S304).

The camera 70 may take a picture even if either one of the left door and the right door is opened, and even if the left door and the right door are opened at a time interval or both are opened at the same time. This is because, when it is determined that the door is opened through one of the door switches, the light inside the refrigerator is turned on.

When the drawer disposed on the left is drawn out more than 130 mm (S310), an image related to the left drawer is captured (S312). The image related to the left drawer may refer to a part corresponding to the left drawer among the captured photos, or may refer to the entire captured photos.

In this case, 130 mm may be changed by the refrigerator manufacturer or the user, but may mean a withdrawal distance that allows the user to withdraw the drawer and access the storage space inside the drawer.

If the left drawer is drawn out less than 130 mm (S310), the previous image captured by the camera 70 is maintained (S310).

The left door sensor detects the opening angle of the left door. At this time, if the left door is opened at an angle smaller than 80 degrees, an image of the left multi-storage space provided in the lower space with the left drawer drawn out is captured (S332).

In this case, 80 degrees may mean an angle at which the left door and various structures installed in the left door do not interfere when the camera 70 captures the storage space. Depending on the specific structure or size of the refrigerator, it is possible to change 80 degrees.

If the left door is opened at an angle greater than 80 degrees, it is possible to retain the previous image captured by the camera 70 (S336).

And it is possible that the door switch 110 detects that the left door is closed (S334).

Meanwhile, for the right drawer and the right door, it is possible to implement the same control flow as that described for the left drawer and the left door. That is, since the above contents are equally applied to S320 to S324 and S340 to S346, a detailed description thereof will be omitted.

When the left door and the right door are closed, it is possible to process the image captured above in the controller 100 and display the processed image on the display 14 (S350). In this case, the display may be a display installed in the refrigerator.

And the camera can stop taking pictures (S352).

Meanwhile, the refrigerator is connected to wifi and determines whether communication is possible (S354), and if communication is possible, it can transmit an image to an external server (S356).

The server determines whether the application of the terminal used by the user is synchronized (S358), and if synchronized, it is possible to display the image on the display 14 of the terminal.

Accordingly, the user may be provided with information on items stored in the refrigerator through the user terminal even if the user is not in front of the refrigerator.

If the refrigerator is not connected to Wi-Fi and cannot communicate with the outside, or if it is not synchronized with the application of the terminal, it is preferable to maintain the previous image on the display 14 of the user terminal and provide the previous image to the user. do.

Hereinafter, components provided to take a picture and obtain a picture at a desired time point will be schematically described.

When the door switch detects that the user starts to open the door, the control unit drives the camera to start taking pictures of the inside of the refrigerator. The camera takes pictures continuously at a certain frame per second. Since the act of opening the door of the user means to take out a desired item from the refrigerator or to store a new item in the refrigerator, by continuously taking pictures of the inside of the refrigerator, You can monitor the state of the storage space for changes.

The storage compartment is separated into various shelves, drawers, baskets, etc. to form storage spaces, and items or containers are stored and stored in the storage space according to the shape or content.

The camera can monitor the storage space by dividing it into a shelf area divided by shelves, a drawer area, and other storage areas. Through this, it is intended that the user can check the storage state of the goods in the area immediately after the user completes the withdrawal or storage operation. If two drawers are provided and two other storage areas are provided in parallel on both sides, information enabling the user to check each of the two drawers and the two other storage areas can be provided through the camera. can

One embodiment of the present invention is to define the other storage area as a storage space formed on the floor of the storage compartment, located in front of the drawer storage area, and at least partially overlapping when the drawer is opened, and including it in the monitoring target. present.

When the door switch detects the opening of the door, the control unit turns on a door sensor included in the hinge assembly of the refrigerator to detect movement according to rotation of the door. It is possible that the door sensor is always turned on, but it is also possible that the door sensor is turned on only when the door is opened by the door switch.

When the door is opened at a certain angle or more, the control unit determines that the door is substantially open to the extent that the user can approach the storage room and carry in or take out goods, and determines the storage state of the shelf area as an object to be upgraded.

In addition, the drawer detecting unit detects the degree of opening of the drawer, and when the drawer is open, the storage state of the drawer area is determined as a subject to be upgraded, and when the drawer is not opened, the storage area is determined as a subject to be upgraded. to decide

The door sensor according to an embodiment of the present invention senses when a certain angle has elapsed in the process of opening and closing the door, and sends related information to the controller. Here, the door sensor can also detect the rotation direction of the door, that is, whether it is a closing direction or an opening direction.

In the embodiment of the present invention, when the on signal of the door switch is detected, the control unit turns on the door sensor, and the door sensor detects when the door passes a certain angle and sends the corresponding time point to the control unit. it becomes At this time, the control unit determines that the door is open. Specifically, at this time, the door is rotated by a predetermined angle and is in an open state.

On the other hand, when the user finishes taking out or putting in goods while the door is open, the open door is closed, and when the door passes the door sensor, the door sensor sends a signal to the control unit, and the control unit sends a signal to the control unit to send the user the item. It is judged that the door is about to close after the warehousing and warehousing of the door. Specifically, at this time, the door is rotated by a certain angle to be opened, but is rotated to be closed.

A process in which an image of the inside of the refrigerator is captured by a camera in the process of opening and closing the door and upgraded to the captured image immediately after the user finishes entering and leaving the goods will be described in detail with reference to an embodiment of the present invention.

When the refrigerator door switch is turned on while the door is closed, the controller turns on the camera and door sensor. The camera may mean switching from a standby mode to a driving mode, and the door sensor may mean switching to a state capable of determining a door opening angle.

The camera may continuously photograph the interior of the refrigerator at a predetermined speed per predetermined time. The photographed image is sent to the controller, and the controller temporarily buffers the transmitted image as an image that includes the entire inside of the refrigerator.

The image (Picture (or photograph)) and image (Image) used in this specification may be used separately from each other.

Specifically, the image (Picture or photograph) means raw data transmitted immediately after being photographed by the camera, and the image (Image) is the raw data (picture or photograph) that is transmitted by the control unit. It may mean data that is buffered, corrected, or transmitted.

When the door switch is turned on and the door is opened at a certain angle so that the control unit determines that the door is open, the control unit determines which area of the entire image contains the storage state of the article according to the open state of the drawer. will decide

For example, when the drawer is in an open state, the control unit controls the area (shelf area) where the entire stored image is stored on the upper shelf of the refrigerator and the drawer storage area (drawer area) stored inside the drawer. It is judged to contain the storage condition of the goods. On the other hand, if it is determined that the drawer is closed, the controller controls the storage space interior region (storage area), which is an area where the entire stored image is formed on the refrigerator shelf area and the front end of the drawer and overlaps with the opening of the drawer. It is judged to contain the storage condition of the goods.

Here, the picture of the entire refrigerator storage space taken by the camera and transmitted is divided into three different storage spaces, the shelf area is always included, and the drawer area and the storage area are the drawer area. It is selectively included according to the open/closed state of the lower. The drawer area and the storage area do not coexist with each other in the picture.

Therefore, one entire picture transmitted from the camera includes at least two areas, one of which is a shelf area, and which storage area the remaining part corresponds to is determined by the open/closed state of the drawer.

Since the present invention is intended to convey to the user the recent arrival and departure status of goods by taking pictures of two different spaces including at least the inside of the drawer, such as the inner space of the drawer and another storage space, with a single camera, the drawer A method of upgrading by photographing only the area and the shelf area in a fixed manner may be another embodiment of the present invention, and a method of photographing only the drawer area and the storage space in a fixed manner may be an embodiment of the present invention. .

Meanwhile, dividing an entire picture of the storage compartment constituting one independent heat-insulated space into storage areas within the storage compartment may be determined by the number of doors that open and close the storage compartment. For example, in a refrigerator in which the storage compartment is opened and closed with one door, the storage area to be photographed can be divided into two or three types. When the storage compartment is opened and closed with two left and right doors, the storage area is It can be divided into 4 or 6 categories.

In this case, the drawer area and the storage area may be partitioned into left and right sides and managed as two partitioned areas in the entire picture, and the shelf area may be integrated and managed as one area. In this case, all of the divided parts in the entire picture may be composed of five types.

In this shelf area, the storage spaces facing the left and right doors are connected to each other. In the shelf area, there may also be cases in which articles, food, or vegetables are stored across left and right sides. Since items can be taken out or stored by opening only one of the left and right doors when stored across the left and right doors, even if only one of the left and right doors is opened, the entire shelf area from the entire image taken is integrated and replaced with the latest data. because there is a need Considering this point, even if only one of the left and right doors is opened, the entire shelf area can be updated.

More specifically, if it is determined that the drawer is open while the door is open, the control unit separates the shelf area and the drawer area from all transmitted photos and temporarily buffers them. In addition, when the user finishes the act of putting in and out of goods in the drawer area and/or the shelf area and tries to close the drawer, that is, when the drawer starts to move in a stopped state after opening, the most recent A necessary part in the drawer area is captured (captured or cut) from the image captured in the picture and stored in the storage unit as an image of the recent storage state of the drawer area. In addition, the captured image of the drawer area in the refrigerator interior image displayed on the refrigerator display is replaced with a recent image and upgraded. In addition, the control unit transmits the captured image of the drawer area to a server system connected through a network.

Meanwhile, if the drawer is not detected when the drawer is open while the door is open, and it is determined that the drawer is closed, the controller determines that the entire picture of the transmitted image includes the shelf area and storage area. Therefore, the shelf area and the storage area are separated from the entire picture and temporarily buffered.

When the user completes the work of putting in and out of the storage area and/or the shelf area and closing the door, the door sensor detects when the door passes a designated angle and transmits a signal to the control unit. The control unit determines that the door is just before closing (it can be expressed as the picture is just before being covered by an obstacle such as a door) by the signal transmitted from the door sensor, and stores the most recent captured image among temporarily buffered images. A necessary part within the area is captured and stored in the memory as an image of a recent storage state of the storage area. At the same time, the captured image is replaced and upgraded for the storage area in the image inside the refrigerator displayed on the display.

At this time, upgrading may mean replacing an existing image with a newer image. In addition, the control unit transmits the captured image of the drawer area to a server system connected through a network.

Meanwhile, when the drawer is in an open or closed state, the shelf area captures the most recent image among buffered images at the time when it is determined that the door is just before closing, and is stored in the memory as the latest storage state of the shelf area.

The captured image (any one or both of the recent shelf area image, recent drawer area image, or recent storage area image) replaces the displayed image on the refrigerator display or is transmitted to the server system or connected to the refrigerator by communication The transmission to the user's mobile device may be performed simultaneously while storing the captured image in the refrigerator memory, or may be performed at once when the user requests it, or may be performed by the user's request signal for each area.

Here, one image transmitted from the camera may be buffered as one image by the controller. If it is determined that the one image is taken with the door open, it may be determined whether the one image includes the drawer area or the storage area according to the drawer open/closed state. When such a determination is made, the corresponding images are extracted and buffered in separate temporary buffers.

If it is determined that one image was taken when the drawer starts to close, the most recent image among the temporarily buffered images is stored in the memory, and when the user's new input/output operation is completed, it is replaced with the latest image. . Also, if it is determined that the door is about to be closed, the most recent image among the temporarily buffered storage area images is stored in the memory.

Meanwhile, for the shelf area, only the shelf area can be extracted from the most recent image immediately before the door is closed and stored in the memory, and a temporary buffer is prepared for the shelf area to extract and buffer only the shelf area. may be

Hereinafter, with reference to FIGS. 52 to 55 , various embodiments in which the most recent internal image of the refrigerator obtained by the various methods described above are transmitted to and utilized by a personal mobile terminal or an operator's server will be described.

52 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 52, when it is determined that the final picture of a specific area inside the refrigerator has been obtained through the steps described with reference to FIG. 2) is transmitted (S1103). The terminal 2 may include a smart phone, a personal digital assistant (PDA), a tablet PC (Tablet PC), and the like, but is not limited thereto. At this time. The controller 100 may transmit to the terminal 2 not only the final image of the inside of the refrigerator, but also various types of information related to the final image described below.

At this time, the control unit 100 may transmit the final image to the terminal 2 as soon as the final image is acquired through the final picture, and after storing the obtained final image in the storage unit 18, according to a predetermined cycle. An image stored in an event format may be transmitted to the terminal 2, but is not limited thereto.

Subsequently, the terminal 2 displays the received final image according to the user's selection (S1105). For example, the terminal 2 may execute an application capable of receiving and displaying an image of the inside of the refrigerator according to a user's selection, or may display the received image of the inside of the refrigerator by performing a multimedia service.

According to this embodiment of the present invention, since the user can check the inside image of the refrigerator using a mobile terminal even without directly looking into the inside of the refrigerator, there is an advantage in that the current situation inside the refrigerator can be remotely grasped.

53 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 53 , as described above, when it is determined that a final photo of a specific region inside the refrigerator has been acquired (S1301), the controller 100 classifies the final photo by region and generates an image, and then the acquired final image is It is controlled to be stored in the storage unit 18 (S1303).

Subsequently, when the terminal 2 receives a user input for displaying the current internal image of the refrigerator, that is, the final image of the specific region of the refrigerator (S1305), the terminal 2 transmits a final image request command to the refrigerator (S1307).

In addition, the control unit 100 transmits the most recently taken and stored refrigerator interior image among the images stored in the storage unit 18 to the terminal 2 as a response to the final image request command (request), the communication unit ( 270) is controlled (S1309).

According to such an embodiment of the present invention, communication between the mobile terminal and the refrigerator is performed only when the user needs it, so that it can be more efficient and economical. Hereinafter, only a case in which an image or the like is transmitted as a response to a request will be described, but a case in which an image or the like is transmitted in an event format as described with reference to FIG. 35 is also included, and is limited thereto. it is not going to be

54 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 54 , as described above, when it is determined that a final picture of a specific area inside the refrigerator has been obtained (S1501), the controller 100 controls the final image acquired from the final picture to be stored in the storage unit 18. (S1503).

Subsequently, when the operator server such as the market server 3 receives a user input for displaying the current internal image of the refrigerator (S1505), the market server 3 transmits a final image request command to the refrigerator (S1507) . At this time, the market server 3 may transmit a final image request command to the refrigerator according to a predetermined cycle without user input.

In response to the image request command, the controller 100 transmits the most recently photographed and stored final image of the inside of the refrigerator among the images stored in the storage unit 18 to the market server 3 ( S1509).

The market server 3 receiving the final image inside the refrigerator analyzes the final image (S1511). The market server 3 analyzes the final image, finds out what is currently stored in the refrigerator and what is not currently stored, and analyzes what was stored in the refrigerator in the past but is not currently stored. Information necessary for service provision can be extracted.

The market server 3 transmits the list of insufficient items extracted through final image analysis to the corresponding refrigerator (S1513).

55 is a ladder diagram for explaining a method of operating a refrigerator according to another embodiment of the present invention.

Referring to FIG. 55 , as described above, when it is determined that a final picture of a specific area inside the refrigerator has been acquired (S1701), the controller 100 controls the final image corrected from the final picture to be stored in the storage unit 18. (S1703).

Subsequently, when a business operator server such as the broadcasting station server 4 receives a user input for displaying the final image inside the refrigerator (S1705), the broadcasting station server 4 transmits a final image request command to the refrigerator (S1707) . At this time, the broadcasting station server 4 may transmit a final image request command to the refrigerator according to a predetermined cycle without user input.

In response to the final image request command, the controller 100 transmits the most recently photographed and stored final image of the inside of the refrigerator among the images stored in the storage unit 18 to the broadcasting station server 4. Control (S1709).

Upon receiving the final image, the broadcasting station server 4 analyzes the final image (S1711). The broadcasting station server 4 analyzes the final image, finds the items currently stored in the refrigerator, and extracts information necessary for providing services, such as analyzing dishes that can be made using the stored items.

The broadcasting station server 4 transmits, through final image analysis, recipe information of dishes that can be made using the currently stored storage to the corresponding refrigerator (S1713).

The control unit 100 displays, on the display 14 installed in the refrigerator, recipe information of a dish that can be made using the currently stored storage received from the broadcasting station server 4 (S1715).

According to such an embodiment of the present invention, there is a practical advantage in that a related business operator can accurately grasp the internal situation of a refrigerator located in a home or business and provide a refrigerator-related service suitable for the home or business.

56 is a diagram explaining the operation of the heater of the camera, and FIG. 57 is a diagram explaining the experimental results on condensation generated by temperature on the transparent window of the camera. It will be described with reference to FIG. 56 below. In FIG. 56, the heater of the form described in FIGS. 14 to 17 will be described as a reference.

56(a) is a graph in which power is intermittently supplied to the heater 84 and is intermittently operated, and FIG. 56(b) is a graph in which power is continuously supplied to the heater 84 and continuously operated. it's a graph

56(a) shows a method in which the temperature of the transparent window 80 is not normally controlled and maintained in an equilibrium state like the temperature of the storage compartment 22, and then high power is applied at a time when the refrigerator camera needs to be taken. In particular, condensation generated on the transparent window 80 may be removed by operating the heater 84 immediately before taking a picture.

As the door 20 is opened, the transparent window 80 may come into contact with moisture contained in the warm outside air coming in through the door 20 . At this time, since the temperature of the transparent window 80 is relatively lower than that of the outside air, moisture contained in the outside air is condensed in the transparent window 80, resulting in condensation.

If the heater 84 is instantaneously driven to remove the dew generated by this dew condensation, there may be a problem that it takes time until the dew on the transparent window 80 is actually removed. Therefore, if the time at which photography is required is earlier than the time at which dew is removed, a problem that dew may be formed on the transparent window 80 may occur.

On the other hand, in FIG. 56(b), the heater 84 is continuously driven so that heat is supplied to the transparent window 80. Since the temperature of the transparent window 80 is always maintained above the condensation generation temperature, condensation does not always occur on the transparent window 80 . Therefore, even when the door 20 is opened and the transparent window 80 contacts the outside air, moisture contained in the outside air does not form dew on the transparent window 80 . Therefore, it is possible to prevent dew from forming on the transparent window 80 when the camera 70 is to momentarily take a picture.

In driving the heater 84, it is necessary to prevent image quality problems caused by dew condensation and to prevent a sharp increase in power consumption of the refrigerator due to the power consumption of the heater 84. In addition, it is desirable to consider the fact that the internal temperature of the furnace should not be affected due to the heat supplied by the heater 84.

The present inventors believe that in order to minimize the power consumption of the heater and not affect the inside of the refrigerator, it is necessary to operate the heater with a long time to turn off the heater. However, if the time to turn off the heater is set for a long time, condensation will occur, so the target temperature is set on the transparent window of the camera, and the outside air temperature of the refrigerator is 32 degrees Celsius and the relative humidity is 85% (the dew point temperature is approximately 29 degrees to 30 degrees) was tested (see Fig. 57). In FIG. 57a, the y-axis means the amount of condensation generated on the transparent window, and in 57b, the y-axis means the percentage of the amount of condensation actually generated at the corresponding temperature compared to the current amount of water vapor.

The on/off operation cycle of the heater derived from repeated experiments could deteriorate the image quality of pictures taken by the camera under the special temperature conditions in the furnace.

Even if an appropriate on/off cycle of the heater is found, a user's refrigerator usage pattern may vary unpredictably during the time when the heater is turned off.

For example, when the user leaves the refrigerator door open for a long time, the camera cover glass (cover glass may mean the above-described transparent window or various types of windows placed in front of the camera lens) can), a large amount of condensation is formed on the surface. In this case, even if the heater is operated to heat the cover glass, since evaporation takes a long time, there is a problem in image quality if photographing is performed at that time. E In addition, it took a lot of time and a lot of power to evaporate already generated dew from the cover glass.

In addition, when the storage compartment is a refrigerating compartment, the temperature inside the refrigerator is usually maintained between 0 degrees Celsius and 7 degrees Celsius, and the internal volume is also larger than the surface area of the cover glass. A large portion of the heat supplied by the heater is inevitably leaked into the cold air inside the storage compartment. Therefore, in order to evaporate dew formed on the cover glass, a large amount of heat must be applied for a short period of time. In order to intensively supply such heat for a short period of time, a heater with a large capacity is used, and instantaneous power consumption greatly increases. Therefore, various problems may occur in a state in which power conditions are not good.

Therefore, it is preferable to prevent a certain amount of dew condensation from occurring on the cover glass continuously from the beginning, and in the present invention, power is applied so that the heater is operated at all times without an on/off cycle.

The method of constantly applying power to the heater may be configured so that the corresponding power is applied to the camera heater immediately when power is applied to the refrigerator, or may be controlled so that power is always applied to the heater during normal operation of the storage compartment through a refrigerator control unit. there is.

Next, the heating temperature of the cover glass will be described. As a result of quality evaluation based on the image samples obtained for each temperature range, it was confirmed that there was no significant problem in the quality of the sample images up to a temperature at which the temperature of the central portion of the cover glass was approximately ½ of the dew point temperature. The absence of a problem in image quality may mean the degree to which the type and quantity of items stored in the storage room can be identified through the photographed picture.

In particular, the quality of the image sample obtained at 13.5 degrees, the point (3) in FIG. 57B, was at an acceptable level. Of course, it is possible to maintain the heating temperature of the cover glass at a temperature higher than the dew point, but power may be supplied to the heater so that the heating temperature of the cover glass is maintained between about 13 degrees Celsius and about 30 degrees Celsius.

On the other hand, through the experimental results, the heating temperature of the cover glass differs depending on the outer size of the cover glass under the condition of supplying the same applied power, and there is no significant difference in the thickness of the cover glass. Therefore, since the image quality was satisfied at 1/2 of the dew point temperature in the same size of the cover glass, it is desirable to set this as the lower limit, but select an appropriate upper limit in terms of power consumption (applied power) of the heater.

When heating the cover glass, the applied power near the dew point temperature (29.2 degrees Celsius) was 0.45 W, and the applied power at about 0.58 times the dew point temperature (17.1 degrees Celsius) was 0.27 W. Therefore, in the range of 0.5 to 0.7 of the dew point temperature, since the power consumption is also approximately 0.5 times, it was able to see a great effect in reducing power consumption.

A mid-band of dew point temperatures can be determined to be approximately 12 to 17 degrees Celsius at ambient conditions, which can usually be varied. Therefore, it is preferable that a current value for maintaining the temperature of the central portion of the transparent window at approximately 12 to 17 degrees Celsius is applied to the heater.

That is, in the present invention, the effect of reducing the power consumed by the heater can be obtained by maintaining the temperature of the cover glass at a temperature lower than the dew point, not the dew point.

58 is a cross-sectional view of a transparent window. It will be described with reference to FIG. 58 below.

The transparent window 80 may be hydrophilic coated. At this time, the transparent window 80 may mean the above-described cover 90.

Even if dew is formed on the transparent window 80 instantaneously, as shown in FIG. 58, the contact angle α of water can be formed within 15 degrees due to the hydrophilic coating. When the hydrophilic coating is applied, the surface tension of water on the corresponding surface is reduced. This is because water can spread widely on the surface of the transparent window 80 . Accordingly, the camera may be fixed to the ceiling of the storage compartment so that the angle between the transparent window and the ceiling of the storage compartment is approximately 10 degrees to 20 degrees.

Therefore, it is possible to minimize distortion of the photographed picture due to water condensing on the transparent window 80 . In addition, the hydrophilic coating may have an advantage in terms of energy efficiency because it does not need to be controlled such as supplying additional electricity after it is once produced.

59 and 60 are diagrams schematically illustrating a structure in which a camera is installed in an inner case. For reference, FIGS. 59A and 60A are views looking up at the side of the storage compartment, and FIGS. 59B and 60B are views looking up at the inner case. It will be described with reference to FIGS. 59 and 60 below. Through these embodiments, it is possible to minimize the occurrence of condensation in the cover 90 by bypassing the path leading to the cover 90 by outside air.

The camera 70 may be installed on a protrusion 500 protruding downward from the ceiling of the inner case 12 . In this case, the protruding part 500 may mean a part protruding downward compared to other parts of the inner case 12 . This protrusion 500 may be implemented through the aforementioned camera housings.

At this time, as shown in FIG. 59 , the camera 70 is installed within the protrusion 500 so as to look at the inner wall of the storage chamber 22 . Through the cover 90 , the camera 70 may take a picture of the inside of the storage compartment 22 . Accordingly, the cover 90 is disposed at a position facing the rear wall of the storage compartment 22 from the protrusion 500 .

At this time, the moisture contained in the outside air contacts the cover 90 and is cooled to generate dew.

However, since the cover 90 is disposed so as to face the rear wall of the storage compartment 22, a path for outside air to reach the cover 90 may be lengthened.

That is, in order for the outside air passing through the door 20 to pass through the storage compartment 22 and contact the cover 90, it must go through a path as shown in FIG. 59B. Some of the moisture it contains may condense in other parts. Therefore, outside air may reach the cover 90 while the amount of moisture gradually decreases in the storage compartment 22 .

That is, even if the door 20 is open, the inside of the storage compartment 22 maintains a lower temperature than the outside of the storage compartment 22 and does not immediately change to the same temperature as the outside air, so that the outside air enters the storage compartment 22 and exchanges heat. this is done

In other words, if the cover 90 is disposed at a location where it is relatively difficult to contact the outside air, the amount of dew generated on the cover 90 may be reduced and the time point at which dew is generated may be delayed.

First, outside air comes into contact with the front surface 501 of the protrusion 500 while entering the inside of the refrigerator. At this time, outside air contacting the front surface 501 further enters the inside of the refrigerator while changing its path to the left and right of the protrusion. The outside air may reach the cover 90 while contacting the side surface 502 of the protrusion 500 . Thus, a large amount of moisture can be condensed before reaching the cover 90 .

In addition, outside air contacting the front surface 501 of the protrusion 500 may descend and rise along the lower surface 503 of the protrusion. Accordingly, a large amount of moisture may be condensed while moving to the front surface 501 and the lower surface 503 .

In addition, the cover 90 is mounted inclined in a form surrounded by the front surface 501, the side surface 502 and the bottom surface 503. Therefore, the path of the outside air and the cover 90 are not perpendicular to each other. Therefore, it can also be prevented that outside air reaches the cover 90 and is rapidly condensed.

In particular, the width of the protruding portion 500, particularly the left and right widths of the front surface 5010, may be wider than the width of the camera 70.

The embodiment shown in FIG. 60 is very similar to the embodiment shown in FIG. 59 . However, it can be said that it is different that the camera is mounted to look vertically downward. Accordingly, the same effect as the example shown in FIG. 59 can be obtained.

Also, the protrusion 500 may include a rear face 504 . Preferably, the vertical length of the front face 501 is greater than the vertical length of the rear face 504 . That is, it is possible that the front surface 501 protrudes more downward from the inner case 12 .

Therefore, in order for outside air to reach the cover 90, a path is lengthened in the horizontal direction by the width of the protruding portion 500, and a path is lengthened in the vertical direction by a difference in protruding height of the protruding portion 500. will lose

Prevention of condensation in the cover 90 through the structure of the protrusion 500 or the structure for mounting the camera is very related to the installation position of the camera. That is, this is because the camera can be installed in an angular range that is tilted backward at a predetermined angle from the vertical bottom of the ceiling of the storage room. Accordingly, it is possible to effectively detour the inflow path of outside air while effectively photographing the entire area within the storage compartment.

In the prior art, there are techniques for mounting the camera on the side wall of the body storage compartment, on the top of the outer body, on the door or on the top of the door.

In the prior art, when a camera is mounted on a side wall of a storage compartment, a recess for accommodating the camera is provided on the side wall, and the camera is mounted in the recess. That is, the camera is prevented from protruding into the storage compartment. It is presumed that this is a configuration for not causing inconvenience to the user when taking out a shelf or an article in the storage room.

However, in the present invention, since the camera is installed on the ceiling of the inner case 12, the user is not disturbed when using the storage compartment 22. This is because there is no reason for the camera to collide with the camera when the user wants to withdraw the product from the storage room, since the camera is not located in the moving path of the product or the shelf.

Furthermore, in the present invention, the cover 90 of the camera 70 is arranged to face the rear, so that the path of outside air reaching the cover 90 can be extended.

In FIG. 59A, while the cover 90 is arranged to look backward, the protruding part 500 protrudes downward, so that the outside air has a three-dimensional movement path to reach the cover 90, and the outside air moves. the path becomes longer.

In FIG. 60A, the cover 90 is disposed to face downward, but the front surface 504 disposed horizontally to the opening of the protruding portion 500, and the front surface 504 at both ends of the front surface 504. A side surface 508 disposed perpendicular to the surface is provided. At this time, the front surface 504 and the side surface 508 are installed so as to protrude downward from the protruding part 500, so that the path for outside air to reach the cover 90 can be lengthened.

That is, the high-temperature air introduced from the outside into the storage compartment opening does not directly reach the camera cover 90 in a straight line and does not come into contact with it, but bypasses left and right and downward along the protrusion 500 to reach it. Therefore, the outside air exchanges heat with the cold air inside the refrigerator to lower the temperature and reduce the amount of saturated water vapor. When it finally reaches the cover 90, since it has a reduced amount of water vapor, it can be sufficiently evaporated by the heater heating the cover 90.

Accordingly, the power to be supplied to the heater 84 is reduced, so that the power consumption of the entire refrigerator can be improved.

In the embodiment shown in FIG. 59 or 60, a body (a cabinet) with a storage compartment formed therein; The storage compartment has an access opening on the front side, a shelf area formed of an upper wall, a lower wall, side walls, and a rear wall made of an insulating material, and is partitioned by a plurality of shelves, and a drawer having at least one drawer. consists of an area; at least one door that is in close contact with the front surface of the storage compartment and opens and closes the storage compartment; a camera device installed on an upper wall of the storage compartment between an opening of the storage compartment and a front edge of the shelves installed in the storage compartment;

The camera device includes a camera module part accommodating electrical components necessary for driving a camera, including a camera lens, and having a camera window formed at a predetermined distance from the lens, and the camera module part. It is configured to include a camera housing part (a camera housing part) for seating and fixing the part to a predetermined position inside;

The camera housing includes a fixing surface in contact with the upper wall of the storage compartment, a front surface formed on an opening side of the storage compartment, side surfaces connecting the front surface and the rear surface, and the camera housing. The refrigerator includes a top surface on which an opening is formed to expose a camera window of the module unit, and the front surface of the camera housing unit is higher than the opening formed on the upper surface of the camera housing unit. can

An upper surface of the camera housing may be inclined at a certain angle toward an opposite direction of the opening of the storage compartment with respect to the upper wall surface of the storage compartment.

The camera housing may further form a rear surface opposite to the front surface at a rear wall side of the storage compartment, and the front surface may be higher than the rear surface.

The opening of the upper surface of the camera housing may be formed at a lower position than the side surface of the camera housing.

A housing installation part having a recessed portion corresponding to a recessed space of the upper wall of the storage compartment may be further included.

The housing mounting portion may be mounted on an upper wall of the storage compartment together with the refrigerator insulation material, and the camera housing portion may be fixed to the housing mounting portion.

The camera module unit fixed to the camera housing unit may be accommodated in a recess formed in the housing mounting unit.

The camera module unit accommodated in the concave portion of the housing mounting unit may be fixed to the camera housing unit to have a gap with an inner surface of the housing mounting unit.

Front and side surfaces of the camera housing may be formed higher than an opening of an upper surface of the camera housing.

In the upper surface of the camera housing, an opening side of the storage compartment may be formed farther from an upper wall surface of the storage compartment than a rear wall side of the storage compartment.

The camera module unit may have a front portion where a camera window is formed and a rear portion opposite to the front portion, and at least one flat surface may be formed on at least one of the front portion and the rear portion. can

A receiving portion for accommodating the camera module portion is formed inside the camera housing portion, and a receiving portion is provided with a flat surface formed in the camera module portion inclined at a certain angle with respect to the upper wall surface of the storage compartment. A seating part may be included.

The recessed part of the housing mounting part may include a seating part inclined at a predetermined angle with respect to the upper wall surface of the storage compartment to place a flat surface formed on the camera module part.

A flat surface may be formed around a camera window of a front portion of the camera module unit, and the flat surface may be formed to be substantially equal to a height of a central portion of the camera lens.

61 is a view showing another type of refrigerator to which the present invention can be applied. It will be described with reference to FIG. 61 below.

The refrigerator of FIG. 61 is a side-by-side refrigerator, and the left and right sides form different storage compartments. That is, the storage compartment on the left is a freezing compartment, the storage compartment on the right is a freezing compartment, and the storage compartment on the left is a refrigerating compartment.

A camera 70 may be installed on the ceiling of the storage compartment 22 so as to face the lower portion of the storage compartment 22 .

In addition, a drawer 50 capable of storing goods and withdrawing from them may be installed in the storage compartment 22 .

Other contents are omitted because they are the same as those of the refrigerator described above.

62 is a screen provided to the user from the refrigerator of FIG. 61 . It will be described with reference to FIG. 62 below.

A user may be provided with images of articles stored in the drawers 50 and 51 and images of articles stored on the upper side of the shelf 40 . In the refrigerator, one drawer 50 may be provided. Accordingly, an image of one drawer may be provided in FIG. 62 as well. However, the barwak kx shown in FIG. 61 may be provided with two drawers 50 and 51 vertically. In this case, it is also possible to provide an image of the inside of each drawer.

In the example described with reference to FIG. 2 , drawers are provided on the left and right sides respectively. However, in FIG. 62, drawers may be provided in the upper and lower directions. Accordingly, there may be two drawer areas, and the lower drawer 50 may protrude forward compared to the upper drawer 51 . In this case, the drawers 50 and 51 may also have markers similarly.

Through this, it is possible to easily grasp food information stored in a plurality of shelf areas and a plurality of drawer areas through a single camera.

The method of upgrading the image provided on the user's screen is omitted because it is the same as the refrigerator described above.

FIG. 63 is a diagram explaining a method of adjusting a picture taken by a camera in the refrigerator of FIG. 61 . It will be described with reference to FIG. 63 below.

Typically, since only one drawer is provided or disposed vertically, one leveling line 15 may be provided in the refrigerator of the type of FIG. 61 . Of course, two horizontal adjustment lines may be provided to separate the two drawers. In addition, it is also possible to adjust the position of the adjustment line by arranging two vertical adjustment lines arranged in parallel. Of course, unlike that shown in FIG. 63, it is also possible to select a desired image using only horizontal adjustment lines without using vertical adjustment lines.

A user may be provided with a limit line 16 indicating a limit range for moving the control line 15. At this time, the limit line 16 may be arranged in parallel with a predetermined interval.

Correction of the error through the adjustment line 15 can be said to be the same as or similar to that of the above-described embodiment.

The present invention is not limited to the above-described embodiments, and as can be seen from the appended claims, modifications can be made by those skilled in the art to which the present invention belongs, and such modifications fall within the scope of the present invention.

10: outer case 12: inner case
20: door 40: shelf
42: first area 50: drawer
52: second area 60: barrier
62: third area 70: camera
100: control unit

Claims (9)

pantry for storing food;
a refrigerator main body constituting a part of the storage compartment and including an opening of the storage compartment; and a door rotatably provided on the main body to open and close the opening of the storage compartment;
a camera for photographing the inside of the storage compartment; and
A main body forming the storage compartment includes a drawer and a shelf, and a door forming the storage compartment includes a basket;
A refrigerator comprising: a display that distinguishes and displays an image of an area stored in the shelf and an area other than the shelf in a single picture taken by the camera.
According to claim 1,
The display is a refrigerator that independently updates and displays images for each region.
According to claim 2,
The refrigerator displays an image of the shelf area and an image of the other area so as to be distinguished from each other.
According to claim 3,
A refrigerator comprising a display configured to distinguish between a frame allocated to display an image of the shelf area and a frame allocated to display an image of an area other than the shelf area.
According to claim 1,
The display is a refrigerator displaying two overlapping areas through one screen.
According to claim 5,
The refrigerator of claim 1 , wherein the two overlapping regions are spaces arranged to overlap each other in a state in which the drawer is drawn out.
According to claim 6,
The display displays an area in a state in which the drawer is withdrawn and an area in a state in which the drawer is not withdrawn with respect to the two overlapping areas on one screen.
According to claim 1,
The refrigerator according to claim 1 , wherein the display is a separate external device other than the refrigerator.
delete

Images