KR20240073841A - Refrigerator and Controlling Method for the same - Google Patents

Abstract

The present invention relates to a refrigerator and a method of controlling the same, and more specifically, to a refrigerator and a method of controlling the same that can provide information about food stored in the refrigerator even if the user does not open the door.
According to one embodiment of the present invention, a storage compartment is formed in the main body of the refrigerator by a fixed insulating wall and is provided with an opening; a door rotatably provided on the main body to open and close the opening; A drawer provided in the storage compartment; The ceiling of the storage room is photographed together with the area where food is stored in the outer space of the drawer in the storage room (first area) and the area where food is stored in the inner space of the drawer in the storage room (second area). a camera provided to be fixed to; And, through the shooting time of the photo in which the first area and the second area are taken together, the part about the first area (first area photo) and the part about the second area (second area photo) in the photo. A refrigerator may be provided that includes a control unit that separates and individually separates and stores the refrigerator.

Description

Refrigerator and its control method {Refrigerator and Controlling Method for the same}

The present invention relates to a refrigerator and a control method thereof, and more specifically, to a refrigerator and a control method that can provide information about food stored in the refrigerator even if the user does not open the door.

In general, a refrigerator is a device that supplies cold air generated by driving a refrigeration cycle and is a device for storing food at a low temperature. In the case of conventional refrigerators, they could simply perform the function of storing food at low temperature. However, in recent years, the need for additional functions in addition to the food storage function has increased.

A refrigerator is a device that stores and stores certain items, and the only way to check the inside is to open the refrigerator door. Additionally, when users try to buy something at a market or mart, if they do not know the amount and type of food stored in the refrigerator, there is the inconvenience of buying the same food repeatedly or not being able to buy the food they need.

According to the prior art, Japanese Patent Publication No. 3450907 and Patent Publication No. 2004-183987, a technology for photographing the interior of a refrigerator by installing a camera on the door is disclosed. Additionally, according to Japanese Patent Laid-Open No. 2001-294308, a technology in which cameras are installed inside the cabinet, inside the drawer, and on the door is disclosed.

However, according to the reviewed prior art, the shooting range of the camera installed inside the refrigerator is limited, so there is a problem that multiple cameras must be used to capture various storage areas.

Therefore, while reducing the number of cameras installed in the building, it is necessary to improve the shooting efficiency of the camera by increasing the range of the area captured by the camera.

Additionally, according to the prior art, there were various problems in obtaining images of the inside of the drawer. For example, there were no specific methods for obtaining photos taken at the desired time, concerns about condensation occurring in the camera, or concerns about improving power consumption by the camera.

Specifically, according to the prior art, when there are a plurality of storage areas including drawers in the storage unit, there is a problem of lack of specificity in providing optimal information to the user considering the specificity and positional relationship of each area.

The present invention basically aims to solve the above-mentioned problems.

In order to solve the above problems through embodiments of the present invention, the present invention provides a storage room display device for a refrigerator that provides information about food stored in the refrigerator even if the user does not open the refrigerator door, and a control method thereof. It is provided.

In addition, through embodiments of the present invention, it is intended to provide a refrigerator and a control method thereof that enable a user to intuitively recognize information about food stored in each of a plurality of storage areas in a storage room.

In addition, through embodiments of the present invention, a refrigerator and its refrigerator are provided in which the user can intuitively recognize the positional relationship between a plurality of storage areas and provide stored food information that substantially matches the user's visual perspective. We want to provide a control method.

In addition, through an embodiment of the present invention, the camera driving time and the camera shooting time are linked to the opening of the door and/or the rotation angle of the door, thereby reducing power consumption due to the camera and providing food through the camera at an optimal time. The purpose is to provide a refrigerator that can obtain storage information and a method to control the same.

In addition, through embodiments of the present invention, it is intended to prevent the quality of photos taken by a camera from being deteriorated due to condensation.

Additionally, through embodiments of the present invention, the latest information about food stored in the refrigerator can be provided to the user.

In addition, through an embodiment of the present invention, it is intended to provide optimal food storage information to users by allowing managers or users to directly correct the area of the photo displayed to the user.

In addition, through embodiments of the present invention, it is intended to provide a refrigerator and a control method thereof that can provide optimal food storage information to the user by installing a camera on a fixed body to prevent the camera from shaking.

In addition, through embodiments of the present invention, it is intended to provide a refrigerator and a control method thereof that can effectively identify and provide food storage information in the drawer while simplifying the configuration by identifying the status information of the drawer through software.

In addition, through an embodiment of the present invention, a refrigerator capable of providing up-to-date food storage information for each of a plurality of storage areas inside a storage room including a drawer through continuous shooting through a single camera and a control method thereof are provided. I want to do it.

In addition, through an embodiment of the present invention, food stored in another storage area that at least partially overlaps vertically with the inner space of the drawer or in another drawer inner space that at least partially overlaps the inner space of the drawer vertically The aim is to provide a refrigerator that can provide storage information using a single camera and a method of controlling the same.

Additionally, through embodiments of the present invention, it is possible to prevent the quality of photos taken by a camera from being deteriorated due to condensation.

In addition, through embodiments of the present invention, it is possible to provide a refrigerator that can reduce the cost of the refrigerator by limiting the number of cameras installed in the refrigerator to one.

In addition, through embodiments of the present invention, it is intended to provide a refrigerator and a control method thereof that can effectively prevent an increase in power consumption due to a camera.

In addition, through embodiments of the present invention, it is intended to provide a refrigerator and a control method thereof that can minimize the load on the control unit and the memory unit and process continuously captured photos effectively and efficiently.

In order to implement the above-described object, according to one embodiment of the present invention, a storage compartment is formed in the main body of the refrigerator by a fixed insulating wall and is provided with an opening; a door rotatably provided on the main body to open and close the opening; A drawer provided in the storage compartment; The ceiling of the storage compartment is photographed together with an area where food is stored in the outer space of the drawer in the storage compartment (first area) and an area where food is stored in the inner space of the drawer in the storage compartment (second area). a camera provided to be fixed to; And, through the shooting time of the photo in which the first area and the second area are taken together, the part about the first area (first area photo) and the part about the second area (second area photo) in the photo. A refrigerator may be provided that includes a control unit that separates and individually separates and stores the refrigerator.

The control unit determines a photo that satisfies update conditions for the first area photo and the second area photo among the captured photos, and the update conditions are preferably different depending on the time of shooting.

The control unit preferably updates the first area photo and the second area photo separately and independently.

Preferably, the control unit determines a photo taken at a specific time when the door is opened and then closed, and extracts and stores an image for updating the first area.

Preferably, the control unit determines a photo taken at a specific point in time when the drawer is retracted after being withdrawn, and extracts and stores an image for updating the second area.

Preferably, the control unit separates the first area photo and the second area photo from each of the plurality of photos taken continuously, and separates and stores them individually.

The first area is an area where food is placed by at least one shelf, and the first area may be located above the second area.

It may include a display provided in front of the door, where the first area photo and the second area photo are displayed separately and the first area photo and the second area photo are updated independently.

The storage compartment may include a third area provided at the bottom of the drawer so as to vertically overlap the drawer in a state in which the drawer is pulled out, and separated from the first area and the second area to store food. You can.

When the drawer is pulled out, the first area and the second area can be photographed together, and when the drawer is retracted, the first region and the third region can be photographed together.

Preferably, the control unit separates the first area photo and the third area photo (third area photo) from the photo, separates them, and stores them individually.

The control unit preferably updates the first area photo, second area photo, and third area photo individually.

In order to implement the above-described object, according to one embodiment of the present invention, a storage compartment is formed in the main body of the refrigerator by a fixed insulating wall and is provided with an opening; a door rotatably provided on the main body to open and close the opening; A drawer provided in the storage compartment; The ceiling of the storage compartment is photographed together with an area where food is stored in the outer space of the drawer in the storage compartment (first area) and an area where food is stored in the inner space of the drawer in the storage compartment (second area). a camera provided to be fixed to; Through the shooting point of the photo in which the first area and the second area are taken together, the part about the first area (first area photo) and the part about the second area (second area photo) are separated from the photo. Thus, a control unit that individually distinguishes and stores; Additionally, a refrigerator may be provided that includes a display that displays the first area photo and the second area photo to distinguish them from each other.

It is preferable that the display displays a first area frame to which the first area photo is allocated and displayed and a second area frame to which the second area photo is allocated and displayed.

The first area may be an area where food is stored by a shelf in the upper space outside the drawer.

Preferably, at least one of the first area photo and the second area photo stored in the control unit is displayed after its size has been corrected to fit the first area frame and the second area frame size.

In the storage compartment, a third area is provided at the bottom of the drawer to vertically overlap the drawer when the drawer is pulled out, and is separated from the first area and the second area to store food. This is desirable.

The control unit separates the first area photo and the portion of the third area (third area photo) from the photo through the shooting time of the photo in which the first area and the third area are taken together, and individually Can be classified and saved.

Preferably, on the display, a third area frame in which the third area photo is allocated and displayed is displayed separately from the lower part of the first area frame.

It is preferable that the left and right widths of the first area frame and the third area frame are the same, and the third area photo is displayed after being corrected to fit the left and right widths of the third area frame.

The drawer includes two drawers that are symmetrical to the left and right, and the photos in the second area can be separated into left and right and individually classified and stored.

It is preferable that the display displays an image captured through the camera and an adjustment line for adjusting the position of a separation line for dividing the image into a plurality of areas.

Preferably, the adjustment line includes at least one horizontal adjustment line and at least one vertical adjustment line, and the adjustment line is movable through a user's manipulation of the display.

In order to implement the above-described object, according to one embodiment of the present invention, a storage compartment is formed in the main body of the refrigerator by a fixed insulating wall and is provided with an opening; a door rotatably provided on the main body to open and close the opening; A drawer provided in the storage compartment; The ceiling of the storage compartment is photographed together with an area where food is stored in the outer space of the drawer in the storage compartment (first area) and an area where food is stored in the inner space of the drawer in the storage compartment (second area). a camera provided to be fixed to; And, a portion of the first area (first area photo) is separated from a photo taken at a specific point in time when the door is opened and then closed, and the second area is separated from a photo taken at a specific point in time when the drawer is withdrawn and retracted. A refrigerator may be provided that includes a control unit that separates the portion (second area photo) and separately separates, classifies, stores, and updates the first area photo and the second area photo.

The door includes a left door and a right door, the drawers are provided on the left and right sides of the storage room, and the control unit divides the part for the left drawer and the part for the right drawer into each other in the photo of the second area. It is desirable to separate, distinguish, store and update them individually.

Door sensors that detect the opening angles of each of the left and right doors may be provided.

The door sensor is preferably provided at a hinge connection portion that connects the upper surface of the main body and the upper surface of the door to allow the door to rotate with respect to the main body.

In order to implement the above-described object, according to one embodiment of the present invention, a storage compartment is formed in the main body of the refrigerator by a fixed insulating wall and is provided with an opening; a door rotatably provided on the main body to open and close the opening; a camera fixed to the storage compartment and taking pictures of the inside of the storage compartment to update food information in the storage compartment; a door switch that detects opening and closing of the door; a door sensor that detects an opening angle of the door; And, according to the detection result of the door switch, the transition point between the standby mode of the camera and the driving mode to start shooting is determined, and according to the detection result of the door sensor, a valid photo for updating among the captured photos is determined. A refrigerator may be provided that includes a control unit that determines a reference point for selecting .

In order to implement the above-described object, according to one embodiment of the present invention, a storage room; A drawer movably provided in the storage room and equipped with a marker; a camera that photographs the drawer from outside the drawer; And, by detecting the position of the marker in photos continuously taken through the camera, status information of the drawer is determined, including at least one of the degree of withdrawal of the drawer, whether or not to withdraw the drawer, the direction of movement, and the stationary state or moving state. A refrigerator may be provided, characterized in that it includes a control unit that operates. Since the marker is moved integrally with the drawer, the status information of the drawer can be determined through the location of the marker.

In order to implement the above-described object, according to one embodiment of the present invention, a storage room; A drawer movably provided in the lower part of the storage compartment and equipped with a marker; a camera fixed to the ceiling of the storage room to photograph the inside of the drawer from the outside of the drawer; And, by detecting the position of the marker in photos continuously taken through the camera, status information of the drawer is determined, including at least one of the degree of withdrawal of the drawer, whether or not to withdraw the drawer, the direction of movement, and a stationary state or a moving state. A refrigerator may be provided, characterized in that it includes a control unit that operates.

In order to implement the above-described object, according to one embodiment of the present invention, a storage compartment is formed in the main body of the refrigerator by a fixed insulating wall and is provided with an opening; a door rotatably provided on the main body to open and close the opening; A drawer provided in the storage compartment; at least one shelf provided in the storage compartment and located above the drawer; And to photograph the area where food is stored by the shelf in the outer upper space of the drawer (first area) and the area where food is stored in the inner space of the drawer (second area) together within the storage room, A refrigerator may be provided that includes a camera fixed to the ceiling of the storage compartment between the front end of the shelf and the opening.

The features of the above-described embodiments of the present invention may be implemented in combination with each other as long as they are not contradictory or exclusive.

According to the present invention, the user does not need to open the door to obtain information about food stored in the refrigerator, thereby preventing leakage of cold air contained in the storage compartment. Therefore, the energy efficiency of the refrigerator can be improved by preventing unnecessary cold air loss.

Additionally, since the present invention can provide users with up-to-date information about food stored in the refrigerator, reliability of food information provided to users can be improved.

Additionally, since the present invention can provide information about food stored in various locations using a single camera, only a structure for installing a single camera is added, making design easy. In particular, costs incurred due to camera use can be reduced.

According to the present invention, dew is prevented from forming on a camera installed in a refrigerator, so photos taken by the camera can be stably provided to the user.

According to the present invention, an individual can remotely check the current situation inside the refrigerator and receive food-related information necessary for the individual from an external business operator.

And according to the present invention, an image captured by a camera of the inside of a drawer can provide a screen similar to what the user actually sees when using the refrigerator.

Additionally, according to the present invention, locations that cannot be viewed at once because they overlap in spatial arrangement can be provided to the user as two-dimensional images on a single screen.

1 is a front view of a refrigerator according to the present invention.
Figure 2 is a view with the door in Figure 1 opened.
Figure 3 is a diagram showing a third area disposed on the floor of a storage room.
Figure 4 is a control block diagram of one embodiment according to the present invention.
Figure 5 is a diagram explaining the angle of view of the camera.
Figure 6 is a side cross-sectional view of Figure 5.
Figure 7 is a screen captured by a camera in the state of Figure 6.
Figure 8 is a diagram explaining the positioning of the camera.
Figure 9 is a cross-sectional view showing the main part of the refrigerator.
Figure 10 is a diagram explaining the operation method of the door sensor.
Figure 11 is a diagram specifically showing the left hinge portion and the right hinge portion.
Figure 12 is a screen provided to the user.
Figure 13 is a diagram explaining a method of adjusting photos taken by a camera.
Figure 14 is a perspective view showing a camera.
Fig. 15 is a diagram showing the main part of the camera.
Figure 16 is a cross-sectional view of Figure 14.
17 is a diagram explaining the arrangement of the heater.
Figure 18 is a diagram illustrating a photo taken while the camera is mounted on the refrigerator upside down.
Figure 19 is a diagram illustrating a photo taken with the camera turned horizontally on the refrigerator.
Figure 20 is a view showing the camera housing in an assembled state.
Figure 21 is a front view of the first housing.
Figure 22 is a front view of the second housing.
Figure 23 is a front view showing a state in which a camera is installed in the first housing.
Figure 24 is a side cross-sectional view of Figure 23.
Figure 25 is a cross-sectional view showing the camera housing installed in the inner case.
Figure 26 is a table comparing the power consumption of cameras in the present invention.
Figure 27 is a diagram comparing supplied standby current and driving current.
Figures 28 and 29 are diagrams explaining the starting point of the camera's photo taking and the camera's continuous shooting.
Figure 30 is a diagram showing an embodiment of the drawer detection unit.
FIG. 31 is a diagram illustrating a method of detecting movement of a drawer in the drawer detection unit of FIG. 30.
Figure 32 is a diagram showing a state in which a marker is displayed on a drawer.
Figure 33(a) is a diagram showing a photograph taken in a state in which the left and right drawers are withdrawn.
Figure 33(b) is a diagram showing a state in which the left drawer is retracted and the right drawer is withdrawn.
34 is a flowchart illustrating a method of operating a refrigerator according to an embodiment of the present invention.
35 is a flowchart illustrating a marker recognition and tracking method according to an embodiment of the present invention.
Figure 36 is a diagram illustrating a marker for detecting when a drawer is withdrawn or inputted according to an embodiment of the present invention.
Figure 37 is a diagram for explaining the shape of a marker according to an embodiment of the present invention.
Figure 38 is a flowchart illustrating a method for detecting a drawer closing operation according to an embodiment of the present invention.
Figure 39 is a flowchart for explaining the operation of the refrigerator performed upon detection of completion of the drawer closing operation according to an embodiment of the present invention.
Figure 40 is a flowchart illustrating a method of controlling an image of a specific area inside a refrigerator taken at a specific time according to an embodiment of the present invention.
41 is a diagram showing various types of markers.
42 is a diagram explaining a marker position recognition method.
43 is a diagram showing the degree to which the drawer is opened.
Figure 44 is a diagram explaining the movement of a marker.
Figure 45 is a control flow diagram according to an embodiment of the present invention.
Figure 46 is a control flowchart explaining a modification of Figure 45.
Figure 47 is a control flowchart explaining another modification of Figure 46.
Figure 48 is a control flowchart explaining another modification of Figure 45.
Figure 49 is a modified example of Figure 45.
Figure 50 is a control flow diagram according to another embodiment of the present invention.
Figure 51 is a diagram explaining the process of updating the image in a state where two doors and two drawers are provided to open and close the storage room.
Figure 52 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.
Figure 53 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.
Figure 54 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.
Figure 55 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.
Figure 56 is a diagram explaining the operation of the camera heater.
Figure 57 is a diagram explaining the results of an experiment on condensation occurring at different temperatures on the transparent window of the camera.
Figure 58 is a cross-sectional view of a transparent window.
Figures 59 and 60 are diagrams briefly showing the structure in which a camera is installed in the inner case.
Figure 61 is a diagram showing another type of refrigerator to which the present invention can be applied.
Figure 62 is a screen provided to the user from the refrigerator according to Figure 61.
FIG. 63 is a diagram illustrating a method of adjusting a photo taken by a camera in the refrigerator according to FIG. 61.

Hereinafter, preferred embodiments of the present invention that can specifically realize the above object will be described with reference to the attached drawings.

FIG. 1 is a front view of a refrigerator according to the present invention, and FIG. 2 is a view with the door in FIG. 1 opened. Hereinafter, it will be described with reference to FIGS. 1 and 2.

The refrigerator according to the present invention is a top mount type in which the freezer compartment, which is a storage compartment where food is stored, and the refrigerator compartment are divided into upper and lower sections, and the freezer compartment is placed on the upper side of the refrigerator compartment, and the freezer compartment and the refrigerator compartment are divided into left and right sides. The same can be applied to a side-by-side-type refrigerator.

However, in this embodiment, for convenience of explanation, the description will focus on the Bottom Freezer-Type, in which the freezer compartment and the refrigerator compartment are divided into upper and lower compartments, and the freezer compartment is located below the refrigerator compartment.

The main body of the refrigerator includes an outer case 10 that forms the overall appearance when viewed from the outside by a user, and an inner case 12 that forms a storage compartment 22 where food is stored inside. A predetermined space may be formed between the outer case 10 and the inner case 12 to form a passage through which cold air circulates. Meanwhile, an insulating material is filled between the outer case 10 and the inner case 12, so that the inside of the storage compartment 22 can be maintained at a relatively low temperature compared to the outside.

In addition, a refrigerant cycle device that generates cold air by circulating refrigerant is installed in a machine room (not shown) formed in the space between the outer case 10 and the inner case 12. The freshness of stored food can be maintained by maintaining the inside of the refrigerator at a low temperature using a refrigerant cycle device. The refrigerant cycle device includes a compressor that compresses the refrigerant, an evaporator (not shown) that converts the liquid refrigerant into a gaseous state and exchanges heat with the outside.

The refrigerator is equipped with doors 20 and 30 that open and close the storage compartment. At this time, the doors may include a freezer compartment door 30 and a refrigerator compartment door 20, and one end of each door is rotatably installed on the main body of the refrigerator by a hinge. The freezing compartment door 30 and the refrigerating compartment door 20 may be formed in plural pieces. That is, as shown in FIG. 2, the refrigerating compartment door 20 and the freezing compartment door 30 may be installed in a manner that opens toward the front around both corners of the refrigerator.

A foaming agent is filled between the outer case 10 and the inner case 12, so that insulation can be achieved between the outside and the storage compartment 22.

The storage compartment 22 forms a space insulated from the outside by the inner case 12 and the door 20. The storage compartment 22 can form a space that is isolated and insulated from the outside when the door 20 seals the storage compartment 22. In other words, the storage compartment 22 is formed by the door 20. It can be said to be a space isolated from the outside through an insulating wall and an insulating wall formed by the cases 10 and 12.

In the storage compartment 22, cold air supplied from the machine room can flow to various places, so that food stored in the storage compartment 22 can be maintained at a low temperature.

The inner case 12 may form a barrier 60 at the bottom of the storage compartment 22. A barrier 60 is installed at the bottom of the storage compartment 22 to distinguish the freezer compartment from the refrigerator compartment. The barrier 60 may have a predetermined thickness and be formed in the inner case 12. The barrier 60 may extend in the horizontal direction.

The storage compartment 22 may include a shelf 40 on which food is placed at the top. At this time, a plurality of shelves 40 are provided, and food can be placed on each shelf 40. The shelf 40 may partition the interior of the storage compartment in the horizontal direction.

A drawer 50 that can be inserted or withdrawn is installed in the storage compartment 22. Food, etc. are accommodated and stored in the drawer 50. Two drawers 40 can be arranged on the left and right sides in the storage compartment 22 . The user can open the left door of the storage compartment 22 to access the drawer located on the left. On the other hand, in order for the user to access the drawer located on the right side, the right door of the storage compartment 22 can be opened.

A space in which food is stored may be formed in the barrier 60. Since the barrier 60 is not provided on the door but on the inner case 12, it does not move as the door rotates and remains fixed. Therefore, users can reliably store or withdraw food.

The storage compartment 22 is divided into a space located on the upper side of the shelf 40, a space formed by the drawer 50, a space formed in the barrier 60, etc., and is a space where food is stored. It can be divided into multiple parts.

At this time, each space is a storage space arranged inside a storage compartment 22, and cold air supplied to the storage compartment 22 can be moved to each space. In other words, because each space is divided enough to allow cold air to move between them, it has a different meaning from the storage room described above.

In particular, unlike storage rooms where each space is an insulated space, temperature differences may exist within one storage room, but they do not form insulated spaces.

Cold air supplied to one storage room does not freely move to other storage rooms, but cold air supplied to one storage room can freely move to each partitioned space installed inside one storage room. That is, cold air located on the upper side of the shelf 40 can move to the space formed by the drawer 50.

Additionally, the present invention may include a camera 70 that takes pictures of the inside of the storage compartment 22. At this time, the camera 70 is provided to maintain a fixed position, so it is possible to photograph the same part.

In particular, the camera 70 is installed on the upper wall of the inner case 12 facing downward, so that it can take pictures of food stored in the storage compartment 22. The captured photo may have the appearance of the user looking down, that is, the appearance of the user looking at the refrigerator when actually using it.

In particular, the camera 70 is installed at a position corresponding to the inside of the drawer when the drawer is in a fully open state, so that the captured photo can provide a screen similar to what the user sees inside the drawer from above.

At this time, the storage compartment 22 may include a first area 42 where food is placed on the shelf 40 and a second area 52 defined by the inner space of the drawer 50. Additionally, when the drawer 50 is pulled out, it may include a third area 62 that overlaps the second area 52, which is a storage space different from the first and second areas. This part will be explained in more detail later.

The first area 42 and the second area 52 may have different overall heights within the storage compartment 22 . That is, the first area 42 and the second area 52 may be lowered in that order.

Photos taken by the camera 70 or various information about the refrigerator can be provided to the user through the display 14 provided on the front of the refrigerator. Additionally, the user can control the refrigerator through the display 14.

Cold air supplied to the storage compartment 22 may be moved to the first area 42 and the second area 52, and cold air located in each area may be moved to other areas.

Figure 3 is a diagram showing a third area disposed on the floor of the storage room. Hereinafter, the description will be made with reference to FIG. 3.

A recessed portion is provided in the barrier 60 so that food can be stored in the portion, and this area can be referred to as the third area 62. The third area 62 is a space formed in the barrier 60.

At this time, the third area is a storage space located in a different location from the first area and the second area, and can provide the user with another space for storing food.

It is possible to store frequently used foods, such as eggs, in the third area 62. For this purpose, it is possible to provide a cover 68 that opens and closes the third area 62. At this time, it is preferable that the cover 68 is made of a transparent material, so that food stored in the third area 62 can be photographed even if the cover 68 obscures it when the camera 70 takes pictures from the upper side. .

Meanwhile, the drawer 50 is placed on the upper side of the barrier 60. Therefore, when the drawer 50 is inserted into the lower space of the shelf, the user can access the third area 62.

On the other hand, when the drawer 50 is pulled out from the lower space of the shelf, the drawer 50 is placed on the upper side of the third area 62, and the user approaches the third area 62. Can not.

That is, when the drawer 50 is pulled out toward the user, the drawer 50 and the third area 62 overlap each other. At this time, when the user looks down at the drawer 50, the third area 62 is not visible to the user, but the inside of the drawer 50 can be seen.

On the other hand, when the drawer 50 is not pulled out toward the user, the drawer 50 and the third area 62 are arranged so as not to overlap each other. When the user looks down at the third area 62, the inside of the drawer 50 is not visible to the user, and only the third area 62 can be shown.

The first area 42, the second area 52, and the third area 62 may have different overall heights within the storage compartment 22. That is, the first area 42, the second area 52, and the third area 62 may be lowered in that order.

Cold air supplied to the storage compartment 22 may be moved to the first area 42, the second area 52, and the third area 62, and the cold air located in each area may be moved to another area. can also be moved.

That is, the area photographed by the camera 70 may be a plurality of storage spaces within the storage compartment 22. The camera 70 takes pictures of the inside of one storage compartment 22, but the inside of one picture may include information about a plurality of partitioned storage spaces.

At this time, the camera 70 can take pictures so that a plurality of storage spaces are included in one picture. In particular, it is possible for one photo to have different storage spaces depending on the time it is taken.

For example, a camera placed in a fixed form photographs the first and second areas or selectively captures the second and third areas depending on the relationship between the time when the picture is taken and the position of the door or the drawer. It is possible to shoot with . Details about this will be described later.

Figure 4a is a control block diagram of one embodiment according to the present invention. Hereinafter, the description will be made with reference to FIG. 4A.

The present invention may include a control unit 100 that divides and manages one photo taken by the camera 70 into a plurality of images. At this time, the image may mean a portion of a photo that has been processed or corrected by the control unit 100.

The control unit 100 can command the camera 70 to take a picture and receive a picture taken by the camera 70.

Additionally, the control unit 100 may provide a portion of the divided image to the display 14 to provide the user with the latest information about food stored in the storage compartment 22. At this time, the display 14 can be installed on the front of the refrigerator, or it can be a separate device separated from the refrigerator. That is, it is possible for a user to receive an image about the storage room 22 and obtain information through an external communication terminal such as a mobile phone.

At this time, the control unit 100 can divide the photos taken by the camera 70 into each area, divide them into a plurality of independent images, and provide them to the display 14. At this time, the photo selected by the control unit 100 may include information about the most recent state of the storage room 22 when the user last accessed the storage room 22 and retrieved or received food from the storage room.

An embodiment of the present invention may include a door switch 110 that can detect whether the door 20 opens or closes the storage compartment 22. At this time, the door switch 110 is provided in the outer case 10, and when the door 20 contacts the outer case 10, it detects that the door 20 seals the storage compartment 22. can do. Additionally, if the door 20 does not contact the outer case 10, it can be detected that the door 20 is opening the storage compartment 22.

Additionally, an embodiment of the present invention may include a door sensor 120 that detects the rotation angle of the door 20. At this time, the door sensor 120 can detect the rotation direction of the door 20 and the rotation angle of the door 20. For example, when the door 20 rotates more than a certain angle, a change occurs in the door sensor 120, so that the door sensor 120 can detect that the door 20 has been rotated more than a certain angle. there is. Additionally, when the door 20 is rotated in a specific direction, it is possible to detect the direction of rotation of the door 20 by detecting changes in pulses generated depending on the direction. Of course, the door sensor 120 may be modified into various forms other than those described above.

In addition, the door sensor 120 includes a light emitting unit and a light receiving unit, so that it is possible to detect the rotation angle of the door 20 by determining whether the light emitted from the light emitting unit is transmitted to the light receiving unit.

In particular, the door sensor 120 may be driven only when the door switch 110 determines that the door 20 has opened the storage compartment 22.

Meanwhile, if the door sensor 120 determines that the door 20 has been rotated at a specific angle, it is also possible to take a picture with the camera 70. In this case, even if the door switch 110 detects the opening of the door 20, the camera 70 does not immediately start taking pictures, but takes pictures when the door sensor 120 reaches a certain angle. will be filmed.

Once the camera 70 starts taking pictures, it is also possible to continue taking pictures at predetermined time intervals. Of course, if the door sensor 120 determines that the door 20 is open by a certain angle, the camera 70 can continue to take pictures, not just one picture, until a command to stop taking pictures is provided. do.

The control unit 100 may include a drawer detection unit 130 that detects whether the drawer 50 is withdrawn or retracted. At this time, the drawer detection unit 130 may mean a part that reads the image of a photo taken by the camera 70 within the control unit 100. The drawer detection unit 130 is capable of detecting not only the movement of the drawer 50 but also the direction of movement of the drawer 50.

In particular, the drawer detection unit 130 may be implemented by software. The drawer detection unit 130 can detect the location of the drawer 50 using information captured in a photo taken by the camera 70.

The drawer 50 is equipped with an identification mark called a marker, and the marker can be used to determine whether the drawer 50 has entered a specific section, stopped if entered, and changed the direction of movement after stopping through a photo taken with the marker. You can find information about it. The marker will be described in detail later.

Since the marker is displayed on the drawer, it has the same movement as the drawer 50. Therefore, it is possible to obtain various information about the movement of the drawer 50 using information from the photo taken by the camera 70, without using a separate sensor to detect the movement of the drawer.

Photos taken by the camera 70 may be stored in the storage unit 18. At this time, the storage unit 18 may be provided in a refrigerator or in a device separate from the refrigerator. At this time, the storage unit 18 may be placed in a refrigerator together with the display 14, or may be installed together with the display 14 in a device other than the refrigerator, a server connected to the refrigerator and a network, or a terminal connected to the refrigerator and the network. there is.

At this time, the storage unit 18 may not store all photos taken by the camera 70. For example, if the storage unit 18 does not receive a command to select a specific photo from the control unit 100, it is possible to delete the previously taken and stored photo and store the currently captured photo (First In , First Out). The storage unit 18 stores only some of the photos taken by the camera 70, thereby reducing storage capacity.

Figure 4b is a control block diagram of a modified form of one embodiment according to the present invention. Hereinafter, description will be made with reference to FIG. 4B.

In FIG. 4B, unlike FIG. 4A, the drawer detection unit 130 does not use information obtained from a photo taken by the camera 70, but can separately detect the movement of the drawer 50.

For example, the drawer detection unit 130 may be formed in the form of a plurality of hall sensors. A plurality of hall sensors may be installed in the movement path of the drawer 50. When the drawer 50 is moved, a plurality of hall sensors can detect each change, thereby detecting the position and direction of movement of the drawer 50.

The drawer detection unit 130 detects the movement of the drawer, and when taking a photo with the camera 70, it is possible to determine the point in time when the interior of the drawer is included in the photo taken by the camera 70. . That is, the drawer detection unit 130 can detect a point in time when information about an item stored inside the drawer can be expressed in a photo taken by the camera.

The drawer detection unit 130 can detect the movement of the drawer and detect the point in time when the drawer is determined to be up to date after the user has used it. That is, the drawer detection unit 130 can detect the point in time when the photo taken by the camera includes the latest information that the user has finished using the drawer. In other words, it is possible to indirectly determine the completion point of product arrival and departure or the end point of product update.

Figure 5 is a diagram explaining the angle of view of the camera. Hereinafter, the description will be made with reference to FIG. 5.

In Figure 5, for convenience of explanation, the drawer 50 is pulled out so that the camera 70 can photograph the second area 52, and the shelf is omitted.

The camera 70 may be disposed on the upper wall of the inner case 12 to be inclined toward the rear wall of the inner case 12.

Meanwhile, a photo taken by the camera 70 may have a horizontal angle of view along the width direction of the storage compartment 22 and a vertical angle of view along the front-to-back direction.

At this time, the horizontal angle of view according to the camera 70 is such that the food stored in the inner space of the two drawers 50, that is, the second area 52, can be seen when the two drawers 50 are pulled out. It is desirable.

In addition, it is possible that the range of the horizontal angle of view according to the camera 70 includes at least part of both left and right ends of the shelf 40. This is because, in order to obtain information about food placed on the upper side of the shelf 40, it is desirable to take pictures from the camera 70 to both ends of the shelf 40.

As the angle of view of the camera 70 changes, the range inside the storage compartment shown in the photo changes.

Figure 6 is a side cross-sectional view of Figure 5, and Figure 7 shows a screen captured by a camera in the state of Figure 6. Hereinafter, it will be described with reference to FIGS. 6 and 7.

An opening 14 is formed at the front of the inner case 12 to allow the user to access the storage compartment 22. Through the opening 14, a user can put food into the storage compartment 22 or take food out from the storage compartment 22. The door 20 can open and close the opening 14.

The camera 70 is installed outside the drawer and can take pictures of the open, closed, and moving states of the drawer by taking pictures while the drawer is moving or stopping. That is, when the camera 70 continuously takes pictures, the pictures taken by the camera 70 may show the drawer in a state in which the position of the drawer has changed depending on the time of shooting.

Additionally, the camera 70 may provide the user with a screen containing the latest entry/exit status of items inside the storage compartment 22.

The camera 70 is preferably installed within a range l between the opening 14 and the front end of the shelf 40. When there are a plurality of shelves 40, it is preferable that they are installed in the area (l, see FIG. 8) between the front end, which is located most forward among the front ends, and the opening 14. That is, it is desirable to install it on the ceiling of the storage room corresponding to this area. Since the camera 70 must selectively take pictures of the first area and the second area 52 or the first area and the third area 62, the camera 70 must take pictures of the first area and the second area 52 rather than the front end of the shelf 40. It cannot be moved to the rear wall of the inner case 12.

Figure 6 shows the vertical angle of view of the camera 70, and one end (front end) of the vertical angle of view is preferably arranged so that it can capture images up to the front end of the drawer 50 when the drawer 50 is open. do. In addition, it is desirable that the other end (rear end) of the vertical angle of view is arranged so that the rear end of the uppermost shelf can be photographed.

Since the vertical field of view of the camera 70 includes the second area 52 by the drawer 50, the camera 70 provides the user with information about the second area 52. can be obtained.

As shown in FIG. 7 , the camera 70 can take pictures of the first area 42 and the second area 52 with one picture. At this time, the camera 70 can take pictures of the two drawers 50 arranged on the left and right sides. In addition, when a plurality of shelves are provided, it can be seen that the first area 42 and the second area 52, which are divided by a plurality of shelves, can be photographed together.

The camera 70 is installed inside the storage compartment 22 and can take pictures containing information about the inside of the storage compartment 22. Since the camera 70 is installed inside the storage compartment, it does not move. In other words, since it is fixedly installed in the storage compartment of the main body, which is generally always fixed, the camera itself does not move. Therefore, when taking a picture with the camera 70, the camera 70 is always in a stopped state.

A photo taken by the camera 70 includes the first area 42 and the second area 52, or includes the first area 42 and the third area 62.

When the camera 70 takes a picture including the first area 42 and the second area 52, the user can view the food and the food stored on the upper side of the shelf 40 through the photo. Information about food stored inside the lower 50 can be obtained.

On the other hand, when the camera 70 takes a picture including the first area 42 and the third area 62, the user can see the food stored on the upper side of the shelf 40 and the food stored on the upper side of the shelf 40 through the photo. Information about food stored in the barrier 60 can be obtained.

The camera 70 photographs the inside of one storage compartment 22 and transmits the captured photos, and the control unit 100 can obtain information on a plurality of storage spaces.

The camera 70 is disposed above the first area 42, the second area 52, and the third area 62, and takes pictures while looking down.

Figure 8 is a diagram explaining the positioning of the camera. Hereinafter, the description will be made with reference to FIG. 8.

As shown in FIG. 8, the camera can be mounted at positions L1, L2, L3, and L4.

L1 is an external space outside the storage compartment 22 and partitioned by the outer case 10. When the camera is installed at the position L1, the user accesses the storage compartment 22 with the door open and uses the refrigerator, but a problem may occur in that the camera is blocked by the user.

As disclosed in the Japanese patent disclosed above, when a camera is installed outside the refrigerator and an attempt is made to film the inside of the storage compartment, the image interference caused by the user is too great, and in particular, the movement of the drawer cannot be tracked because the drawer is obscured. In the embodiment of the present invention, it was impossible to simultaneously determine the internal image of the drawer and the movement state of the drawer. Additionally, the L1 position may mean that the camera is located on the door. In this case, since the door is movable, the camera may be moved during filming. Therefore, it can be said that it is very difficult to obtain stable and clear photos.

L2 is the interior of the storage compartment 22 and refers to the upper wall, that is, the ceiling, of the storage compartment 22. In the embodiment of the present invention, this corresponds to the location where the camera is installed, and there is no problem of it being obscured by the user. Additionally, since the upper wall of the storage compartment 22 corresponds to the inner case 12, it is convenient to place electric wires for supplying electricity to the camera. Additionally, since the inner case 12 is generally always fixed, the camera can be stably mounted in a fixed configuration.

L3 is the inside of the storage compartment 22 and is a side wall of the storage compartment 22. At this time, the camera is installed to be tilted to take pictures from one side of the storage compartment 22 toward the other side of the storage compartment 22. If the camera is to be placed in the middle based on the height of the storage compartment 22 rather than at the top of the storage compartment 22, it must be installed on the side wall of the inner case 12.

If the camera is installed on one side wall, the camera lens has no choice but to be installed facing the other side wall, and therefore, there is a problem in that an asymmetrical image of the inside of the storage compartment 22 is obtained due to the difference in distance between the left and right sides.

L4 is the inside of the storage compartment 22 and refers to one end of the shelf 40. In order for the camera to be placed based on the height of the storage compartment 22, it must be supported by a specific structure. In this case, the camera may be fixed to one end of the shelf 40.

In this case, since the distance between the drawer 50 and the camera is shorter than that of L2, there is a problem that a camera with a larger angle of view compared to that of L2 must be used. Additionally, a problem arises in that the degree of distortion in the resulting photo becomes more severe. In addition, various problems that cause inconvenience to users may occur, such as installing the wires supplied to the camera on the shelf 40. Additionally, the shelf 40 is generally movable. Therefore, the camera itself may shake when placing food on the shelf.

The camera applied to the embodiment of the present invention has a field of view of 120 degrees, is VGA level, and is capable of providing performance of 20fps.

Figure 9 is a cross-sectional view showing the main part of the refrigerator. Hereinafter, the description will be made with reference to FIG. 9.

A basket 21 for storing food may be provided inside the door 20. The basket 21 may be installed on the side where the door 20 faces the storage compartment 22.

There may be a problem that the basket 21 is shown in the photo taken by the camera 70. Therefore, in an embodiment of the present invention, it is possible to detect whether the door 20 has been rotated more than a certain angle θ through the door sensor 120 and take a picture through the camera 70. Of course, it is also possible to transmit information about the relevant time point from the door sensor 120 to the control unit 100 so as to select a photo taken at a time point adjacent to the time point detected by the door sensor 120.

For example, the constant angle θ can be from 60 degrees to 80 degrees. Of course, the angle may vary depending on the capacity of the refrigerator, the size of the door, or the front-to-back length of the basket 21.

At this time, the constant angle (θ) may mean an angle at which the basket 21 may not appear in the photo when the camera 70 takes a photo, and this constant angle (θ) is the angle of the basket 21. It may vary depending on the left and right width and the length before and after.

The door switch 110, which detects whether the door 20 is opened or closed, may be installed on the upper side of the refrigerator. At this time, the door switch 110 detects that the door 20 seals the storage compartment when pressed by the door 20, and when it is not pressed by the door 20, the door 20 closes the storage compartment. You can sense that it is open.

A left hinge portion 300 may be provided to rotatably install the left door 20 in the refrigerator, and a right hinge portion 320 may be provided to rotatably install the right door 20 in the refrigerator. The left hinge part 300 and the right hinge part 320 include a part coupled to the door 20 by providing a rotation axis around which the door 20 can rotate, and a part installed in the outer case 10, It can be divided into a connection part that connects two parts. That is, it can be divided into door coupling parts 300c and 302c, outer case coupling parts 300a and 320a, and connection parts 300b and 320b connecting them.

A left door sensor 120L that detects rotation of the left door 20 may be installed in the left hinge portion 300.

A right door sensor 120R that detects the rotation of the right door 20 may be installed in the right hinge portion 320.

The left door sensor 120L and the right door sensor 120R can independently detect the rotation angle of each door.

Figure 10 is a diagram explaining the operation method of the door sensor. Hereinafter, the description will be made with reference to FIG. 10.

The door sensor 120 may include a light emitting unit 122 that emits light and a light receiving unit 124 that receives the light emitted from the light emitting unit 122.

The light emitted from the light emitting unit 122 may be reflected through the upper surface of the door 20 and transmitted to the light receiving unit 124. In this case, it may be made of a material that reflects well so that light reflection can occur stably.

When the light emitted from the light emitting unit 122 is transmitted to the light receiving unit 124, it can be detected that the door 20 has been rotated by an angle smaller than the angle at which the door sensor 120 is installed. That is, this may mean that the door 20 is rotated at an angle smaller than a certain angle θ or is not rotated while the storage compartment 22 is closed.

On the other hand, if the light emitted from the light emitting unit 122 is not transmitted to the light receiving unit 124, it may be detected that the door 20 has been rotated by an angle greater than the angle at which the door sensor 120 is installed. In other words, this may mean that the door 20 is rotated to a state greater than a certain angle θ so that the user can access the storage compartment 22.

To this end, as shown in FIG. 9, the left hinge portion 300 and the right hinge portion 320 may be mounted across the top surface of the outer case 10 of the main body and the top surface of the door 20. Accordingly, as the door 20 rotates about the rotation centers 302 and 322, the top surface of the door or a portion of the upper surface of the door (for example, a stepped portion) may function as a light receiver. Of course, the left door sensor 120L and the right door sensor 120R can be installed in each connection part 300b and 320b to perform the function of a light emitter. Due to this positional relationship between the sensors (120L, 120R) and the door, the sensors (120L, 120R) can determine when the door opens more than a certain angle (θ) and when the door closes at a certain angle (θ). You can.

Figure 11 is a diagram specifically showing the left hinge portion and the right hinge portion. In Figure 11, other components of the refrigerator are omitted for convenience of explanation. Hereinafter, the description will be made with reference to FIG. 11.

The door sensor may be placed on the front of the inner case, that is, in front of the opening (i.e., where the user is standing) in order to detect the rotation of the door.

A left door sensor 120L for detecting rotation of the left door 20 may be installed in the left hinge portion 300.

A right door sensor 120R may be installed in the right hinge portion 320 to detect the rotation of the right door 20.

The left door sensor or the right door sensor may be disposed in an area between the opening and the respective rotation centers of each hinge part. That is, it can be placed in the connection portions 300b and 320b of each hinge portion.

The door sensor 120 may be connected to other components through a connector (c) and a wire in order to receive electricity or transmit signals to the outside. At this time, the door sensor 120 may be connected to the above-described control unit 100 and provide the acquired signal information.

The left hinge portion 300 may have a rotation center 302 around which the left door 20 rotates. At this time, the left door sensor 120L may be placed to the right of the rotation center 302. The space to the right from the center of rotation 302 is located at the bottom of the left door 20, especially the upper surface of the left door 20, depending on the rotation angle of the left door 20, or may not be located below the left door sensor 120L. It is a space where you can Therefore, the left door sensor 120L is preferably located at the connection portion 300b of the left hinge portion 300. Through this, when the size of the door or basket changes, the constant angle (θ) can be flexibly varied. That is, the space on the right side of the connection part 300b can be easily varied, and thus the position of the left door sensor 120L on the connection part 300b can be easily varied.

The wire w connected to the left door sensor 120L may be connected through the rotation center 302. Additionally, the wire w can be connected to the refrigerator main body through the outer case 10.

The right hinge portion 320 may have a rotation center 322 around which the right door 20 rotates. At this time, the right door sensor 120R may be placed to the left of the rotation center 322. The space to the left of the rotation center 322 is a space in which the right door 20 may or may not be located below the right door sensor 120R depending on the rotation angle of the right door 20. Therefore, for the same reason, it would be desirable for the right door sensor 120R to be installed in the connection portion 320b of the right hinge portion 320.

The time point detected by the door sensor 120 may be the time point at which a photo is captured for each area inside the storage compartment 22, specifically, the first area 42 or the third area 62. . In other words, the time point detected by the door sensor 120 may mean the time point at which a photo of the external storage space of the drawer 50 is captured.

Meanwhile, because the door sensor 120 radiates and receives light, the door sensor 120 irradiates light downward. Light irradiated downward is reflected upward again by the door 20.

The door sensor 120 and the door 20 may maintain a short distance between each other so that the door sensor 120 can receive a sufficient amount of light. At this time, the vertical distance between the door sensor 120 and the door 20 can be maintained at a maximum of 20 mm.

Figure 12 shows examples of screens provided to users. Hereinafter, the description will be made with reference to FIG. 12.

In Figure 12a, the first area 42, the second area 52, and the third area 62 are all provided to the user, and in Figure 12b, the second area 52 and the third area 62 are provided to the user. 12C shows the form in which the first area 42 and the second area 52 are provided to the user.

First, the description will be based on FIG. 12A.

One photo taken by the camera 70 includes a first image showing the first area 42, a second image showing the second area 52, and a third image showing the third area 62. It can be divided into a third image. That is, one photo or multiple photos can be divided by area. At this time, the first image, the second image, and the third image may mean a shape cut from a photograph taken to a range showing the corresponding area.

These classified photos can be independently provided to the user as shown in FIG. 12A. That is, photos of the area are individually provided to the user, so the user can easily obtain information about the food stored in the area.

The display 14 can be arranged so that a plurality of storage areas can be displayed within a frame.

That is, on the upper side of one square frame, the second image showing the second area 52 is placed on the left and right sides, respectively, and on the lower side of the square frame, the third image showing the third area 62 is placed on the left side. and are placed on the right, respectively.

A plurality of areas may be arranged on the display 14 as an image with the same width in one frame. To this end, the control unit can crop the captured photo, correct its size, and provide it to the display 14. At this time, the corrected image is displayed at the corresponding position of the frame.

At least some of the images provided on the display are corrected to have the same width, so that the user has a feeling similar to looking at the storage room with the actual refrigerator door open, which can have a two-dimensional arrangement effect.

The display 14 can provide two overlapping areas through one screen, providing food information in a state that the actual user cannot see at once.

In particular, since the second image and the third image are spaces arranged to overlap when the drawer is pulled out, the user cannot obtain information from the two storage spaces at the same time when the drawer is pulled out. However, the display 14 can provide two storage space information simultaneously.

The control unit 100 can individually update (replace) each image and provide the user with information about the type of food contained in each area.

For example, if the part corresponding to the second image is upgraded through the photo taken by the camera, and the part corresponding to the third image does not need to be upgraded, only the second image can be replaced.

On the contrary, if the part corresponding to the third image is upgraded through the photo taken by the camera, and the part corresponding to the second image does not need to be upgraded, only the third image can be replaced.

It is possible for the second and third images in one frame to be upgraded independently of each other and unrelated to other images.

Also, among the two second images placed on the left and right, only the second image placed on the left may be upgraded, or only the second image placed on the right may be upgraded. At this time, the third image can also be upgraded separately on the left and right sides.

That is, the display 14 provides a frame in which a plurality of images are provided to provide the user with information about items stored in the storage room, but it is possible for the plurality of images to be individually upgraded regardless of other images. In particular, because the second image and the third image are images selected from different photos, the control unit can determine which area the photo represents and upgrade each image.

At this time, the second image and the third image may have the same or substantially similar width. That is, it is possible for the portions corresponding to the second and third images obtained from the photo to be converted to the same width through correction by the control unit and provided on the display.

If the sizes of the second image and the third image are different, it is possible to select different sizes according to the number of pixels, that is, pixels, of the second image and the third image in the photo. Otherwise, it is possible to select the second image and the third image with the same number of pixels and correct them in the control unit so that they have different sizes when displayed on the display 14.

It is possible to arrange the second image and the third image vertically to have a sense of flatness in a vertical arrangement. Originally, the area shown by the second image is placed above the area shown by the third image. Therefore, the user can intuitively perceive the image provided by the display and easily understand the information provided by the image.

Of course, the display 14 can provide the user with a screen in which the first image and the third image are simultaneously replaced. In contrast, a screen in which only the second image is replaced can be provided to the user while the first image and the third image remain the same.

The refrigerator compartment shelf shown in FIG. 12A may refer to the first area 42, the vegetable compartment may refer to the second area 52, and the multi-storage corner may refer to the third area 62. Of course, it is possible to change it into various arrangements and forms depending on the user's preference or the convenience of the image provider.

Meanwhile, the screen of FIG. 12A may be provided on the display 14 installed in the refrigerator, but may also be provided on a separate external device other than the refrigerator, for example, an external terminal device such as a smartphone. Therefore, users can obtain information about food stored in the refrigerator while leaving home and use this information for shopping. Therefore, the user can determine the type of food stored in the refrigerator even without opening the door. The screen provided to the user may display one storage room divided into a plurality of storage areas. Therefore, users can easily obtain information on food stored in each location in one storage room classified by location.

At this time, the image shown in the vegetable compartment shows the area from the front end of the drawer 50 to the front end of the shelf 40 in the upper and lower directions when the drawer 50 is pulled out. The image shown in the multi-storage corner shows the overlapping area when the drawer 50 is retracted without being obscured by the door 20. The shelf shows an area above the shelf 40 to the user.

In one embodiment of the present invention, the screen provided to the user does not show a single storage room as is, but separate images are placed in separate parts, so the user can easily obtain information. In other words, rather than using the photo shown in FIG. 7 as is, each area in one photo is separated and used independently and individually.

As shown in FIG. 12B, a screen of a different type from FIG. 12A may be provided. In this case, it is possible to take a picture including the second area 52 and the third area 62 with the camera 70. Otherwise, even if the camera 70 takes a photo including all of the first area 42, the second area 52, and the third area 62, the user may receive an image corresponding to the two areas. It is possible to provide only

Another type of screen may be provided as shown in FIG. 12C. In this case, it is possible to take a picture including the first area 42 and the second area 52 with the camera 70. Otherwise, even if the camera 70 takes a photo including all of the first area 42, the second area 52, and the third area 62, the user may receive an image corresponding to the two areas. It is possible to provide only

That is, according to one embodiment of the present invention, it is also possible to provide related images to the user after photographing all three areas. However, as shown in FIGS. 12b and 12c, it is possible to provide the user with at least an image corresponding to the internal storage area of the drawer, that is, the second area, and an image corresponding to an area other than the second area.

Figure 13 is a diagram explaining a method of adjusting photos taken by a camera. Hereinafter, the description will be made with reference to FIG. 13.

Figure 13 may be an example of a screen displayed on the display 14. Of course, if the refrigerator is not equipped with a display, FIG. 13 may represent a screen displayed on a separate display device of the worker when manufacturing the refrigerator. It may also be a screen displayed on the user's mobile phone.

When the camera 70 is installed, the camera 70 may not be installed exactly in the desired location due to assembly tolerances.

Therefore, in an embodiment of the present invention, it is possible for an operator or user to change the adjustment line 15 that divides each area through a photo or video taken by the camera 70.

The adjustment line 15 may include a vertical adjustment line and a horizontal adjustment line. Vertical adjustment lines can be placed in the center of the image or at both ends. The user can select the part where the image provided on the display 14 is to be cut by moving the vertical adjustment line up and down and the horizontal adjustment line left and right using buttons shown on the screen.

Meanwhile, a limit line 16 that limits the movable range of the adjustment line 15 may also be shown. At this time, the limit line 16 can be arranged at predetermined intervals to the left and right around the vertical adjustment line. Additionally, the limit line 16 can be arranged at predetermined intervals above and below the horizontal adjustment line.

By providing the limit line 16, if the desired degree of image is not obtained despite the movement of the adjustment line 15, it is determined to be defective and the installation position of the camera 70 can be adjusted.

In a configuration in which two drawers are provided in the storage compartment as shown in FIG. 13, the vertical adjustment line can be placed between the two drawers.

And the horizontal adjustment line can be placed at the front end of the shelf.

When the movement of the horizontal adjustment line and the vertical adjustment line is completed, the lower left corner of the intersection of the horizontal adjustment line and the vertical adjustment line may form an image of the left side of the second or third area. At this time, the image may be from the horizontal adjustment line to a specific number of pixels in the horizontal direction, and may be from the vertical adjustment line to a specific number of pixels in the vertical direction.

The lower right corner of the intersection of the horizontal adjustment line and the vertical adjustment line may form an image of the right side of the second or third area. At this time, the image may be from the horizontal adjustment line to a specific number of pixels in the horizontal direction, and may be from the vertical adjustment line to a specific number of pixels in the vertical direction.

The upper side of the horizontal adjustment line may constitute an image of the first area. At this time, the image may range from the horizontal adjustment line to a specific number of pixels in the vertical direction.

Meanwhile, as shown in FIG. 13, it is also possible to check whether a photo of the portion corresponding to the drawer is obtained by separately showing the left and right photos of the third area corresponding to the vegetable compartment.

When the adjustment by the user or operator is completed, the input completion button can be clicked to input that the movement of the adjustment line 15 has been completed. Therefore, when the camera installation error is small, it is possible to adjust the position of the vertical and/or horizontal lines that separate each area in the captured photo. Through this, parts corresponding to a specific area can be clearly displayed. Of course, if there are multiple areas divided up and down or divided left and right, a plurality of the above-mentioned horizontal adjustment lines or vertical adjustment lines may be provided.

FIG. 14 is a perspective view showing the camera, FIG. 15 is a view showing main parts of the camera, and FIG. 16 is a cross-sectional view of FIG. 14. Hereinafter, description will be made with reference to FIGS. 14 to 16.

The camera 70 may include a camera module 71 including a camera lens 71a and a body 71b, and cases 73 and 74 in which the camera module 71 is accommodated.

The camera module 71 may include an image sensor that converts image information passing through the camera lens 71a into a digital signal.

The case may include a first case 73 and a second case 74 that is coupled to the first case 73 to form a predetermined space therein. The camera body 71b and the camera lens 71a can be accommodated inside the first case 73 and the second case 74. The degree of coupling between the first case 73 and the second case 74 can be maintained by a separate sealing member to prevent moisture or dust from entering the interior.

Since the camera body 71b and the camera lens 71a are inserted into the space formed by the first case 73 and the second case 74, the inner case 12 and the outer case 10 ) does not come into contact with the foaming agent filled between the Additionally, the camera module 71 can be protected without direct contact with the cold air contained in the storage compartment 22.

The first case 73 may be provided with a transparent window 80 installed at the front of the camera lens 71a and a heater 84 that provides heat to the transparent window 80. The transparent window 80 is preferably made of a transparent material.

The heat provided by the heater 84 can prevent dew from forming on the transparent window 80.

If the heater 84 is installed on the camera lens 71a to prevent dew formation, the camera lens 71a may be deformed. Since the possibility of occurrence of aberration and permanent damage due to such deformation increases, in the embodiment of the present invention, the transparent window 80 is not brought into direct contact with the camera lens 71a.

Both plastic and glass can be used as a material for the transparent window 80 in terms of camera image acquisition, but plastic has poor thermal conductivity. Therefore, when plastic is used as the transparent window 80, power consumption may increase compared to when glass is used. Therefore, it is desirable that the transparent window 80 is made of glass to reduce power consumption.

As the thermal conductivity increases, the heat supplied by the heater 84 can be easily conducted from the transparent window 80 to parts that are not in direct contact with the heater 84. For example, when the thermal conductivity is lowered, the temperature is high in the area adjacent to the heater 84, while the temperature is low in the area away from the heater 84, making it difficult to prevent condensation. Additionally, because more heat energy must be supplied by the heater 84, energy efficiency decreases.

Meanwhile, since the heater 84 makes direct surface contact with the transparent window 80, heat transfer efficiency can be improved.

The heater 84 is disposed on the opposite side of the side where the transparent window 80 is exposed to the storage compartment 22, so that the heat generated by the heater 84 does not pass through the transparent window 80 and the storage compartment 22 Supply to (22) can be prevented.

In particular, the heater 84 may include a heating coil that generates heat, and the heating coil may make surface contact with the transparent window 80.

In addition, it is preferable that the transparent window 80 and the camera lens 71a are provided at a predetermined distance g. This is to take into account deformation that may occur when heat is supplied to the transparent window 80 and assembly tolerances.

Additionally, the camera 70 may receive electricity supplied from the outside and include a cable 79 that transmits signals related to the acquired photos. The cable 79 may electrically connect the camera module 71 to external components.

Meanwhile, the case is provided with a first seating piece 75 that can secure the camera 70 to other components. At this time, it does not matter whether the first seating piece 75 is provided in the first case 73 or the second case 74.

The first seating piece 75 includes a surface having a predetermined area, so that the camera 70 can be placed at a desired installation location and at a desired angle.

The first seating pieces 75 are provided on both sides of the camera 70, so that the left and right sides of the camera 70 can be fixed by external components.

A second seating piece 76 may be formed in the second case 74 to have a predetermined area and to be stepped. The second seating piece 76 is provided in a different part from the first seating piece 75, so that, like the first seating piece 75, the camera 70 is positioned at a specific position and at a predetermined angle to other components. Ensure stable combination. Accordingly, the camera 70 can be fixed to have three or more support points at a predetermined angle with respect to other components through the first seating piece 75 and the second seating piece 76. Details about the support structure of the camera 70 will be described later.

The camera 70 can use VGA Resolution level resolution, and VGA (640x480), 300,000 pixel level resolution can be used considering image data transmission speed and processing time.

The interface of the camera 70 is USB Type and can be driven using 5V, 500mA USB power.

Additionally, the current value supplied to the camera 70 may be 87 mA in standby mode and 187 mA in operation mode (67 mA for heater). Therefore, power consumption proportional to the supplied current value can be consumed. At this time, the operation of the camera 70 is divided into a standby mode and an operation mode, and power can be applied to the heater 84 at all times to prevent condensation on the surface of the transparent window 80.

Meanwhile, the camera 70 operates in standby mode when the rated voltage is applied, and switches to the operation mode when the door switch 110 detects that the door 20 is open to take pictures. That is, when power is applied to the refrigerator, the camera operates in standby mode, and when the door switch 110 detects that the door 20 is open, it can be switched to an operation mode for taking pictures.

Figure 17 is a diagram explaining the arrangement of the heater. Hereinafter, the description will be made with reference to FIG. 17.

The camera 70 can have a horizontal viewing angle of 115°, a vertical viewing angle of 95°, and a diagonal viewing angle of 145°. That is, the camera 70 can be arranged so that its horizontal angle of view is larger than its vertical angle of view. Through this arrangement, when the width direction length (horizontal direction in FIG. 17) of the storage compartment 22 is longer than the front-back length (vertical direction in FIG. 17) of the storage compartment 22, food information about the storage compartment 22 You can obtain photos that can provide .

Meanwhile, the heater 84 can be formed along the outline of the transparent window 80. At this time, the heater 84 may be arranged to form a square so as not to block the angle of view of the camera 70.

Additionally, when the heater 84 has a square shape, heat can be supplied to the transparent window 80 over a relatively large area, so energy efficiency for the heater 84 can be improved.

In particular, when a gap g is provided between the camera lens 71a and the transparent window 80 as shown in FIG. 16, the angle of view with respect to the transparent window 80 takes a rectangular shape as shown in FIG. 17. If there is no gap g and the transparent window 80 and the camera lens 71a are in contact with each other, the heater 84 contacts one surface of the transparent window 80, so the camera lens 71a ) can be provided in a square shape with sides of length equal to or greater than the diameter of ).

However, as described above, since there is a gap g and the vertical and horizontal angles of view are different from each other, it is preferable that the heater 84 has a rectangular shape with one side longer than the other.

Figure 18 is a diagram explaining a picture taken with the camera mounted on the refrigerator in a vertical direction (front-back direction), and Figure 19 is a diagram explaining a picture taken with the camera mounted in a horizontal direction (left-right direction) on the refrigerator. It is a drawing. Hereinafter, it will be described with reference to FIGS. 18 and 19.

Figure 18a shows the camera turned upward by 5 degrees, Figure 18b shows the camera installed at the desired position, and Figure 18c shows the camera turned downward by 5 degrees. A photo of the size shown in FIG. 18 can provide the user with information about food stored in the pantry, but if it is tilted at a larger angle, it cannot provide the user with the desired information.

Figure 19a shows the camera turned 4 degrees to the left, Figure 19b shows the camera turned 2 degrees to the left, and Figure 19c shows the camera installed at the desired position.

Compared to a photo in which the camera is installed to rotate up and down as shown in FIG. 18, a photo in which the camera is not installed at a desired horizontal angle as shown in FIG. 19 provides a relatively greatly distorted screen.

As can be seen in Figures 18 and 19, even if there is a slight difference in the installation angle of the camera, it can lead to significant distortion of the captured photo, so it is necessary to manage the assembly tolerance in the installation angle of the camera so that it occurs as little as possible. Of course, minor errors can be adjusted through the adjustment line illustrated in FIG. 13, making it possible to provide an optimal photo.

FIG. 20 is a view showing the camera housing in an assembled state, FIG. 21 is a front view of the first housing, FIG. 22 is a front view of the second housing, and FIG. 23 is a front view showing a camera installed in the first housing. 24 is a side cross-sectional view of FIG. 23, and FIG. 25 is a cross-sectional view showing the camera housing installed in the inner case. Hereinafter, description will be made with reference to FIGS. 20 to 25.

In order to install the camera 70 in the inner case 12, a camera housing may be provided. That is, a camera housing can be provided to accommodate the camera shown in FIG. 14 and to fix the camera to the ceiling of the storage room.

The camera housing includes a first housing 400 coupled to the lower surface of the ceiling of the inner case 12, a second housing 410 coupled to the upper surface of the ceiling of the inner case 12, and the first housing 400. ) may include a third housing 420 and a fourth housing 430 coupled to the. Here, the first housing 400 can be said to be coupled to the upper surface of the ceiling of the storage room.

The third housing 420 may close the front of the camera 70 so that parts other than the transparent window 80 of the camera 70 are not directly exposed to the storage compartment 22. That is, the third housing 420 can function as a cover that covers the front of the camera 70. At this time, the third housing 420 and the fourth housing 430 may be formed as one component. Of course, the fourth housing 430 may be a deco housing interposed between the third housing 320 and the first housing 400.

The camera 70 is assembled in the first housing 400 to manage the left and right tilt degree (horizontal tilt degree) and front and rear tilt degree (vertical tilt degree) of the camera 70. You can.

As shown in FIG. 21, the first housing 400 may be provided with a first seating portion 402 on which the first seating piece 75 of the camera 70 is seated. At this time, two first seating portions 402 are provided so that the left and right first seating pieces 75 can be fastened to each other.

The first seating piece 75 is provided to be inclined at a predetermined angle on the left and right sides of the camera 70, so that it can be arranged to be inclined at a desired angle while the camera 70 is installed in the inner case 12. .

A fastening hole 404 is provided in the first seating portion 402, so that the first seating piece 75 and the first seating portion 402 can be fixed with a screw. The first mounting portion 402 is arranged at different heights in the horizontal direction, so that the camera 70 can be stably fixed.

The first housing 400 may be provided with a second seating portion 408 on which the second seating piece 76 of the camera 70 can be seated. At this time, the second seating portion 408 is also formed to have a predetermined area, so that the degree of inclination of the camera 70 can be fixed while making surface contact with the second seating piece 76.

That is, the camera 70 is coupled to the first housing 400 by contacting it through a plurality of surfaces, so that the camera 70 can be fixed to be tilted at a desired angle. In other words, it can be fixed to have three or more support points and simultaneously supported at a predetermined angle with respect to the first housing 400. Of course, the horizontal angle can be fixed to be parallel to the first housing 400, and the vertical angle can be fixed to be a predetermined angle.

The first housing 400 may be provided with a fastening hole 406 so that it can be coupled to the second housing 410 with a screw.

As shown in FIG. 22, the second housing 410 has a contact surface 416 that can be brought into surface contact with the upper surface of the ceiling of the inner case 12. At this time, the contact surface 416 has an overall square shape and is widely distributed, so that the second housing 410 can be stably seated on the upper surface of the ceiling of the inner case 12. Therefore, the second housing ( The contact surface 416 of 410 has a surface parallel to the ceiling of the inner case 12. Through this, the baseline or reference plane for the installation angle of the camera can be made the same as the ceiling surface of the inner case 12.

The second housing 410 may have an accommodation space 412 in which the cable 79 of the camera 70 can be accommodated. A through hole 418 is formed on one side of the receiving space 412, so that wires connected from other components of the refrigerator can be exposed to the receiving space 412. At this time, electric wires connected from other components of the refrigerator may be connected to the cable 79 in the accommodation space 412. Accordingly, electricity can be supplied to the camera 70 through the cable 79, and signals for pictures taken by the camera 70 can be transmitted to other components.

The second housing 410 may be provided with a fastening portion 414 coupled to the first housing 400. The fastening part 414 may be coupled to the fastening hole 406 with a screw. It is preferable that errors due to assembly do not occur when the fastening part 414 is coupled with the fastening hole 406. Therefore, it is also possible to be formed to protrude by a predetermined height so as to make surface contact with the fastening hole 406.

As shown in FIGS. 23 and 24 , when the camera 70 is assembled to the first housing 400, the camera 70 may be arranged in an inclined form by a set angle. As shown in FIG. 24, one side 400a of the first housing 400 is shaped to be flat so that it can be in contact with the inner case 12. The camera 70 is inclined at a predetermined angle compared to one side 400a of the first housing 400. And the one side surface 400a is coupled to the ceiling bottom of the inter case 12 so as to be parallel with the contact surface 416 of the second housing 410. Therefore, as the camera 70 is coupled to the first housing 400 and the first housing 400 is coupled to the second housing 420, the installation angle of the camera 70 with respect to the ceiling surface of the storage room The error can be significantly reduced.

Referring to FIG. 25, the second housing 410 is disposed to be exposed to the upper part of the inner case 12, that is, the space between the inner case 12 and the outer case 10 where the foaming liquid is filled. do. The second housing 410 is placed outside the storage compartment 22, but is placed in a space defined by the outer case 10. After the second housing 410 is glued to the inner case 12 and its position is temporarily fixed, it is finally fixed by the foaming liquid filled between the inner case 12 and the outer case 10. You can.

The first housing 400 may be coupled to the second housing 410 with the inner case 12 interposed therebetween. When the second housing 410 is fixed to a specific position of the inner case 12, the position of the camera 70 is changed because the first housing 400 is coupled to the second housing 410. It can be fixed.

With the camera 70 tilted at a specific angle to the first housing 400, the third housing 420 and the fourth housing 430 may be coupled to the second housing 410. . As described above, the third housing 420 and the fourth housing 430 are composed of one component, and a component that performs the function of the third housing 420 is added to the first housing 400. It is also possible to just combine them.

The third housing 420 may allow only the transparent window 80 of the camera 70 to be exposed to the storage compartment 22 and prevent other parts from being exposed. Therefore, it is possible to prevent the camera 70 from being affected by moisture, dust, cold air, etc. present in the storage compartment 22.

The structure of the camera housing, the installation direction and installation form of the camera housing with respect to the inner case, etc. also have technical significance for reducing condensation occurring on the transparent window 80. Related specific explanations will be provided later.

Figure 26 is a table comparing the power consumption of the camera in the present invention, and Figure 27 is a diagram comparing the supplied standby current and driving current. Figure 26a shows a state in which current is supplied to the camera in the comparative example, and Figure 27b shows a state in which current is supplied to the camera in the present invention. Hereinafter, it will be described with reference to FIGS. 26 and 27.

In the comparative example, 70 mA of standby current is supplied to the camera 70, but when the door is opened, 50 mA is additionally supplied, so that a driving current of 120 mA is supplied to the camera 70. In the comparative example, if the camera 70 is opened 25 times a day, 253 Wh is consumed per month, and if it is opened 50 times a day, 254 Wh is consumed per month.

On the other hand, in the embodiment of the present invention, 20 mA of standby current is supplied to the camera 70, and 100 mA is additionally supplied at the time the door is opened, so that a driving current of 120 mA can be supplied to the camera 70. If the camera 70 is opened 25 times a day, 74 Wh is consumed per month, and if it is opened 50 times a day, 76 Wh is consumed per month.

Comparing the two experimental results, it can be seen that the power consumed by the camera 70 can be reduced by approximately 70% by lowering the standby current. Therefore, it was confirmed that the power consumed by the camera 70 can be significantly reduced by lowering the standby current under the premise that the driving current is the same. This is because the time the door is open is relatively very short when the refrigerator is operated continuously for 24 hours. Therefore, assuming that the current value required for shooting in the operating mode is constant, the lower the current value applied to the camera in the standby mode, the lower the current consumption.

If the standby current is lowered, the timing at which the driving current is supplied to the camera 70 is delayed, as shown in FIG. 27. This is because the delay time inevitably increases due to the difference between the standby current value and the driving current value. In other words, when switching from standby mode to drive mode, photo taking is not performed immediately, but a predetermined delay time must elapse before the photo is taken.

Therefore, the time at which a picture is actually taken by the camera 70 is delayed. If there is a large difference between the time when the command to take a picture is delivered to the camera 70 and the time when the picture is actually taken by the camera 70, the picture actually taken by the camera 70 provides food information to the user. It may not be provided.

In other words, the lower the standby current, the lower the power consumed by the camera 70. However, the delay time for increasing the current to drive the camera 70 may be longer. Therefore, it is desirable to calculate the normal time that elapses for the user to be able to take a picture at the desired time and select the standby current value so that the delay time is shorter than the corresponding time. For this reason, according to an embodiment of the present invention, it is desirable to gradually increase the applied current from the standby current value to the driving current value. That is, the applied current value can be gradually increased from a relatively low standby current value to a relatively high driving current value with sufficient delay time. In other words, a sudden increase in the current value can be prevented. Therefore, it is possible to reduce power consumption and improve camera stability.

Therefore, in the embodiment of the present invention, as shown in FIGS. 28 and 29, the camera 70 is controlled to start operating at the time the door 20 is opened. That is, when it is determined that the door is opened through the door switch, it switches from standby mode to drive mode. However, among the photos taken by the camera, valid photos are taken after at least a predetermined time has elapsed from the time the door is opened. This is because it physically takes more than a few seconds for the door to open and the user to update the food. In other words, even if the delay time is long, photos can be obtained at the desired time. Therefore, in this embodiment, it can be said that the transition point from the camera's standby mode to the drive mode is performed based on the detection result of the door switch. And, it can be said that the reference point for selecting a valid photo is performed based on the detection result of the door sensor.

Figure 28 is a diagram explaining the starting point of the camera's photo taking and the camera's continuous shooting. Hereinafter, description will be made with reference to FIG. 28.

Meanwhile, the present invention may include the door switch, as shown in FIG. 4. When the door switch 110 detects the opening of the storage compartment 22, the control unit 100 may command the camera 70 to start taking pictures. That is, when the door 20 is moved or rotated to open the storage compartment 22, the camera 70 can start taking pictures.

On the other hand, when the door switch 110 detects that the storage compartment 22 is closed, the control unit 100 may command the camera 70 to stop taking pictures. That is, when the door 20 moves or rotates and the storage compartment 22 is sealed, photo taking with the camera 70 can end.

As shown in FIG. 28, when electricity from an external power source is supplied to the refrigerator, standby current is supplied from the camera 70 (S1). At this time, the camera 70 may be driven in standby mode. In standby mode, the camera 70 does not take pictures.

Opening of the door 20 may be detected by the door switch 110 (S10).

When it is determined that the door 20 opens the storage compartment 22, the camera 70 can start taking pictures (S14). In particular, the camera 70 can continuously take pictures at predetermined time intervals.

At this time, since a current greater than that supplied in standby mode is supplied to the camera 70, the camera 70 can be operated in drive mode. In order to supply a current greater than the standby current or driving current to drive the camera 70, a certain amount of time, that is, a delay time, inevitably occurs. That is, this is because a certain amount of time is actually required for the size of the current supplied to the camera 70 to increase. After a predetermined delay time has elapsed after the current for performing the driving mode is supplied to the camera 70, the driving current is supplied to the camera 70 so that the camera 70 can finally take pictures. .

Since the camera 70 switches to a driving mode for taking pictures from the time the door 20 is opened, the starting point of taking pictures is early, thereby reducing the risk of not being able to take pictures at a specific time. If you supply standby current before the desired time to take a picture and then increase the current value at the desired time, the time to take the picture will be delayed due to the delay time.

Additionally, the number of pictures taken per second by the camera 70 can be reduced. Assuming that the camera 70 takes 30 pictures per second, the possibility of changes inside the storage compartment 22 occurring during the time interval during which pictures are taken, that is, 1/30 second, is extremely small. The interior of the storage compartment 22 can be changed by the user's actions, because a typical person is unlikely to complete an action in the storage compartment 22 for 1/30 of a second. That is, the time interval between images taken by the camera 70 can be relatively increased. Accordingly, the relatively inexpensive camera 70 can be used, thereby reducing the cost of the refrigerator.

A photo taken by the camera 70 may be stored in the storage unit 18 (S16). Additionally, as the number of photos taken per second by the camera 70 decreases, the number of photos stored in the storage unit 18 also decreases, allowing the storage unit 18 to be operated more efficiently.

Meanwhile, the door sensor 120 can detect whether the door 20 is rotated at a specific angle (S18).

If the door sensor 120 does not detect whether the door 20 is rotated, photos stored in the storage unit 18 that are older than a predetermined time can be deleted (S90). That is, photos that are not processed to be provided to the user are deleted, thereby preventing the capacity of the storage unit 18 from increasing.

On the other hand, if the door sensor 120 determines that the door 20 has been rotated at a specific angle, a photo taken at that time is selected (S92). At this time, the selected photo may be processed by the control unit 100 as a valid photo and provided to the user. For example, a photo processed by the control unit 100 may be transmitted to the display 14.

In order for a user to access the storage compartment 22, a certain amount of time is required to rotate the door 20. Since the delay time of the camera 70 elapses during that time, a picture can be taken when the door 20 is rotated at a specific angle. Therefore, the control unit 100 can select a photo at a desired time.

If the camera 70 switches from standby mode to drive mode to take a picture at the time detected by the door sensor 120, there may be a problem that the state of the storage room after the delay time has elapsed from that point has no choice but to be photographed. You can.

Meanwhile, since the camera 70 starts shooting when the door 20 is opened, it is possible for a relatively small amount of standby current to be supplied to the camera 70. If a small standby current is supplied, the delay time until the driving current is supplied increases, but in the above-described control flow, the camera 70 starts taking pictures early, so there is no problem of not being able to take pictures at the desired time. .

Figure 29 is a diagram explaining the starting point of the camera's photo taking and the camera's continuous shooting. Hereinafter, the description will be made with reference to FIG. 29.

In FIG. 29 , unlike FIG. 28 , there is a difference in that the photo selection time is not determined according to the rotation state of the door 20, but rather according to the state of the drawer 50. Other parts are the same, and the same parts are omitted for convenience of explanation. However, it can be said that operation of the drawer 50 is premised on opening the door 20. The reference time for selecting a specific photo may be determined by the judgment of the door sensor, and the reference time for selecting a photo for a specific area within a specific photo may be determined by the judgment of the door sensor and the judgment of the drawer detection unit 130. You will be able to. Therefore, in any case, the decision result of the door sensor is required to determine the reference point of time for a valid photo.

In order to store food in the drawer 50 or take food stored in the drawer 50 out, the user can pull out the drawer 50. At this time, in order to use the drawer 50, the user can pull out the drawer 50 by a sufficient length.

After the user completes using the drawer 50, the drawer 50 can be pushed in. The point in time when this state occurs may mean a specific state described in S19.

Since the camera 70 starts taking pictures from the time the door 20 is opened at the door switch 110, if the drawer 50 is placed in a specific state, the picture taken at that time is taken. It can be selected and provided to the user (S92).

When the door 20 is opened, a picture is taken by the camera 70, and the movement of the drawer 50 can be monitored from the time the picture is taken, that is, the time the door 20 is opened.

FIG. 30 is a diagram illustrating an embodiment of a drawer detection unit, and FIG. 31 is a diagram explaining a method of detecting movement of a drawer in the drawer detection unit of FIG. 30. Hereinafter, it will be described with reference to FIGS. 30 and 31.

Referring to FIG. 30, it is possible to implement the drawer detection unit 130 as a separate component rather than including it in the control unit 100.

The drawer detection unit 130 may have one light emitting unit and two light receiving units. That is, the moving direction of the drawer 50 can be detected by analyzing the point in time when light emitted from one light emitting unit enters the two light receiving units.

A plurality of slits 134 may be successively arranged in the drawer 50 to prevent light emitted from the light emitting unit of the drawer detection unit 130 from being reflected when incident on the slit 134. Therefore, light can be sequentially incident on the two light receivers.

In FIG. 31, the x-axis represents time, and the y-axis represents a signal when light is received from two light receivers (Photo TR1 and Photo TR2). That is, in Figure 31a, light is received from Photo TR1 before Photo TR2, and in Figure 31b, light is received from Photo TR2 before Photo TR2.

In Figure 31b, it means a moving state in which the user withdraws the drawer 50 to use it, and in Figure 31a, the user finishes using the drawer 50 and returns the drawer 50 to its original position. It can mean a state.

Therefore, when the drawer detection unit 130 detects a signal as shown in FIG. 31A, the control unit 100 determines that it is time to acquire a photo taken by the camera 70, and at that point, the drawer 50 You can select a photo corresponding to the inside, that is, the third area. That is, the acquisition time of the photo corresponding to the third area may be determined according to the detection or judgment result of the drawer detection unit 130. Of course, in the photo obtained at this time, the part corresponding to the third area can be selected, separated, and updated independently and individually from the part corresponding to other areas.

Figure 32 is a diagram showing a state in which a marker is displayed on a drawer. Hereinafter, the description will be made with reference to FIG. 32.

Unlike FIGS. 30 and 31, the movement of the drawer 50 can be detected using a marker, which is a mark specific to the drawer 50, even without using a separate mechanical or electronic device. .

The drawer 50 may be provided with various types of markers (patterns having black and white colors). The camera 70 can analyze the movement of the marker through the captured photo to determine whether the drawer 50 is moving and the direction of movement. At this time, the marker analysis can be performed in the drawer detection unit 130 of the control unit 100. That is, the drawer detection unit 130 can determine the state of the drawer 50 by analyzing the movement of the marker in photos taken continuously. Accordingly, the acquisition time of the photo corresponding to the third area may be determined according to the detection or judgment result of the drawer detection unit 130. Of course, in the photo obtained at this time, the part corresponding to the third area can be selected, separated, and updated independently and individually from the part corresponding to other areas.

In Figure 32, the marker includes a simplified image of the shape of the camera and the words 'Smart Sensor'. Meanwhile, the marker may include various types of patterns, and may be provided in a form where black and white intersect each other, so that it can be recognized by the camera 70.

At this time, the marker may be displayed on the front upper side of the drawer 50 so that it can be recognized by the camera 70.

FIG. 33(a) is a view showing a photograph taken in a state in which the left and right drawers are pulled out, and FIG. 33(b) is a view showing a state in which the left drawer is pulled in and the right drawer is pulled out. Hereinafter, the description will be made with reference to FIGS. 33(a) and 33(b).

In FIG. 33, the marker is provided at the center of the upper side of the drawer 50, and an example of forming a wave-shaped pattern is shown.

The first area 42 may be expressed in one photo taken by the camera 70. Additionally, one of the second area 52 or the third area 62 may be shown in one photo.

That is, a photo taken with the left and right drawers pulled out as shown in FIG. 33(a) shows the first area 42 disposed on the left and right sides and the second area 52 disposed on the left and right sides.

On the other hand, in the photo taken with only the right drawer pulled out as shown in FIG. 33(b), the first area 42 arranged on the left and right, the third area 62 on the left, and the second area 52 on the right ) can be shown.

That is, when the drawer 50 is pulled out, the third area 62 is not shown in the photo, but the second area is shown. On the other hand, when the drawer 50 is retracted, the second area 52 is not shown in the photo and the third area 62 is shown.

The camera 70 takes pictures at the same location. The drawer 50 can be moved, and therefore pictures obtained by the camera 70 can capture pictures of various areas depending on the position of the drawer 50.

Meanwhile, as shown in FIG. 33(b), when the drawer 50 provided on the left is completely retracted into the lower space of the shelf 40, the marker is not recognized by the camera 70. That is, no marker is shown in the photo taken by the camera 70.

34 to 40, a method for detecting movement of a drawer using a marker will be described.

Figure 34 is a flowchart for explaining a method of operating a refrigerator according to an embodiment of the present invention.

Referring to FIG. 34, when the door switch 110 detects the opening of the door 20, the camera 70 starts taking a picture (S2101), and the control unit 100 performs a marker recognition operation and marker tracking. Start the operation (S2103).

Hereinafter, steps S2101 and S2103 will be described with reference to FIG. 35 in more detail.

Figure 35 is a flowchart for explaining a marker recognition and tracking method according to an embodiment of the present invention.

Referring to FIG. 35, when the door 20 is opened, the camera 70 starts taking pictures (S2301).

The control unit 100 controls the camera 70 to obtain a photo of the interior of the drawer 50 (S2303).

Next, the control unit 100 processes the acquired photo (S2305) and determines whether a predetermined marker is recognized in the acquired photo (S2307).

In order to determine whether a predetermined marker is recognized, the control unit 100 may compare whether a marker recognized from an acquired photo is the same as a prestored marker.

The predetermined marker may be determined by the business operator or user, and may be stored in the storage unit 18 in advance.

A marker may be formed in a searchable area in a photo taken from a camera. Below, with reference to FIGS. 36 and 37, markers that can be recognized and tracked by the control unit 100 will be described in detail.

Figure 36 is a diagram illustrating a marker for detecting when a drawer is withdrawn or inputted according to an embodiment of the present invention.

Referring to (a) of FIG. 36, a marker may be formed in an area that can be searched by the camera 70 from the time the drawer 50 begins to open. For example, when the camera 70 is located at the top of the refrigerator and can photograph the plane of the drawer 50, the marker may be located at the front end of the top of the drawer 50. Additionally, the field of view of the camera 70 may be installed such that it can capture images from a state in which the drawer 50 is fully closed to a state in which the drawer 50 is fully opened.

In the case where the marker is located at the top of the drawer 50, there is no obstruction at the top of the drawer 50 when the drawer 50 is closed after the storage inside the drawer 50 is completed. It is planned and is not limited to this.

Referring to (b) of FIG. 36, when looking at the drawer compartment 54 and the drawer 50 where the drawer 50 is stored from the perspective of the camera 70, the camera 70 looks at the drawer ( The inside of the drawer 50 including the marker can be photographed in a full range from the state in which the drawer 50) begins to open to the state in which the drawer 50 is fully opened.

Figure 37 is a diagram for explaining the shape of a marker according to an embodiment of the present invention.

Referring to FIG. 37, the marker may be formed in a form in which the same shape of two colors intersect and repeat.

Referring to (a) of FIG. 37, the marker may be formed in the form of a plurality of alternating bands of white squares and black squares arranged in an alternating manner.

Referring to (b) of FIG. 37, the marker may be formed in the form of a plurality of alternating bands of white slanted squares and black slanted squares arranged in an alternating manner.

In this way, the repetitive type of marker has the advantage of enabling the control unit 100 to recognize and track the marker even when part of the marker is obscured from the camera shooting range. However, markers are not limited to repetitive shapes, nor are they limited to multiple colors. As described above, the marker is sufficient to recognize whether or not a moveable internal component of the refrigerator, such as the drawer 50, is moved.

Referring again to FIG. 35, when a predetermined marker is recognized, the control unit 100 detects the movement of the recognized marker (S2311). The control unit 100 may continuously or periodically analyze and process acquired photos to determine whether the recognized marker is moving or stationary. For example, the control unit 100 may determine whether the marker moves or stops when the opening operation of the drawer 50 continues or is completed, or when the closing operation of the drawer 50 begins, continues, or is completed. .

Alternatively, if the predetermined marker is not recognized, the control unit 100 notifies the user that marker recognition has failed (S2309).

For example, if an obstacle is located in the marker, the camera may not be able to obtain a photo including the marker, or the control unit 100 may not be able to recognize the marker from the acquired photo. In this case, the control unit 100 may inform the user through the display 14 of information indicating that marker recognition has failed and information indicating a method for successful marker recognition, such as removing obstacles.

Subsequently, if marker recognition fails, the control unit 100 may repeatedly perform the step of processing a newly acquired photo through a camera and recognizing a predetermined marker.

Referring again to FIG. 34, after marker recognition, when a drawer closing operation is detected (S2105), the control unit 100 detects the inside of the drawer (the third area 62) through a photo taken at that time. Obtain the final photo (S2107). At this time, the final photo of the interior of the drawer obtained by the control unit 100 may be the interior of the drawer just before the drawer closing operation is detected or the interior of the drawer immediately after the drawer closing operation is detected, It is not limited to this.

Whether or not the drawer closing operation is detected will be explained with reference to FIG. 38.

Figure 38 is a flowchart to explain a method of detecting a drawer closing operation according to an embodiment of the present invention.

Referring to FIG. 38, the control unit 100 determines whether the end of the opening operation of the drawer 50 has been detected (S2501). For example, the control unit 100 recognizes the marker from a previously acquired photo of the drawer 50, tracks the movement of the marker, and when detecting that the marker has stopped moving, the opening operation of the drawer 50 is performed. You can detect that it has ended.

Next, the control unit 100 determines whether the start of the closing operation of the drawer 50 has been detected (S2503). For example, the control unit 100 may detect that the closing operation of the drawer 50 has started when the marker stops moving and then starts moving again in the opposite direction.

Meanwhile, the operation of the control unit 100 that detects a drawer closing operation related to another embodiment of the present invention will be described with reference to FIG. 39.

Figure 39 is a flowchart for explaining the operation of the refrigerator performed when the completion of the drawer closing operation is detected according to an embodiment of the present invention.

When the end of the drawer closing operation is detected (S2701), the control unit 100 determines whether the final photo has been successfully obtained through marker recognition (S2703).

The control unit 100 can detect that the closing operation of the drawer 50 has ended through image processing by the drawer detection unit 130.

If acquisition of the final photo fails as a result of the marker not being recognized properly due to an obstacle, etc., the control unit 100 notifies the user that marker recognition has failed (S2705).

The control unit 100 may inform the user of information that marker recognition has failed through the display 14 or the like. This can guide the user to remove obstructions and guide the user to withdraw and retract the drawer for normal marker recognition and final image acquisition.

Referring again to FIG. 34, the control unit 100 displays the final image using the display 14, etc. (S2109). At this time, the form provided to the user may be an excerpt of only the portion corresponding to the third area 62 from the obtained final photo. Next, with reference to FIG. 40, final image control according to another embodiment of the present invention will be described.

Figure 40 is a flowchart for explaining a method of controlling an image of a specific area inside a refrigerator taken at a specific time according to an embodiment of the present invention.

Referring to FIG. 40, the control unit 100 can obtain the final photo (S2901) and crop the image corresponding to the desired area from the final photo. And the obtained final image is stored in the storage unit 18 (S2903).

Next, the control unit 100 determines whether user input for displaying the final image is received through user input, etc. (S2905).

For example, when acquiring the final photo, the control unit 100 may form a user input unit in the form of an icon along with information indicating the success of acquiring the final photo on the display 14 configured as a touch screen, and the user may determine whether the icon is selected or not. It is possible to determine whether input is received. The user input unit may include, but is not limited to, physical keys as described above in addition to icons formed on the touch screen.

Next, when the control unit 100 receives a user input for displaying the final image processed from the final photo, it displays the final image (S2907). The control unit 100 may display the final image using the display 14. This display of the final image can be said to replace the existing image. Therefore, it can be said to be an update of the final image.

Of course, this update of the final image may be performed automatically regardless of user input.

According to this embodiment of the present invention, the refrigerator can detect whether the drawer, etc. is moving by recognizing a marker previously formed on the drawer, etc. Additionally, the refrigerator can obtain the final state of stored items stored in drawers, etc. in the form of images using a camera installed inside the refrigerator. At this time, the refrigerator can determine the optimal time to obtain the final storage state of the drawer, etc., using the marker formed on the drawer, etc.

Hereinafter, the installation location of the camera and the photos that can be obtained accordingly will be described.

An embodiment of the present invention installs the camera 70 outside the drawer 50, that is, in the storage compartment 22, so that the inside of the drawer 50 and the drawer body forming the drawer 50 (i.e. By continuously taking photos including the movement path (including the front and back walls, the top of both walls, etc.), not only the storage status of items inside the drawer but also the movement status of the drawer can be recognized at the same time. Of course, the picture taken by the camera 70 is also taken up to the upper space of the drawer 50, that is, the part corresponding to the first area 42.

Basically, photos taken by a camera must be able to include both the opening and closing motion of the drawer and the path the drawer moves. Therefore, the camera must be installed in a location that can film the opening, closing, and movement path of the drawer from outside the drawer. In other words, it is preferable that the camera be installed in a location that can take a single picture of at least both the closed state and the fully open state of the drawer.

Furthermore, the camera 70 captures not only the drawer area but also another storage area of the storage room, for example, a shelf area divided by a shelf where food or items are stored, all included in one photo with one shot. Thus, one camera can be configured to cover other storage areas including the drawer area, that is, multiple storage areas.

For this purpose, the drawer 50 is preferably formed on the bottom of the refrigerator storage compartment. The drawer 50 may be composed of a front wall, two side walls, a rear wall, a bottom wall, and an opening at the top. The camera 70 must be provided at the top of the drawer 50 so that at least a portion of the interior of the drawer and the upper part of the front wall of the drawer when the drawer is fully open are exposed.

Then, a printed portion (marker) made of a specific pattern is formed or attached at any position on the upper surface of the front wall or both sides of the drawer that is exposed to the camera.

At this time, since the camera 70 is fixed at a predetermined position, it is necessary to configure the position of the printing part of the drawer 50 to change relatively. That is, when the camera 70 receives a driving command from the control unit, it continuously takes pictures at a constant speed and sequentially transmits the captured internal images to the control unit 100. The control unit 100 tracks the location of the printing unit from a photo of the storage of items in the interior received from the camera 70 and determines information about the open state of the drawer as well as the movement of the drawer.

The shooting range of the camera can be determined by the camera's angle of view, which consists of a vertical angle of view, which refers to the shooting range in the vertical direction, and a horizontal angle of view, which determines the shooting range in the horizontal direction.

One end of the vertical angle of view of the camera 70 must include at least the front wall of the drawer when the drawer is fully open, and the other end must include a print located on the upper surface of the front wall or both walls of the drawer when the drawer is closed. It should be decided to include the part where it can be confirmed that the part is not exposed, for example, the part of the front of the upper side of the drawer that covers the upper opening of the drawer.

At this time, it is possible to simultaneously recognize the inner area of the drawer and another storage space with one camera. Another embodiment of the present invention provides a separate storage space (hereinafter referred to as Storage compartment on the floor of the storage room) can be configured. In this case, the vertical angle of view of the camera 70 can be configured to recognize another storage space (floor storage compartment) including the inner area of the drawer even in the shooting range that includes the drawer fully open and drawer fully closed states. It may be possible.

Meanwhile, in another embodiment of the present invention with respect to the vertical angle of view of the camera 70, the interior area of the drawer and the shelf area can be photographed simultaneously. Therefore, the range of the other end of the vertical angle of view can be determined to include any part of the plurality of shelves arranged at regular intervals above and below. In order for the camera vertical angle to include the shelf area, a preferred example may include up to a portion of the uppermost shelf among the plurality of shelves, and more preferably, at least the uppermost shelf disposed at the top of the storage compartment. This is to include the rear end.

According to these embodiments, it is possible to check the storage status of items in at least two storage spaces, including the inner area of the drawer, with a single photo taken with one camera.

For this purpose, a desirable installation position of the camera is required for the drawer formed at the bottom of the storage compartment, and the inventors of the present invention first reviewed the camera installation position disclosed or known through data published at the time of filing the application for the present invention. We examined whether it was possible to film the inside of the refrigerator storage room by installing it on the top of the door outside the refrigerator storage room.

When installing a camera outside the storage room (door), the camera is always exposed to external temperature and condensation does not form around the camera, so there is no need for a separate condensation prevention structure. However, since the user's storage of items inside the drawer and in the storage space within the storage room is always done with the door open, it was difficult to photograph the inside of the storage room with a camera mounted on the door. In particular, the purpose of the present invention is to obtain not only the movement state of the drawer outside the drawer but also the storage state of items inside the drawer. However, the user's work in the drawer is always in a state where the door is open, and when the door is closed, the drawer is also in a closed state, which does not correspond to the spirit of the present invention at all.

Therefore, in order to photograph at least the inside of the drawer and the state of movement of the drawer, which is the purpose of the present invention, it is desirable to arrange the storage room so that the storage room can be photographed from inside the refrigerator, that is, from the main storage room where the drawer to be photographed is located, rather than from outside the refrigerator. do.

Meanwhile, in accordance with the spirit of the present invention, when the camera is installed at the bottom of any one of the shelves located at the top of the drawer with the camera facing toward the opening of the main storage compartment, it is impossible to photograph the shelf area first. It is difficult to install and wire the driver that can drive the camera on the shelf, and since a wide-angle camera must be used to capture the entire movement path of the drawer, there may be a problem of increased distortion in the captured image.

The inventors of the present invention found that when a camera is installed on the side wall of a storage room in a range where the drawer can be exposed, the image showing the storage condition of items inside the drawer is distorted due to the asymmetric image due to the difference in distance from the camera in the left and right directions of the refrigerator storage room. I could tell it was serious. Therefore, it was found that the satisfaction of these users could decrease significantly.

Above all, the purpose of the present invention is to detect the open/closed state and degree of the drawer by tracking the moving state of the drawer without a separate sensor. The marker may be photographed in a relatively enlarged state (image distortion). Therefore, the size (pixel size) of the marker portion increases with respect to the overall size of one image, so the change in position may not be clearly visible. Additionally, the range that needs to be processed to determine whether the marker position has changed increases, which can increase the image processing time, so there may be cases where continuously transmitted images are missed or cannot be processed.

Therefore, it was found that installing the camera on the side wall or corner of the storage room, which was already known at the time of filing the present invention, does not meet the purpose of the present invention.

In one embodiment of the present invention, the camera is installed on the upper wall of the storage compartment corresponding to the upper opening surface of the drawer, and the camera is installed on the upper surface of the storage compartment where the closed state of the drawer and the fully open state of the drawer are projected. It was installed within the range, and the direction of the camera was toward the rear wall of the storage room, toward the inside of the storage room, but the camera was installed at a certain angle so that its focus was toward a certain point on the rear wall of the storage room. As a result, it was possible to obtain images with minimized distortion for the above-described embodiments of the vertical angle of the camera of the present invention.

In determining the position of the camera, another embodiment of the present invention is installed on the upper wall of the refrigerator storage compartment, and the front end of the uppermost shelf among the shelves arranged up and down in the storage compartment at the front opening of the storage compartment is on the upper wall of the storage compartment. It is placed between the projected positions. In addition, the direction of the camera is toward the rear wall of the storage room and toward the inside of the storage room, and the camera is installed at a certain angle so that its focus is directed to a certain point on the rear wall of the storage room.

The camera is preferably located in the central part of the upper part of the storage compartment, and the printing unit located on the upper surface of the front wall or both sides of the drawer to correspond to this is preferably installed on the upper surface of the front wall of the drawer. More preferably, it is proposed near the upper surface of the front wall of the drawer corresponding downward to the position of the camera installed on the upper surface of the storage compartment.

In a refrigerator in which the opening of a storage compartment is opened and closed by one or more doors, that is, the left and right doors, the camera is installed on the upper surface (ceiling surface) of the storage compartment at the boundary between the left and right doors, and the drawers on the left and right are installed. In one case, the printing unit is located close to the left and right boundaries. That is, from the perspective of looking into the inside of the storage room, it is preferable that the left drawer is located on the upper right side of the front wall or the upper end of the right side wall, and the right drawer is located on the upper left side of the front wall or the upper end of the left side wall. In other words, it is desirable for the printing portions to be formed in a position close to the vertical lower part of the camera. This is because the farther left and right you are from the vertical lower part, the higher the risk of distortion in the printing area. An example of this is shown in Figure 41b.

At this time, if the printing unit is installed on both sides of the upper surface of the front wall of the left and right drawers or at the ends of the upper surfaces of both sides, the drawers used in the storage room opened and closed by the left and right doors can be commonly used regardless of the left and right positions, making it easier to use the drawers. This can increase efficiency.

As described above, according to the camera position and the printing unit position according to the present invention, the entire movement path from the closed state to the open state of the drawer is included within the shooting range of the fixed camera, so the drawer is drawn without a separate sensor device. It is possible to detect not only the open/closed status of the lower, but also the degree of open/closed and the stop or movement of the drawer, and at the same time, the storage status of the items inside the drawer can be detected or confirmed with a single camera.

In addition, in addition to the inner area of the drawer, the storage status of items in the floor storage compartment installed on the floor of the storage room can be checked when the drawer is closed, making it possible to recognize multiple storage areas with a single camera.

In addition, when the camera's angle of view is expanded to include the shelf area, it is possible to selectively recognize three storage areas with a single camera, including the inner area of the drawer as well as the storage compartment on the floor of the storage room and the storage area.

Meanwhile, since the camera installed on the upper surface of the storage compartment and the printing unit formed or attached to the upper part of the drawer are roughly aligned vertically, the printing unit portion is shown in the entire image of the interior captured and transmitted by the camera. There is almost no distortion, and the space occupied by the printing unit can be relatively minimized, making it possible to minimize the time to track and determine the location of the printing unit through image processing.

Hereinafter, a method for recognizing marker tracks formed on a drawer will be described.

A printing part (marker or printing paper) that can be recognized by image reading can be installed on a specific part of the drawer body that is exposed to the camera. By tracking the position of the marker part, the drawer's open/closed degree and open/closed status can be detected. The marker part is configured to be exposed to the camera from the closed position to the fully open position along the movement path of the drawer, so that the open/closed state of the drawer can be determined based on the position of the marker part. In other words, various status information of the drawer can be determined through changes in the position of the marker in continuously captured photos.

As described above, the printing unit (marker) structure may be made up of a specific pattern in order to track and recognize the movement status or open/close status of the drawer. At this time, the printing unit may be placed in various positions on the upper surface of the drawer exposed to the camera.

The printing unit is preferably installed at the top of the front wall of the drawer or at the top of both walls of the drawer, which can be the boundary of the maximum open portion of the drawer.

The pattern shape of the printing unit can be formed by repeating patterns having a certain shape of a certain size in contrasting colors, and the shape can be formed in various shapes such as square, triangle, circle, and wavy shape.

There is a risk that this printing part may detract from the overall aesthetics of the refrigerator, so some users may be reluctant to use it. Therefore, it is necessary to minimize the repetition of patterns so that the aesthetic sense felt by the user is not rejected, and furthermore, a printing unit in a form that can appeal to the user's aesthetic sense can be considered.

However, this limitation in the design aspect is not limited to a simple design issue, and it can lead to cases where the location of the drawer cannot be accurately recognized. In other words, it affects the recognition rate of the drawer's printing section, ultimately resulting in the inability to accurately detect the location of the drawer.

In other words, considering the design element that requires drastically reducing the size of the printing part, it is possible to select a pattern that cannot be repeated several times with contrasting colors in a pattern of a certain shape, but only repeats it 2 to 4 times. In this case, the control unit that processes the image captured inside the interior may not accurately recognize the boundary of the printing unit (marker) in the entire image.

Therefore, according to a preferred embodiment of the present invention to solve this problem, drawers are usually made of bright colors, transparent or translucent, so it is possible to select a color that contrasts with them. Referring to FIG. 41A, the length L1 in the direction perpendicular to the moving direction (forward and backward) of the drawer can be set to be longer than the length L2 in the drawer moving direction. And a first color portion having a predetermined length in a vertical direction is formed, with a color contrasting with the first color portion or the same color as the drawer with approximately the same thickness in front or behind the first color portion with respect to the drawer moving direction. A second color part is formed, and the second color part is continuously formed repeatedly with the same color as the first color part, with approximately the same thickness, and the same length. That is, it is possible to sequentially arrange the first color part and the second color part. At this time, it is more desirable that the vertical length (L1) of the first color section, the second color section, and the outer size of the markers arranged in the first color section be larger than the drawer direction length (L2).

According to another marker type of the present invention, as shown in Figure 41b, both ends of the first color portion at the front and rear ends are connected to each other to surround the second color portion. The third embodiment in the form of a marker shown above can be an example of this structure.

That is, in the preferred embodiment of the marker presented in the present invention, the length (L1) in the direction perpendicular to the direction of movement of the drawer is formed to be larger than the length (L2) in the direction of movement of the drawer, but the portion bordering the drawer (especially the The idea is to form a color (dark color) that contrasts with the drawer (in relation to the direction in which the lower moves), and include a light color part in the middle that contrasts with the dark color.

If the printing part is configured in this way, it can be configured in a small size, so not only can it meet the user's aesthetic requirements, but the size of the printing part in one entire image captured by the camera becomes relatively small. Therefore, the image processing area for detecting changes in the position of the printing unit becomes smaller, thereby improving image processing speed. This is advantageous in that the image processing speed of the control unit matches the speed of image capture and transmission, thereby further improving the accuracy of acquiring images at the correct time. For example, if the size of the part that needs to be processed in photos taken continuously is large, a lot of load is placed on the control unit. Therefore, a very expensive control unit must be used. However, if the size of the part to be processed is small, the load on the control unit can be reduced. Therefore, it is possible to effectively process images even when using a low-cost control unit.

Meanwhile, as shown in FIG. 41B, it is desirable for the marker to include a letter, symbol, or pattern so that the user can intuitively understand its purpose by looking at the marker.

Below, we will explain how to recognize the marker coordinates of the drawer and process the camera image.

Figure 42 is a diagram explaining a marker position recognition method. Hereinafter, a method for recognizing the location of a marker in a drawer will be described with reference to FIG. 42.

When the user wants to store (stock) or release an item inside the drawer, the user first opens the door. When the door is separated from the front of the storage room, the door switch detects it, and the control unit receives a signal from the door switch and operates the camera installed in the air in the storage room. When a camera drive voltage other than the standby voltage is applied to a camera that has received a drive command from the control unit, the camera is ready to drive and starts continuous shooting.

The user will then try to open the drawer by holding the handle on the front wall of the drawer mounted in the refrigerator storage compartment and pulling the drawer. At this time, since the camera is continuously taking pictures, it takes pictures of the open state of the drawer at regular intervals and sends the images to the control unit.

Meanwhile, a certain type of marker is formed on a part of the drawer that is exposed to the camera, and the control unit that receives the image sent from the camera divides the entire image into certain pixels and assigns coordinate values at regular intervals, The coordinates of the marker are analyzed on the coordinates of the image.

Therefore, if an image captured while the drawer is closed is sent to the control unit, the corresponding image marker portion will not be exposed, and the marker coordinates at this time will have coordinates of 0.

When the user opens a closed drawer, the camera takes a picture with the marker exposed, and the camera can take a picture that includes the marker. When this image is sent to the control unit, the control unit recognizes the coordinate value of the marker on the entire image and compares it with the previous position. For example, when the coordinate value of the marker at point 'A' in Figure 42 changes from '0' to '50' as a certain coordinate value, the control unit can determine that the drawer is moving based on the change in coordinate value. there is. In particular, if it changes from '0', it can be judged that the drawer has just started to open.

If the user continues to open the drawer, the controller may determine that the drawer is moving through a series of processes as described above and that the drawer is being opened because the coordinate value changes to a larger value.

When the user opens the drawer to the fully open state, the control unit recognizes that the coordinate value no longer changes and determines that the drawer is fully open or that the drawer is open and stopped. For example, the point 'B' in Figure 42 is a point where the coordinate value of the marker has not changed from '90' to '90', so it is recognized as 'the drawer is stopped' or 'the drawer is fully open'. You can judge.

Typically, after the drawer is opened, regardless of whether it is fully opened or partially opened, if it is in a stopped state, it can be determined that the user is performing some work on the drawer. Even when the drawer is stopped, the camera continues to capture images inside the cabinet and sends them to the control unit. The control unit monitors whether there is a change in the coordinates of the marker.

When the user completes the act of placing an item in the drawer or removing an item from the drawer, the user closes the drawer again, and at this time, from the image captured by the camera, the control unit confirms that the marker coordinate value for the entire image has changed. This determines that the drawer is moving again. For example, point 'C' in Figure 42 is the point where the coordinate value of the marker changes from '90' to '70', and the size of the coordinate value changes in the direction of reduction, which means that the drawer has just started to close. It can be determined that the user has completed the task on the drawer.

In the same way, if the marker's coordinate value is '0', that is, the marker is not recognized from the image, it is determined that the drawer is completely closed.

From the image captured by the camera, the control unit determines the location of the marker based on the coordinates of one entire photo, and determines the moving and stationary state of the drawer, the closing and opening of the drawer, the degree of opening of the drawer, and whether the drawer is starting to open. We explained how to track and determine when a drawer begins to close.

Hereinafter, camera image processing and a method of obtaining a desired image according to the marker position of the drawer will be described. A detailed method of how to process and obtain the internal image of the drawer from the moving state of the drawer will be described. In particular, a method of obtaining an image at a point in time determined to be immediately after the user has completed work on the drawer will be described.

When the marker coordinates of the drawer change beyond '0' and it is determined that the drawer is in the process of moving in the direction of opening, the control unit detects only the coordinate change for the internal video captured by the camera and sent to the control unit, and takes the entire picture. Buffering occurs in a temporary buffer and no processing is performed on each storage area demarcated in the photo.

When the drawer continues to open and reaches a fully open state or opens to a certain degree and enters a stopped state, a situation occurs in which the user performs work on the drawer. The control unit determines this state based on the fact that the coordinates of the marker do not change, and from this point on, for the transmitted image, the image corresponding to the drawer area is drawn from the storage area divided from the entire image and moved to the temporary buffer for storage. do. In other words, in the section corresponding to point 'D' in the above figure, the control unit cuts out the necessary part from the part divided into the drawer area of the entire picture inside the drawer and moves it to the temporary buffer for the drawer area, which is called the 'drawer internal image refresh section'. ' is defined.

In other words, the image processing method is different in the section where the drawer is open and the section where the drawer is in a stopped state. The reason is that the user's work on the drawer reflects that the drawer will be performed in a stopped state. As a result, the image processing time for tracking coordinate changes in the section where the drawer moves can be correspondingly faster, thereby improving the efficiency of data processing.

Meanwhile, when the drawer starts to close again from a stop state, the control unit determines that the coordinate value changes to a smaller value as soon as the user has completed the task, and at that point, the temporary buffer for the image inside the drawer is Among the temporarily buffered images, the most recent image is selected and saved. In other words, an image of the inner area of the drawer is captured immediately after the user completes the task.

The image of the latest item storage status in the drawer area captured (saved) in this way replaces the image of the corresponding area on the display online or directly connected to the control unit, or is transmitted to the network server. In addition, the latest images transmitted and stored in the network server are provided so that the user can check them at any time when desired through a display installed in the refrigerator or a mobile terminal connected to the network server.

Next, we will explain how to search for the location of a marker by selecting a specific section rather than searching for the location of the marker for the entire photo taken by the drawer.

In other words, it is possible to set the search range for the drawer area. On the drawer's movement path, the degree of openness of the drawer can be divided into sections and the method of tracking the marker in each section can be varied. As a result, it is possible to shorten the processing (analysis) time for photos taken and transmitted by the camera and match the image processing speed with the camera shooting speed.

In general, a drawer used in a refrigerator cannot be pulled out in its entirety to the extent that the drawer is separated from the storage compartment and falls to the outside, even if the drawer is completely opened. In other words, the distance that a drawer can normally be drawn out, the normal opening distance, is generally about 50% of the total depth of the drawer (the length before and after the drawer in the direction of movement of the drawer).

The drawer withdrawal distance may be determined as an area to which coordinates of the drawer marker must be assigned in the entire image processed by the control unit. In this case, the control unit can determine the location of the drawer from the time the drawer is withdrawn. In particular, in order to detect a change in the marker coordinates of the drawer, the size corresponding to the drawer withdrawal distance becomes the data size that the control unit must process. Additionally, in order to detect a change in coordinates, the control unit must compare and judge two images at the same time, and the processing time of this process may be burdensome as it must be able to respond to the shooting speed of the camera.

It is preferable that the camera's shooting speed (time interval for taking pictures) of the inside of the camera is determined to approximately match the speed at which the control unit judges and processes the transmitted image.

If you follow this method, you can quickly determine the location of the marker in the photo taken by the camera and ultimately determine the location of the drawer. And while the control unit processes images for each area divided by each storage space in the overall picture of the interior, there is no difficulty in imaging and processing new images continuously sent from the camera. In other words, the image for each area is cut from the entire image and moved to a temporary buffer for each area depending on whether the door is open or the drawer is open. Processing is possible without missing all images captured by the camera. It will be done.

However, since data for two images (previous image and current image) of the size described above must be processed, the processing capacity of the control unit must be increased. Therefore, if the processing capacity of the control unit is increased to meet these conditions, the unit cost of the processor may rise rapidly. there is. Additionally, for image processing when the drawer is not open, the processing capacity becomes too large, which causes the problem of not being able to select efficient parts.

In order to solve this problem, the inventors of the present invention focused on the fact that it is practically difficult to fully understand the inside of the drawer when the draw-out distance of the drawer is not fully open, that is, when the drawer is opened by a certain distance, It provides a method of different data processing by separating the images corresponding to the drawer area into stages according to the draw distance of the drawer.

Figure 43 shows the degree to which the drawer is opened. Figure 43a shows the drawer fully open, that is, about 50% of the entire length of the drawer is open, Figure 43b shows the drawer open about 30% of the total length, and Figure 43c shows the drawer fully open. About 10% of the length is open.

For example, as shown in Figure 43c, when the user opens the drawer by only about 10% of the total front and rear length, it is difficult to fully identify the items stored inside the drawer, and Figure 43b approx. Only when about 30% of the drawer is opened does it become possible to understand the storage state of the items inside the drawer to some extent, and this state is enough to recognize the drawer as open.

As shown in Figure 43a, when the drawer is fully opened, about 50% of the entire drawer is pulled out. At this time, the items inside the drawer can be fully seen in the fully open state. In other words, the drawer must be in an open state of about 30% to 50% before the user can store or remove items from the drawer. This section can be viewed as a substantially open state of the drawer.

Taking this into consideration, the control unit determines that if the drawer withdrawal distance is less than 30% as 'the drawer is not open' and if it is more than 30% as 'the drawer is open' and processes the image differently. The data load to be processed can be significantly reduced.

The control unit converts the image captured by the camera into an image, determines the image size corresponding to the drawer movement trajectory, and determines the pixel size up to the point or distance at which the drawer is withdrawn by about 30% from the designated drawer movement area to determine the image section. compartmentalize.

That is, when the drawer is drawn out to the range corresponding to Figures 43A to 43B, the control unit can determine the movement of the coordinates of the marker. In this case, the load on the control unit required is reduced compared to the case of determining the movement of the marker's coordinates in the range corresponding to Figures 43A to 43C.

In the section where 30% is drawn when the drawer is closed, it is only determined whether the marker position has reached the position of the pixel corresponding to the 30% distance. In other words, it does not compare the marker coordinates of the previous photo with the marker coordinates of the current photo (a prerequisite for determining a change in the marker's position), but simply determines whether the marker's position is within a certain pixel range and temporarily buffers the entire image. I do it.

In this section, since only one photo data is processed without the need to compare two photo data, the data processing capacity of the control unit does not need to be large, and it is possible to smoothly respond to the camera shooting speed.

On the other hand, when the drawer is opened beyond the 30% level, that is, when the marker reaches a position above a certain pixel in the photo processed by the control unit, two photos (taken just before) are taken to determine the stopped state of the drawer. The coordinate value of the marker can be determined by comparing the photo taken and the photo taken afterwards.

If the search section is limited in this way, the size of the photo becomes about 2/5 of the size compared to if the search section was not limited. Furthermore, since the size of the two photos to be processed is about 4/5 that of a single photo, the data to be processed in the control unit is smaller or substantially the same as that of an entire photo. It can be level.

Therefore, the data processing speed for tracking the change in coordinates of two photos can correspond to the speed at which the camera shoots, making it possible to process data (coordinate tracking or monitoring) without missing any images captured by the camera, so the control unit Even without further increasing the processing capacity, it is possible to prevent delays in processing speed, allowing the desired image to be acquired at the correct time.

In conclusion, as shown in FIG. 44, it is possible to divide a portion of the distance that can be retrieved by the drawer from a photo taken with a camera into a first section and a second section.

In the first section, the position of the marker for the pixel of the captured photo can be determined. At this time, the first section can determine which pixel position the marker is in one photo without comparing the coordinate values of the marker.

Meanwhile, in the first section, even if the drawer is actually open, it is possible to classify the drawer as being in a substantially closed state because it is difficult for the user to withdraw or bring items into the drawer.

The second section can be set as a 'coordinate search range' that compares the coordinate values of markers shown in two consecutively taken photos. In the second section, since two photos are compared, changes in the marker's coordinates can be detected. For example, if the marker is at the same location in two consecutive photos, it may be determined that the drawer was stopped during the time the two photos were taken. If the position of the marker changes in two consecutive photos, it can be determined that the drawer was moved during the time the two photos were taken. At this time, depending on the direction of change in the position of the marker, it can be determined whether the drawer is moved in the closing direction or the drawer is moving in the opening direction.

In the second section, since the opening that allows the user to withdraw or bring in items into the drawer is exposed to the user, it is also possible to classify the drawer as being in a substantially open state.

Figure 45 is a control flow diagram according to an embodiment of the present invention. Hereinafter, the description will be made with reference to FIG. 45.

It is determined that the door 20 opens the storage compartment 22 (S10). At this time, opening and closing of the door 20 may be detected by the door switch 110.

When the door 20 is opened, the door sensor 120 may be driven. That is, the door sensor 120 can detect whether the door 20 is rotated more than a certain angle θ. That is, the door sensor 120 does not operate unless the door 20 is opened, and may start operating when the door 20 is opened.

Additionally, when the door 20 is opened, the camera 70 can operate and start taking pictures (S14). At this time, the camera 70 is capable of taking 10 pictures at predetermined time intervals, for example, 1 second.

The photo taken by the camera 70 is stored in the storage unit 18 (S16). At this time, when a plurality of pictures are taken by the camera 70, the most recently taken picture may be stored, and the previously taken picture may be deleted due to capacity limitations of the storage unit 18.

Meanwhile, whether the drawer 50 is opened can be detected by the drawer detection unit 130 (S20). The drawer detection unit 130 can detect that the drawer 50 is pulled out from the lower space of the shelf 40.

And when the drawer detection unit 130 detects whether the drawer 50 is closed, the photo taken at that time or the photo taken closest to that time is selected as the final photo (with the drawer open) ( S40). Since the camera 70 takes a set number of pictures per second, it is possible that pictures are not taken exactly at the point when the drawer 50 begins to close. However, if the drawer detection unit 130 determines the movement of the drawer 50 by analyzing the movement of the marker expressed in the photo taken by the camera 70, the photo is taken at the time it is determined to be closed. It is possible to choose.

The control unit 100 divides the final photo into the first area 42, the second area 52, and the third area 62 (S42). At this time, the control unit 100 can specify the number of pixels based on the adjustment line 15 to distinguish each area and obtain a final image for each area.

As described above, when the drawer 50 is pulled out, the second area 52 and the third area 62 overlap each other.

Meanwhile, since information that the drawer 50 has been withdrawn is obtained in S20, it can be determined that the second image has been obtained from the control unit 100.

And information about the second area 52 can be updated (S46). The control unit 100 can transmit a second image showing the second area 52 to the display 14 and change the corresponding part to be based on the final photo.

On the other hand, if S30 determines that the drawer 50 has not started to close, the photo can be taken again as in S14.

The state in which the drawer 50 is not closed in S30 can be viewed as the user not terminating access to the second area 52. That is, the user can store new food in the second area 52 or remove stored food from the second area 52 while stopping the drawer 50.

On the other hand, when the drawer 50 is closed, it can be predicted that the user terminates access to the second area 52 and inserts the drawer 50 into the lower space of the shelf 40.

Meanwhile, in a state in which the drawer 50 is not opened in S20, it can be determined whether the door 20 is about to close (S60).

At this time, whether the door 20 is about to close can be detected by the door sensor 120.

When the user terminates access to the storage compartment 22, he closes the door 20 and seals the storage compartment 22. That is, the user can store new food in the storage compartment 22 or take out stored food from the storage compartment 22 while allowing the door 20 to open the opening 14 .

If the user closes the door 20, it may be determined that the user terminates access to the storage compartment and the food remains stored in that state until the door 20 is opened again.

If it is determined in S60 that the door 20 is about to close, the control unit 100 selects a photo taken at that time or a photo taken closest to that time as the final photo (with the drawer closed) (S70). That is, the control unit 100 may select a photo taken at or near the time when photo taking by the camera 70 ends as the final photo.

The control unit 100 divides the final photo into the first area 42, the second area 52, and the third area 62 (S72). At this time, the final photo may include a first image showing the first area 42. This is because the latest status of the food stored in the first area 42 is that the user has terminated access to the storage room 22. Therefore, the user can obtain accurate food information about the first area 42 through the first image obtained at this time.

Meanwhile, since information that the drawer 50 has been inserted is obtained in S60, it can be determined that the first image and the third image have been obtained from the control unit. As described above, when the drawer 50 is pulled out, the second area 52 and the third area 62 overlap each other, but the first area and the third area do not overlap each other. This is because it is a location where we can film together.

And information about the first area 42 and the third area 62 can be updated (S76). The control unit 100 transmits a first image showing the first area 42 and a third image showing the third area 62 to the display 14, and displays the corresponding part based on the final photo. You can change it to:

Meanwhile, although not shown in the drawing, the door 20 is rotated to seal the storage compartment 22, so that when the storage compartment 22 is closed, the operation of the camera 70 can be stopped. At this time, the door switch 110 can detect whether the door 20 is sealing the storage compartment 22.

That is, in the present invention, taking into account the time when the user ends use, the photo obtained at that time is processed to provide the user with information about the food stored in the corresponding area. Therefore, the user can obtain accurate information about the food stored in the storage compartment 22.

Meanwhile, the door sensor 120, the door switch 110, and the drawer detection unit 130 may be installed two each on the left and right sides. In this case, individual information about the left and right doors and left and right drawers can be obtained. Of course, if the drawer detection unit 130 detects the movement of the drawer through analysis of the marker in the taken photo, the drawer detection unit 130 is only one, but the markers are installed in each of the two drawers. It is possible to become

Therefore, based on this information, it is possible to update only the left part of one photo when only the left door is open. On the other hand, if only the right door is open, it is possible to update only the right part of one photo.

Of course, when the left and right doors are opened together, if the above-mentioned conditions are met, it is possible to update the left and right parts of one photo together and provide them on the display 14.

FIG. 46 is a control flowchart explaining a modification of FIG. 45. Hereinafter, the description will be made with reference to FIG. 46.

In the modified example of Figure 46, S60 is specifically divided into S62 and S64, and only the differences are explained in detail.

That is, whether the door 20 is about to close can be specifically determined by determining whether the door 20 is rotated in the closing direction (S62) and whether the door 20 is rotated by a certain angle θ or more. there is.

In other words, when the door 20 is rotated in the closing direction (S60) and the door is rotated below a certain angle (S64), photo taking by the camera 70 can be ended.

In order to obtain information about the food stored in the storage compartment 22 from the photo taken by the camera 70, there should be no interference from the door 20 or the basket installed on the door 20. Therefore, the expression that the door 20 is about to close can also mean a state in which the door 20 and other components are not represented in the photo taken by the camera 70.

FIG. 47 is a control flowchart explaining another modification of FIG. 46. Hereinafter, the description will be made with reference to FIG. 47, but only the differences from FIG. 46 will be described.

In another modification according to FIG. 47, unlike FIG. 46, the rotation direction of the door 20 is not determined.

If it is determined in S12 that the door has been rotated more than a certain angle, the door 20 is rotated in the opening direction at the corresponding moment. Therefore, typically, the user rotates the door 20 again, and when the door 20 reaches a state where it is rotated by a certain angle, the user closes the door 20.

Therefore, in another modified example, S62 in FIG. 46 is omitted, and if it is determined in S20 that the drawer is not opened, S64, which determines whether the door has been rotated below a certain angle, is immediately performed.

FIG. 48 is a control flowchart explaining another modification of FIG. 45. Hereinafter, description will be made with reference to FIG. 48.

Figure 48 explains the control flow of detecting the movement of the drawer 50 using a marker. For convenience of explanation, overlapping parts will be omitted, and only the differences will be explained.

Overall, S20 and S30 can be specifically divided into stages S22, S24, and S26.

After the photo is stored in the storage unit 18 as in S16, it is determined whether the marker is recognized by the camera 70 (S22).

Since the camera 70 takes pictures from the upper side of the inner case 12 toward the lower side, there may be cases where the marker is not visible from the camera 70. That is, when the handle portion of the drawer 50 is obscured at the front end of the shelf 40, that is, when the drawer 50 is not pulled out at all, the marker may not be recognized.

When a marker is recognized in a photo taken by the camera 70, it can be determined whether the marker is stopped (S24). At this time, as described above, after identifying the movement path of the marker in one photo, it is also possible to divide it into a first section and a second section and determine the movement of the marker only in the second section.

Since the camera 70 captures a plurality of photos at predetermined time intervals, it is possible to determine whether the marker has moved by comparing and analyzing the photos. Since the marker is displayed on the drawer 50, it can be determined that when the marker is moved, the drawer 50 is also moved.

In addition, it can be determined whether the marker has moved a certain distance or more from the position where the drawer 50 was closed (S26).

If the marker does not move beyond a certain distance, an appropriate image of the second area 52 cannot be obtained because the drawer 50 is not sufficiently opened. For example, in a photo taken when only 1/3 of the drawer 50 is pulled out, the inner space of the drawer 50 is not sufficiently shown, and the image obtained from the photo shows that the user is in the second area. This is because information about (52) cannot be obtained. At this time, the certain distance can be determined by whether the marker has entered the search range for determining the movement of the marker.

If it is determined that the marker has moved more than a certain distance, it can be considered that the user has finished accessing the second area 52 and has finished using the second area 52.

Accordingly, a photo acquired at a time when the three conditions are satisfied or at a time close to the time may include information about the food stored in the second area 52.

FIG. 49 is a diagram showing a modified example of FIG. 45. Hereinafter, description will be made with reference to FIG. 49.

In FIG. 49 , unlike FIG. 45 , there is a difference in that the camera is not activated immediately when the door 20 is opened by the door switch 110.

When the door switch 110 determines that the door 20 is open, the door sensor 120 is driven.

Then, the door sensor 120 determines whether the door 20 has been rotated more than a certain angle (S12). At this time, if the door 20 is not rotated more than a certain angle, the camera 70 does not take a picture.

When the door 20 is rotated more than a certain angle, the camera 70 takes a picture (S14). At this time, the camera 70 is capable of taking 10 pictures at predetermined time intervals, for example, 1 second.

The photo taken at this time includes the first image of the first area 42. Meanwhile, since the door 20 is rotated more than a certain angle θ, interference from a basket installed on the door 20 may not occur in the photo. That is, the basket installed on the door 20 may not appear in the photo.

Furthermore, in S60, the door sensor 120 can detect whether the door 20 is about to close. At this time, just before the door 20 is closed, it may mean a state in which the door 20 is rotated at a certain angle or less while the door 20 is rotated in the closing direction.

The camera 70 can take pictures until just before the door 20 is closed and stop taking pictures (S49). In Figure 45, the door 20 is closed and the camera 70 takes pictures until the moment the door switch 110 is pressed. However, in a modified example, when the door 20 is rotated to a certain angle by the door sensor 120, the camera 70 stops taking pictures. At this time, the constant angle may mean an angle that prevents interference from the door 20 as well as the basket installed on the door 20 in the photo taken by the camera 70.

Figure 50 is a control flow diagram according to another embodiment of the present invention. Hereinafter, the description will be made with reference to FIG. 50. The control flow of FIG. 50 can be continuously performed at predetermined time intervals when it is determined that the door 20 is opened by the door switch 110.

First, the door sensor 120 determines whether the door 20 is opened by a certain angle θ (S100).

When the door 20 is opened at a certain angle or more, it is determined that there is no interference with the door 20, and the camera 70 takes a picture (S110). At this time, the camera 70 is capable of taking one picture.

Then, it is determined whether the drawer 50 has been drawn out beyond the first set distance (S120). At this time, the first set distance may mean a distance at which food information about the second area 52 can be sufficiently obtained when a picture is taken by the camera 70. For example, the first set distance may mean 2/3 or 1/2 of the total length of the drawer 50.

The withdrawal distance of the drawer 50 can be detected by the drawer detection unit 130.

Meanwhile, if it is determined that the drawer 50 has been drawn out beyond the first set distance, the control unit 100 can divide the photo by region and divide it into a plurality of images (S122).

Since the photo is obtained while the drawer 50 is pulled out, a first image and a second image can be obtained from the photo.

The control unit 100 can update the first image and the second image (S124).

On the other hand, if it is determined in S120 that the drawer 50 has not been drawn out by more than the first set distance, it can be determined whether the drawer 50 has been drawn out by less than the second set distance (S130).

At this time, the second set distance may mean a distance at which food information about the third area 62 can be sufficiently obtained when a picture is taken by the camera 70. For example, the second set distance may mean 1/3 of the total length of the drawer 50. Of course, it is also possible to select the second setting distance as '0' to select a state in which the drawer 50 is completely retracted into the lower space of the shelf.

Meanwhile, if it is determined that the drawer 50 has been drawn out below the second set distance, the control unit 100 can divide the photo by region and divide it into a plurality of images (S132).

Since the photo is obtained while the drawer 50 is pulled out, a first image and a third image can be obtained from the photo.

The control unit 100 can update the first image and the third image (S134).

In another embodiment of the present invention according to FIG. 50, the control is repeated when the door 20 is opened, and is completely terminated when the door 20 closes the storage compartment 22. Since no further updates to the image occur in the terminated state, the final image for the food stored in the storage compartment 22 can be obtained.

Figure 51 is a diagram explaining the process of updating the image in a state where two doors and two drawers are provided to open and close the storage room. Hereinafter, the description will be made with reference to FIG. 51.

Contents that overlap with the above-described embodiments will be briefly described.

When electricity is supplied to the refrigerator, the door switches 110 installed on the left and right sides may individually determine whether the left or right door is opened (S302, S303).

When the door switch 110 determines that either the left door or the right door is open (S303), the camera 70 starts taking pictures (S304).

The camera 70 can take pictures not only when either the left door or the right door is opened, but also when the left door and the right door are opened at a time or at the same time. This is because, when it is determined that the door is opened through one of the door switches, the lights inside the room are turned on.

If the drawer placed on the left is pulled out more than 130 mm (S310), an image related to the left drawer is captured (S312). The video related to the left drawer may refer to the portion of the captured photo corresponding to the left drawer, or may refer to the entire captured photo.

At this time, 130mm may vary depending on the refrigerator manufacturer or user, but may refer to the withdrawal distance that allows the user to withdraw the drawer and access the storage space inside the drawer.

If the left drawer is pulled out less than 130 mm (S310), the immediately preceding image captured by the camera 70 is maintained (S310).

And the left door sensor detects the opening angle of the left door. At this time, if the left door is opened at an angle less than 80 degrees, an image of the left multi-storage space provided in the lower space with the left drawer pulled out is captured (S332).

In this case, 80 degrees may mean an angle at which the left door and various structures installed on the left door do not interfere when photographing the storage space with the camera 70. Depending on the specific structure or size of the refrigerator, 80 degrees can vary.

If the left door is opened at an angle greater than 80 degrees, it is possible to maintain the immediately preceding image captured by the camera 70 (S336).

And it is possible for the door switch 110 to detect that the left door is closed (S334).

Meanwhile, it is possible to implement the same type of control flow as described for the left drawer and left door for the right drawer and the right door. That is, since the above-mentioned contents are equally applied to S320 to S324 and S340 to S346, detailed descriptions are omitted.

When the left door and the right door are closed, it is possible to process the image captured above in the control unit 100 and display the processed image on the display 14 (S350). At this time, the display may be a display installed in the refrigerator.

And the camera can stop taking pictures (S352).

Meanwhile, the refrigerator is connected to Wi-Fi to determine whether communication is possible (S354), and if communication is possible, images can be transmitted to an external server (S356).

The server determines whether the application is synchronized with the application of the terminal used by the user (S358), and if synchronized, it is possible to display the image on the display 14 of the terminal.

Therefore, even if the user is not in front of the refrigerator, he or she can receive information about the items stored in the refrigerator through the user terminal.

If the refrigerator is not connected to Wi-Fi and cannot communicate with the outside, or is not synchronized with the application of the terminal, it is desirable to maintain the previous image on the display 14 of the user terminal and provide the previous image to the user. do.

Hereinafter, the components provided to take a picture and obtain a picture at a desired time will be briefly described.

When the door switch detects that the user has begun to open the door, the control unit operates the camera to start filming the interior. The camera continuously takes pictures at a certain frame per second. Since the user's act of opening the door means removing a desired item from the refrigerator or storing a new item in the refrigerator, the inside of the refrigerator is taken continuously by taking pictures of the inside of the refrigerator storage compartment. You can monitor whether there are any changes in the status of the storage space.

The storage room is divided into various shelves, drawers, or baskets to form a storage space, and items or containers are stored and stored in the storage space depending on the shape or content.

The camera can monitor the storage space by dividing it into a shelf area divided by shelves, a drawer area, and other storage areas. Through this, the intention is to allow the user to check the storage status of goods in the area immediately after the user completes the withdrawal or warehousing operation. If two drawers and two other storage areas are provided in parallel on both sides, information that allows the user to check each of the two drawers and two other storage areas can be provided through the camera. You can.

One embodiment of the present invention defines the other storage area as a storage space formed on the floor of the storage room, located in front of the drawer storage area, and at least partially overlapping when the drawer is opened, and includes it in the monitoring target. present.

When the door switch detects the opening of the door, the control unit turns on the door sensor included in the refrigerator hinge assembly to detect movement according to the rotation of the door. The door sensor may always be turned on, but it is also possible for the door sensor to be turned on only when the door is opened by the door switch.

When the door is opened beyond a certain angle, the control unit determines that the door is substantially open enough for the user to access the storage room and bring in or take out items, and determines the storage status of the shelf area to be upgraded.

In addition, the drawer detection unit detects the degree of opening of the drawer, and when the drawer is opened, the storage status of the drawer area is determined as a target for upgrading, and when the drawer is not opened, the storage area is determined as a target for upgrading. decide.

The door sensor of one embodiment of the present invention detects when a certain angle has passed in the process of opening and closing the door and sends related information to the control unit. Here, the door sensor can also detect the rotation direction of the door, that is, whether it is a closing direction or an opening direction.

In an embodiment of the present invention, when the on signal of the door switch is detected, the control unit turns on the door sensor, and the door sensor detects when the door passes a certain angle and sends the corresponding time to the control unit. It becomes. At this time, the control unit determines that the door is open. Specifically, at this time, the door is opened by being rotated by a certain angle.

Meanwhile, when the user completes the act of withdrawing or putting in goods while the door is open, the open door is closed. When the door passes the door sensor, the door sensor sends a signal to the control unit, and the control unit uses this signal to It is determined that the entry and exit actions are completed and the door is about to close. Specifically, at this time, the door is opened by being rotated by a certain angle, but is also rotated to close.

The process of capturing an image inside the store from a camera during the opening and closing process of the door and upgrading the image to the image captured immediately after the user completes the act of entering or leaving an item will be described in detail with reference to an embodiment of the present invention.

When the refrigerator door switch is turned on while the door is closed, the control unit turns on the camera and door sensor. The camera may mean switching from standby mode to driving mode, and the door sensor may mean switching to a state that can determine the opening angle of the door.

The camera can continuously photograph the interior of the refrigerator at a set speed over a certain period of time. The captured image is sent to the control unit, and the control unit temporarily buffers the transmitted image as an image containing the entire interior.

Picture (or photograph) and image used in this specification can be used separately.

Specifically, a picture (picture or photograph) refers to raw data transmitted immediately after being captured by a camera, and an image (image) refers to the raw data (picture or photograph) transmitted by the control unit. It may refer to data that is buffered, corrected, or transmitted.

When the door switch is turned on, the door is opened at a certain angle, and the control unit determines that the door is open, the control unit determines which area the entire image contains the storage status of the goods according to the open state of the drawer. is decided.

For example, when the drawer is open, the control unit controls the area (shelf area) where all the stored images are stored on the upper shelf of the refrigerator and the drawer storage area (drawer area) where the entire stored image is stored inside the drawer. It is judged to contain the storage condition of the goods. On the other hand, if it is determined that the drawer is closed, the control unit determines that the entire stored image is located in the inner area of the storage space (storage area), which is an area formed in the refrigerator shelf area and the front end of the drawer and overlapped by the opening of the drawer. It is judged to contain the storage condition of the goods.

Here, the picture of the entire refrigerator storage compartment captured and transmitted by the camera is divided into three different storage spaces, and the shelf area is always included, and the drawer area and the storage area are divided into three different storage spaces. It is optionally included depending on the open/closed status of the lower. The drawer area and the storage area do not coexist with each other in the photo.

Accordingly, one entire picture transmitted from the camera includes at least two areas, one of which is the shelf area, and which storage area the remaining part corresponds to is determined by the open/closed state of the drawer.

The present invention is intended to convey to the user the latest arrival and departure status of goods by photographing at least two different spaces, including the interior of the drawer, such as the interior space of the drawer and another storage space, with a single camera. A method of upgrading by fixedly photographing only the area and the shelf area can be another embodiment of the present invention, and fixedly photographing only the drawer area and the storage space can also be an embodiment of the present invention. .

Meanwhile, dividing one overall picture of the storage compartment, which constitutes an independent insulated space in a refrigerator, into storage areas within the storage compartment can be determined by the number of doors that open and close the storage compartment. For example, in a refrigerator where the storage compartment is opened and closed with a single door, the storage area to be photographed can be broadly divided into two or three types, and when the storage compartment is opened and closed with two left and right doors, the storage area is It can be broadly divided into 4 or 6 types.

At this time, the drawer area and the storage area can be divided into left and right sides and managed as two separate areas in the entire photo, and the shelf area can also be managed by integrating them into one area. In this case, the partitioned parts within the entire photo may be composed of five types.

In this shelf area, the storage spaces facing the left and right doors are connected to each other. In the shelf area, items, food, vegetables, etc. may be stored on the left and right. When stored across the shelf, goods can be retrieved or received by opening only one of the left and right doors, so even if only one of the left and right doors is opened, the entire shelf area can be integrated from the entire captured image and replaced with the latest data. Because there is a need. Taking this into account, it is possible for the entire shelf area to be updated even if only one of the left and right doors is opened.

To explain this in more detail, if it is determined that the drawer is open while the door is open, the control unit separates the shelf area and drawer area from the entire transmitted photo and temporarily buffers them. And, when the user finishes loading and unloading items in the drawer area or/and shelf area and tries to close the drawer, that is, when the drawer starts to move from a stopped state after being opened, the most recent among the temporarily buffered images From the video captured, the necessary part of the drawer area is captured (captured or cut) and stored in the storage as an image of the recent storage state of the drawer area. At the same time, the captured image of the drawer area in the refrigerator interior image displayed on the refrigerator display is upgraded by replacing it with the most recent image. Additionally, the control unit transmits the captured image of the drawer area to a server system connected to the network.

Meanwhile, if the door is determined to be closed because the drawer is not detected to be open while the door is open, the control unit determines that the entire picture of the transmitted video includes the shelf area and the storage area. Therefore, the shelf area and the storage area are separated from the entire photo and temporarily buffered.

When the user completes the task of loading and unloading goods from the storage area and/or shelf area and closes the door, the door sensor detects when the door passes a designated angle and sends a signal to the control unit. The control unit determines that the door is about to close (which can be expressed as just before the photo is obscured by an obstacle such as a door) based on the signal transmitted from the door sensor, and stores the most recently captured image among the temporarily buffered images. Necessary parts within the area are captured and stored in memory as an image of the most recent storage state of the storage area. At the same time, the captured image is replaced and upgraded for the storage area in the refrigerator interior image displayed on the display.

In this case, upgrading may mean replacing the existing image with the latest image. Additionally, the control unit transmits the captured image of the drawer area to a server system connected to the network.

Meanwhile, whether the drawer is in an open or closed state, the shelf area captures the most recent image among the buffered images at the time when the door is judged to be just before closing and stores it in memory as the latest item storage status of the shelf area.

The captured image (one or all of the most recent shelf area image, most recent drawer area image, or most recent storage area image) replaces the displayed image on the refrigerator display, transmits it to a server system, or connects to the refrigerator through communication. Transmission to the user's mobile device may be carried out simultaneously while storing the captured image in the refrigerator memory, may be done all at once when the user requests it, or may be done for each individual area according to the user's request signal.

Here, one image transmitted from the camera may be buffered as one image by the control unit. If it is determined that the one image was taken with the door open, it may be determined whether the one image includes the drawer area or the storage area depending on the open/closed state of the drawer. When such a determination is made, the corresponding images are extracted and buffered in separate temporary buffers.

If it is determined that one image was taken at the point when the drawer begins to close, the most recent image among the temporarily buffered images is stored in memory, and is replaced with the most recent image when a new entry/exit action by the user is completed. . Additionally, when it is determined that the door is about to close, the most recent image among the temporarily buffered images of the storage area is stored in the memory.

Meanwhile, for the shelf area, only the shelf area can be extracted from the most recent image right before the door is closed among all images being buffered and stored in memory, and a temporary buffer can be prepared for a separate shelf area to extract and buffer only the shelf area. It may be possible.

Hereinafter, with reference to FIGS. 52 to 55 , various embodiments in which the most recent image of the interior of the refrigerator obtained by the various methods described above are transmitted to a personal mobile terminal or a business operator server and utilized are described.

Figure 52 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.

Referring to FIG. 52, when the control unit 100 determines that the final photo of a specific area inside the refrigerator has been obtained through the steps described with reference to FIG. 34 (S1101), it sends the corrected final image to the terminal in the form of an event (S1101). 2) Transmit to (S1103). The terminal 2 may include, but is not limited to, a smartphone, a Personal Digital Assistant (PDA), a Tablet PC, etc. At this time. The control unit 100 can transmit not only the final image inside the refrigerator but also various information related to the final image, which will be described below, to the terminal 2.

At this time, the control unit 100 may transmit the final image to the terminal 2 as soon as the final image is acquired through the final photo, and after storing the acquired final image in the storage unit 18, the control unit 100 may transmit the final image to the terminal 2 at a predetermined period. Images stored in event format can also be transmitted to the terminal 2, but are not limited to this.

Next, the terminal 2 displays the final image received according to the user's selection (S1105). For example, the terminal 2 may run an application that can receive and display an image of the interior of the refrigerator according to user selection, or may perform a multimedia service and display the received image of the interior of the refrigerator.

According to this embodiment of the present invention, the user can check the image of the inside of the refrigerator using a mobile terminal even without directly looking inside the refrigerator, so there is a practical benefit of being able to remotely determine the current situation inside the refrigerator.

Figure 53 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.

Referring to FIG. 53, when it is determined that the final photo of a specific area inside the refrigerator has been obtained (S1301), as described above, the control unit 100 divides the final photo by area to generate an image, and then the obtained final image is Control it to be stored in the storage unit 18 (S1303).

Next, when the terminal 2 receives a user input for displaying the current image inside the refrigerator, that is, the final image of a specific area of the refrigerator (S1305), it transmits a final image request command to the refrigerator (S1307).

In addition, the control unit 100 operates the communication unit ( 270) is controlled (S1309).

According to this embodiment of the present invention, communication between the mobile terminal and the refrigerator is performed only when the user needs it, making it more efficient and economical. Hereinafter, only the case where images, etc. are transmitted in response to a request will be described, but the case where images, etc. are transmitted in the event format as described with reference to FIG. 35 is also included, and is limited to this. It doesn't work.

Figure 54 is a ladder diagram for explaining a refrigerator operating method according to another embodiment of the present invention.

Referring to FIG. 54, as described above, when the control unit 100 determines that the final photo of a specific area inside the refrigerator has been obtained (S1501), it controls the final image obtained from the final photo to be stored in the storage unit 18. (S1503).

Next, when a business server such as the market server 3 receives a user input for displaying the current image inside the refrigerator (S1505), the market server 3 transmits a final image request command to the refrigerator (S1507) . At this time, the market server 3 may transmit a final image request command to the refrigerator according to a predetermined cycle without user input.

And, in response to the image request command, the control unit 100 causes the most recently captured and stored final image of the inside of the refrigerator among the images stored in the storage unit 18 to be transmitted to the market server 3 ( S1509).

The market server 3, which has received the final image inside the refrigerator, analyzes the final image (S1511). The market server (3) analyzes the final image to find out what is currently stored in the refrigerator and what is not currently stored, and to analyze what was stored in the refrigerator in the past but is not currently stored. Information needed to provide services can be extracted.

The market server 3 transmits the list of insufficient items extracted through final image analysis to the refrigerator (S1513).

Figure 55 is a ladder diagram for explaining a refrigerator operation method according to another embodiment of the present invention.

Referring to FIG. 55, as described above, when the control unit 100 determines that the final photo of a specific area inside the refrigerator has been obtained (S1701), it controls the final image corrected from the final photo to be stored in the storage unit 18. (S1703).

Next, when the business server, such as the broadcasting station server 4, receives a user input for displaying the final image inside the refrigerator (S1705), the broadcasting station server 4 transmits a final image request command to the refrigerator (S1707) . At this time, the broadcasting station server 4 may transmit a final image request command to the refrigerator according to a predetermined cycle without user input.

And, in response to the final image request command, the control unit 100 transmits the final image of the inside of the refrigerator that was most recently captured and stored among the images stored in the storage unit 18 to the broadcasting station server 4. Control (S1709).

The broadcasting station server 4, which has received the final image, analyzes the final image (S1711). The broadcasting station server 4 can extract information necessary for providing services, such as analyzing the final image, finding out the items currently stored in the refrigerator, and analyzing dishes that can be made using the currently stored items.

The broadcasting station server 4 transmits recipe information for dishes that can be made using the currently stored items to the refrigerator through final image analysis (S1713).

The control unit 100 displays recipe information for dishes that can be made using the currently stored items received from the broadcasting station server 4 on the display 14 installed in the refrigerator (S1715)

According to this embodiment of the present invention, there is a practical benefit in that a related business operator can accurately identify the internal situation of a refrigerator located in a home or business and provide refrigerator-related services suitable for the home or business.

Figure 56 is a diagram explaining the operation of the camera's heater, and Figure 57 is a diagram explaining the results of an experiment on condensation occurring at each temperature on the transparent window of the camera. Hereinafter, the description will be made with reference to FIG. 56. In Figure 56, the description will be based on the heater of the type described in Figures 14 to 17.

Figure 56(a) is a graph in which the heater 84 is intermittently operated by intermittent power supply, and Figure 56(b) is a graph in which the heater 84 is continuously operated by continuously supplied power. It's a graph.

Figure 56(a) shows a method in which the temperature of the transparent window 80 is maintained in an equilibrium state with the temperature of the storage compartment 22 without normally controlling it, and then high power is applied momentarily when refrigerator camera photography is needed. In particular, condensation generated on the transparent window 80 can be removed by operating the heater 84 immediately before taking a picture.

When the door 20 is opened, the transparent window 80 may come into contact with moisture contained in the warm outside air that enters through the door 20. At this time, since the temperature of the transparent window 80 is relatively low compared to the outside air, moisture contained in the outside air condenses on the transparent window 80, causing condensation.

If the heater 84 is driven momentarily to remove dew generated by this condensation phenomenon, there may be a problem that time is needed until the dew on the transparent window 80 is actually removed. Therefore, if the time at which photography is needed is earlier than the time at which dew is removed, a problem may arise in which dew may form on the transparent window 80.

On the other hand, in Figure 56(b), the heater 84 continues to operate and heat is supplied to the transparent window 80. Since the temperature of the transparent window 80 is always maintained above the condensation generation temperature, condensation does not always occur on the transparent window 80. Therefore, even if the door 20 is opened and the transparent window 80 comes into contact with the outside air, moisture contained in the outside air does not form dew on the transparent window 80. Therefore, it is possible to prevent dew from forming on the transparent window 80 when the camera 70 needs to take a picture.

When operating the heater 84, it is necessary to ensure that there are no image quality problems due to condensation and that the power consumption of the refrigerator does not increase rapidly due to the power consumption of the heater 84. It is also desirable to take into account that the heat supplied by the heater 84 should not affect the temperature inside the refrigerator.

The present inventors believed that in order to minimize the power consumption of the heater and not affect the interior of the refrigerator, it would be necessary to operate the heater with a long off time. However, for concerns that condensation may occur if the heater is turned off for a long time, the target temperature was set on the transparent camera window, and the ambient temperature of the refrigerator was 32 degrees Celsius and the relative humidity was 85% (the dew point temperature is approximately Celsius). The experiment was conducted under conditions ranging from 29 to 30 degrees (see Figure 57). In Figure 57a, the y-axis means the amount of condensation generated on the transparent window, and in 57b, the y-axis means the percentage of the amount of condensation actually generated at the corresponding temperature compared to the current amount of water vapor.

The heater's on/off operation cycle, derived from repeated experiments, could deteriorate the image quality of photos taken by the camera under special temperature conditions in the store.

Even if an appropriate on/off cycle for the heater is found, the user's refrigerator usage pattern during the time the heater is off may vary unpredictably.

For example, if the user leaves the refrigerator door open for a long time, the camera cover glass (cover glass may refer to the transparent window described above or may refer to various types of windows placed in front of the camera lens) (Can) A large amount of condensation forms on the surface. In this case, even if the cover glass is heated by operating the heater, evaporation takes a lot of time, so there is a problem with the image quality if taken at that time. EAlso, it took a lot of time and a lot of power to evaporate the already generated dew from the cover glass.

Additionally, if the storage compartment is a refrigerating compartment, the temperature inside the storage compartment is usually maintained between 0 degrees Celsius and 7 degrees Celsius, and the internal volume is also larger than the surface area of the cover glass. A large portion of the heat supplied by the heater inevitably leaks into the cold air inside the storage room. Therefore, in order to evaporate the dew generated on the cover glass, a large amount of heat must be applied for a short period of time. In order to supply such heat intensively for a short period of time, a heater with a large capacity is used, and the instantaneous power usage increases significantly, which can cause various problems in poor power conditions.

Therefore, it is desirable to prevent a certain amount of condensation from continuously occurring on the cover glass from the beginning, and the present invention is characterized in that power is applied to operate the heater at all times without an on/off cycle.

The method of constantly applying power to the heater can be configured to apply power to the camera heater as soon as power is applied to the refrigerator, or the refrigerator control unit can be controlled to always apply power to the heater during normal operation of the storage compartment. there is.

Next, the heating temperature of the cover glass will be explained. As a result of quality evaluation based on image samples obtained for each temperature range, it was confirmed that there was no significant problem in the quality of sample images up to a temperature where the temperature of the center of the cover glass was approximately ½ of the dew point temperature. No problem with video quality may mean that the type and quantity of items stored in the storage room can be identified through the captured photos.

In particular, the quality of the image sample obtained at 13.5 degrees, point (3) in Figure 57b, was at an acceptable level. Of course, it is possible to maintain the heating temperature of the cover glass above the dew point, but power can be supplied to the heater so that the heating temperature of the cover glass is maintained between about 13 degrees Celsius and about 30 degrees Celsius.

Meanwhile, the experimental results showed that the heating temperature of the cover glass differed depending on the outer size of the cover glass under the condition that the same applied power was supplied, and there was no significant difference in the thickness of the cover glass. Therefore, since the image quality was satisfactory at 1/2 the dew point temperature for the same cover glass size, it is desirable to set this as the lower limit and select an appropriate upper limit in terms of the heater's power consumption (applied power).

When heating the cover glass, the applied power near the dew point temperature (29.2 degrees Celsius) was 0.45 W, and the applied power at about 0.58 times the dew point temperature (17.1 degrees Celsius) was 0.27 W. Therefore, in the range of 0.5 to 0.7 of the dew point temperature, the power consumption is approximately 0.5 times as much, which has a significant effect in reducing power consumption.

Typically, in variable outdoor conditions, the intermediate range of dew point temperature can be determined to be approximately 12 to 17 degrees Celsius. Therefore, it is preferable that a current value is applied to the heater to maintain the temperature of the central portion of the transparent window at approximately 12 to 17 degrees Celsius.

That is, in the present invention, the effect of reducing the power consumed by the heater can be achieved by maintaining the temperature of the cover glass at a temperature lower than the dew point, rather than at the dew point.

Figure 58 is a cross-sectional view of a transparent window. Hereinafter, the description will be made with reference to FIG. 58.

The transparent window 80 may be hydrophilic coated. At this time, the transparent window 80 may refer to the cover 90 described above.

Even if dew momentarily forms on the transparent window 80, the water contact angle (α) can be formed within 15 degrees due to the hydrophilic coating, as shown in FIG. 58. When a hydrophilic coating is applied, the surface tension of water on the surface decreases. Therefore, water can spread widely on the surface of the transparent window 80. Accordingly, the camera can be fixed to the ceiling of the storage compartment such that the angle between the transparent window and the ceiling of the storage compartment is approximately 10 to 20 degrees.

Therefore, it is possible to minimize distortion of the photograph taken due to water condensing on the transparent window 80. In addition, hydrophilic coatings can have advantages in terms of energy efficiency because there is no need for control such as supplying additional electricity once produced.

Figures 59 and 60 are diagrams briefly showing the structure in which the camera is installed in the inner case. For reference, Figures 59a and 60a are views looking at the side of the storage compartment, and Figures 59b and 60b are views looking up from the top of the inner case. Hereinafter, it will be described with reference to FIGS. 59 and 60. Through these embodiments, it is possible to minimize the occurrence of condensation on the cover 90 by bypassing the path of external air to the cover 90.

The camera 70 may be installed on a protrusion 500 that protrudes downward from the ceiling of the inner case 12. At this time, the protrusion 500 may refer to a part that protrudes downward compared to other parts of the inner case 12. This protrusion 500 may also be implemented through the camera housings described above.

At this time, as shown in Figure 59, the camera 70 is installed within the protrusion 500 to face the inner wall of the storage compartment 22. The camera 70 can take pictures of the inside of the storage compartment 22 through the cover 90. Accordingly, the cover 90 is disposed at a position facing the rear wall of the storage compartment 22 from the protrusion 500.

At this time, moisture contained in the outside air contacts the cover 90 and cools, thereby generating dew.

However, because the cover 90 is arranged to face the rear wall of the storage compartment 22, the path through which outside air reaches the cover 90 may be long.

That is, in order for the outside air passing through the door 20 to pass through the storage compartment 22 and contact the cover 90, it must pass through the path shown in Figure 59b. At this time, the outside air comes into contact with the cold air in the storage compartment 22. Some of the moisture it contains may condense in other areas. Accordingly, the outside air may reach the cover 90 as the amount of moisture in the storage compartment 22 gradually decreases.

That is, even if the door 20 is open, the inside of the storage compartment 22 maintains a lower temperature than the outside of the storage compartment 22, and since the temperature does not immediately change to the same as outside air, heat exchange occurs as outside air enters the storage compartment 22. This comes true.

In other words, if the cover 90 is placed in a position where it is relatively difficult to contact the outside air, the amount of dew generated on the cover 90 may be reduced and the time at which dew is generated may be delayed.

First, as the outside air enters the interior, it comes into contact with the front surface 501 of the protrusion 500. At this time, the external air contacting the front surface 501 changes its path to the left and right of the protrusion and further enters the inside of the cavity. This outside air may reach the cover 90 while contacting the side 502 of the protrusion 500. Accordingly, a large amount of moisture may condense before reaching the cover 90.

Additionally, external air in contact with the front surface 501 of the protrusion 500 may descend and rise on the lower surface 503 of the protrusion. Accordingly, a large amount of moisture may be condensed while moving to the front surface 501 and the lower surface 503.

In addition, the cover 90 is tilted and mounted in a shape surrounded by a front surface 501, a side surface 502, and a bottom surface 503. Therefore, the path of the outside air and the cover 90 are not perpendicular. Accordingly, external air can also be prevented from reaching the cover 90 and rapidly condensing.

In particular, the width of the protrusion 500, especially the left and right widths of the front surface 5010, may be wider than the width of the camera 70.

The embodiment shown in Figure 60 is very similar to the embodiment shown in Figure 59. However, it can be said that it is different in that the camera is mounted to look vertically downward. Therefore, the same effect as the example shown in FIG. 59 can be achieved.

Additionally, the protrusion 500 may include a rear surface 504. It is preferable that the vertical length of the front surface 501 is greater than the vertical length of the rear surface 504. That is, it is possible that the front surface 501 protrudes further downward from the inner case 12.

Therefore, in order for outside air to reach the cover 90, the path is lengthened in the horizontal direction by the width of the protrusion 500, and in the vertical direction, the path is lengthened by the difference in the protruding height of the protrusion 500. You lose.

Preventing condensation on the cover 90 through the structure of the protrusion 500 or the structure for mounting the camera is very related to the installation location of the camera. That is, the camera can be installed in an angle range that tilts backwards at a predetermined angle from the vertical at the vertical lower part of the ceiling of the storage room. Therefore, it is possible to effectively photograph the entire area within the storage room while effectively diverting the inflow path of external air.

In the prior art, there are technologies for mounting the camera on the side wall of the main body storage compartment, the outer top of the main body, the door, or the upper part of the door.

In the prior art, when mounting a camera on the side wall of a storage room, there is a technique of providing a recess in the side wall to accommodate the camera and mounting the camera in the recess. In other words, the camera is prevented from protruding into the interior of the storage room. It is presumed that this configuration is intended to avoid inconvenience to the user when removing shelves or items from the storage room.

However, in the present invention, the camera is installed on the ceiling of the inner case 12 so that the user is not disturbed when using the storage compartment 22. This is because when the user wants to retrieve an item from the storage room, there is no reason to collide with the camera because the camera is not located in the movement path of the item or shelf.

Furthermore, in the present invention, the cover 90 of the camera 70 is arranged to face the rear, so that the path through which external air reaches the cover 90 can be extended.

In Figure 59a, the cover 90 is arranged to face the rear, and the protrusion 500 protrudes downward, so that the outside air has a three-dimensional movement path to reach the cover 90, thereby allowing the outside air to move. The path becomes longer.

In Figure 60a, the cover 90 is arranged to face downward, but the front surface 504 is disposed horizontally in the opening of the protrusion 500, and the front surface 504 is located at both ends of the front surface 504. A side 508 disposed perpendicular to the surface is installed. At this time, the front surface 504 and the side surface 508 are installed to protrude further downward from the protrusion 500, so that the path through which outside air reaches the cover 90 can be lengthened.

That is, the high-temperature air flowing in from the outside to the storage compartment opening does not reach and directly contact the cover 90 of the camera in a straight line, but detours left and right and downward along the protrusion 500 to reach it. Therefore, as the outside air exchanges heat with the cold inside the room, the temperature decreases and the amount of saturated water vapor decreases, so the water vapor it contains condenses and mixes with the cold inside the room. When it finally reaches the cover 90, it has a reduced amount of water vapor and can be sufficiently evaporated by the heater heating the cover 90.

Accordingly, the power that must be provided to the heater 84 is reduced, thereby improving the overall power consumption of the refrigerator.

The embodiment shown in Figure 59 or Figure 60 includes a cabinet having a storage compartment formed therein; The storage compartment consists of an access opening at the front, an upper wall, a lower wall, both side walls, and a rear wall made of an insulating material, a shelf area divided by a plurality of shelves, and a drawer having at least one drawer. Consists of a region; at least one door that is in close contact with the front of the storage compartment and opens and closes the storage compartment; It includes a camera device installed on the upper wall of the storage compartment between the opening of the storage compartment and a front edge of the shelves installed in the storage compartment;

The camera device includes a camera module part that accommodates electrical components necessary for operating a camera, including a camera lens, and has a camera window formed at a certain distance from the lens, and the camera module. It is configured to include a camera housing part that seats and fixes the part at a predetermined position inside;

The camera housing portion includes a fixing surface in contact with the upper wall of the storage compartment, a front surface formed on an opening side of the storage compartment, side surfaces connecting the front and rear surfaces, and the camera. A refrigerator is provided that includes a top surface with an opening formed to expose the camera window of the module unit, and the front of the camera housing unit is formed higher than the opening formed on the top surface of the camera housing unit. You can.

The upper surface of the camera housing may be inclined at a certain angle with respect to the upper wall of the storage compartment in a direction opposite to the opening of the storage compartment.

The camera housing may further form a rear side opposite to the front side on the rear wall of the storage compartment, and the height of the front side may be higher than the height of the rear side.

The opening in the upper surface of the camera housing may be formed at a lower position than the side surface of the camera housing.

It may further include a housing installation part having a recessed portion to correspond to a recessed space inside the upper wall of the storage compartment.

The housing mounting unit may be mounted on the upper wall of the storage compartment together with the refrigerator insulation material, and the camera housing unit may be fixed to the housing mounting unit.

The camera module part fixed to the camera housing part may be accommodated in a depression formed in the housing mounting part.

The camera module part accommodated in the recessed part of the housing mounting part may be fixed to the camera housing part so as to have a gap with the inner surface of the housing mounting part.

The front and both sides of the camera housing may be formed to be higher than the opening position of the top surface of the camera housing.

The upper surface of the camera housing may be formed so that the opening side of the storage compartment is farther from the upper wall of the storage compartment than the rear wall of the storage compartment.

The camera module unit has a front portion where a camera window is formed and a rear portion on the opposite side, and at least one flat surface is formed on at least one of the front portion and the rear portion. You can.

A receiving portion in which the camera module portion is received is formed inside the camera housing portion, and the receiving portion is tilted at a certain angle with respect to the upper wall of the storage compartment and a flat surface formed in the camera module portion is placed. It may include a seating part.

The recessed portion of the housing mounting portion may include a seating part on which a flat surface formed in the camera module portion is tilted at a certain angle with respect to the upper wall of the storage compartment.

A flat surface may be formed around the camera window of the front part of the camera module unit, and the flat surface may be formed to be approximately the same height as the center of the camera lens.

Figure 61 is a diagram showing another type of refrigerator to which the present invention is applicable. Hereinafter, the description will be made with reference to FIG. 61.

The refrigerator in Figure 61 is a double-door refrigerator, with the left and right sides forming different storage compartments. That is, it is possible to configure the storage compartment on the left as a freezer, the storage compartment on the right as a freezer, and the storage compartment on the left as a refrigerator.

A camera 70 may be installed on the ceiling of the storage compartment 22 to face the lower portion of the storage compartment 22.

In addition, a drawer 50 that can store goods and can be withdrawn and retracted may be installed in the storage compartment 22.

Other details are omitted since they are the same as the refrigerator described above.

Figure 62 is a screen provided to the user from the refrigerator according to Figure 61. Hereinafter, description will be made with reference to FIG. 62.

The user may be provided with images of items stored in the drawers 50 and 51 and images of items stored on the upper side of the shelf 40 . A refrigerator may be provided with one drawer 50. Therefore, an image of one drawer may be provided in Figure 62 as well. However, the barwak kx shown in Figure 61 may be provided with two drawers 50 and 51 above and below. In this case, it is also possible to provide images of the inside of each drawer.

In the example described in Figure 2, drawers are provided on the left and right respectively. However, in Figure 62, drawers may be provided at the top and bottom, respectively. Therefore, there may be two drawer areas, and the lower drawer 50 may protrude forward compared to the upper drawer 51. In this case, a marker may also be provided in each drawer (50, 51).

Through this, it is possible to easily identify food information stored in multiple shelf areas and multiple drawer areas through a single camera.

The method of upgrading the image provided on the user screen is omitted since it is the same as that of the refrigerator described above.

FIG. 63 is a diagram explaining a method of adjusting a photo taken by a camera in the refrigerator according to FIG. 61. Hereinafter, description will be made with reference to FIG. 63.

Since only one drawer is usually provided or arranged vertically, one horizontal adjustment line 15 may be provided in the refrigerator of the type of FIG. 61. Of course, two horizontal adjustment lines may be provided to separate the two drawers. In addition, it is also possible to adjust the position of the control line by placing two vertical control lines arranged in parallel. Of course, unlike what is shown in Figure 63, it is also possible to select the desired image using only the horizontal adjustment line without using the vertical adjustment line.

The user may be provided with a limit line 16 indicating a limit range for moving the adjustment line 15. In this case, the limit line 16 may be arranged in parallel with a predetermined interval.

Correction of the error through this adjustment line 15 can be said to be the same or similar to the above-described embodiment.

The present invention is not limited to the above-described embodiments, and as can be seen from the appended claims, modifications can be made by those skilled in the art, and such modifications fall within the scope of the present invention.

10: Outer case 12: Inner case
20: door 40: shelf
42: First area 50: Drawer
52: Second area 60: Barrier
62: Third area 70: Camera
100: control unit

Claims (5)

pantry for storing food;
a refrigerator body that forms part of the storage compartment and includes an opening of the storage compartment;
a door rotatably provided on the main body and opening and closing the opening of the storage compartment;
Drawers and shelves installed on one side and the other of the storage room, respectively;
A basket provided inside the door;
A camera installed on one side of the main body to photograph the drawer and shelf;
A refrigerator including a display that displays images captured by the camera According to claim 1,
The refrigerator is characterized in that the camera selectively changes the shooting position of the storage compartment by a control unit. According to claim 1,
The refrigerator is characterized in that the control unit obtains food information in the storage room from image information captured by the camera. According to claim 1,
The camera is comprised of a camera lens and a transparent window installed in front of the camera lens, and the transparent window is equipped with a heater. According to claim 3,
The refrigerator is characterized in that the control unit provides information on the type, number, and storage date of the stored food to the display.

Images