JP6778949B2 - Storage management system - Google Patents

Description

The present invention relates to a stored item management system for managing stored items stored in a storage chamber of a refrigerator.

It is important for the user to properly manage the stored items stored in the refrigerator storage room such as the refrigerator compartment. In order to respond to this, with the development of image analysis technology using a computer, storage items management that grasps the type of stored items and the storage position in the storage room from the analysis of image data taken by the camera installed in the storage room. The system has been proposed.

In such a stored item management system, when a plurality of stored items are placed on top of each other, it may not be possible to photograph the stored items located on the lower side. In that case, it is not possible to accurately grasp the type of stored items and the storage position in the storage room. To deal with this problem, the camera installed in the storage room can be rotated up and down so that even if the stored items are placed on top of each other, the lower stored items can be photographed. The system has been proposed. (See, for example, Patent Document 1)

JP-A-2002-267336

In the system described in Patent Document 1, depending on the storage location of the stored items in the storage room and the placement of the stored items, no matter how the camera is rotated, it may not be possible to take a picture of the overlapping lower stored items. Therefore, there are still cases where it is not possible to grasp the type of stored items stored in the refrigerator and the storage position in the storage room.

Therefore, an object of the present invention is to solve the above-mentioned problems, and to surely grasp the type of stored items and the storage position in the storage room regardless of the storage location of the stored items and the storage method of the stored items in the storage room. The purpose is to provide a storage management system that can be used.

The present invention is a stored item management system.
Includes two or more retroreflective photoelectric sensors and retroreflective surfaces located in the front of the refrigerator storage chamber, the presence or absence of a detection object in the front of the storage chamber, and the photoelectric sensor of the detection object in existence. A detector that detects the angle with respect to
The imaging device installed at the front of the storage room and
A control unit that performs control processing based on data from the detection unit and the imaging device, and
With
The control unit
Based on the angles detected by the two or more photoelectric sensors, the position of the detected object on the virtual plane at the front of the storage chamber is calculated.
It is characterized in that the type of the detected object is determined based on the image data captured by the photographing device.

The retroreflective photoelectric sensor includes a light projecting unit that emits light in the visible light region and an infrared region, and a light receiving unit that receives the reflected light. The light projecting unit and the light receiving unit are usually arranged adjacent to each other, and the light emitted from the light emitting unit is reflected by the reflecting surface, and the reflected light enters the light receiving unit. When a detected object exists between the photoelectric sensor and the reflecting surface, the amount of light received by the light receiving portion changes due to shading. Based on this, the retroreflective photoelectric sensor can detect the presence / absence of the detected object and the angle of the existing detected object with respect to the photoelectric sensor.

In particular, since the reflecting surface is composed of a retroreflective surface in which the light incident from the light source returns to the direction of the light source, the angle of the position where the detected object exists with respect to the photoelectric sensor can be accurately detected.

The angle with respect to the photoelectric sensor means the angle formed by the line connecting the photoelectric sensor and the detected object with the reference line of the photoelectric sensor. For example, when the line connecting the two photoelectric sensors is used as the reference line, the positions of the two photoelectric sensors and the distance between them are known. Therefore, from the angle detected by the two photoelectric sensors, the storage room is subjected to the triangulation method. It is possible to calculate the two-dimensional position of the detected object on the virtual plane in front of. Even when the three photoelectric sensors are provided, the position of the stored object can be calculated by the same method.

The photographing by the photographing apparatus can be continued at all times, or can be taken at an arbitrary timing including when the photoelectric sensor detects the presence of the detected object.
Any image analysis technique can be used to determine the type of detected object based on the image data captured by the imaging device. The type of the detected object to be determined includes the determination of whether or not the stored object is present in the user's hand, and the determination of the type of the stored object held in the user's hand.

The control unit can be provided exclusively for the stored items management system, the refrigerator control device can perform the function of the control unit, and the remote server can perform the function of the control unit by using an internet line or the like. You can also.

In the present invention, the position of the detected object on the virtual plane at the front of the storage chamber is calculated based on the angles detected by two or more photoelectric sensors, and the detected object is calculated based on the image data captured by the photographing device. Determine the type. Thereby, when the stored items are stored or taken out, in some cases, the two-dimensional position of the stored items at the opening position of the storage chamber and the type of the stored items can be determined at an arbitrary timing during the operation. In particular, this can be achieved with a simple device configuration such as a photoelectric sensor and a photographing device.

As described above, in the present invention, since the position and type of the stored items can be detected at the opening position of the storage room, there is no problem even if the stored items are stacked vertically and stored in the storage room. Regardless of the storage location of the stored items and the storage method of the stored items, the type of the stored items and the storage position in the storage room can be surely grasped.

Further, the present invention is characterized in that, of the four surfaces on the opening side of the inner wall of the storage chamber, two or more of the photoelectric sensors are arranged on one surface, and the retroreflective surface is arranged on the remaining three surfaces. And.

In the present invention, by appropriately arranging the photoelectric sensor and the retroreflective surface according to the operable angle range of the photoelectric sensor, a rational arrangement of the detection unit capable of covering the entire opening position of the storage chamber is realized. it can.

Further, the present invention is characterized in that the photographing apparatus starts photographing based on the photoelectric sensor starting to detect the presence of the detected object.

In the present invention, since the photographing device starts photographing based on the photoelectric sensor starting to detect the presence of the detected object, it is possible to efficiently and surely acquire the image data when storing or removing the stored object. it can.

Further, the present invention is characterized in that the photographing apparatus ends photographing based on the fact that the photoelectric sensor finishes detecting the presence of the detected object.

In this configuration, further, since the photographing device finishes shooting based on the fact that the photoelectric sensor finishes detecting the presence of the detected object, it is possible to efficiently acquire the image data when storing or taking out the stored object. Can be done.

Further, in the present invention, the control unit obtains image data immediately after the photoelectric sensor starts detecting the presence of the detected object and image data immediately before the photoelectric sensor finishes detecting the presence of the detected object. It is characterized in that it is determined whether the stored material is stored or taken out by comparison.

Based on the image data, if there is an item in the user's hand immediately after starting the detection of the presence of the detected object, and there is no item in the user's hand immediately before the detection of the presence of the detected object is completed. In that case, it can be determined that the stored items are stored in the storage room. On the other hand, if there is no stored item in the user's hand immediately after starting the detection of the presence of the detected object, and there is a stored item in the user's hand immediately before the detection of the presence of the detected object is completed. Can determine that the stored item has been taken out of the storage room.

According to the present invention, it is possible to surely obtain important information for the management of stored items with a simple device configuration such as a photoelectric sensor and a photographing device.

It is possible to provide a stored item management system that can surely grasp the type of stored item and the storage position in the storage room regardless of the storage place of the stored item and the storage method of the stored item in the storage room.

It is a figure which shows the refrigerator provided with the storage thing management system which concerns on one Embodiment of this invention, (a) is the front view in the state which the door is open, (b) is AA of (a). It is sectional drawing which shows the cross section. It is a schematic diagram which shows that the detected object is detected by the detection part including the retroreflection type photoelectric sensor and the retroreflection surface. It is a schematic diagram which shows that the angle with respect to the photoelectric sensor of a detected object is detected by the position in the light receiving part of the photoelectric sensor which changes the amount of light incident by the light shielding by the detected object. It is a block diagram which shows the outline of the stored matter management system which concerns on one Embodiment of this invention. It is a schematic diagram which shows that the position of the detected object on the virtual plane of the front part of a storage chamber is calculated based on the angle detected by two photoelectric sensors. It is a flowchart which shows the main routine which shows an example of the control process for the storage thing management.

Next, a specific embodiment of the present invention will be described in detail with reference to the drawings.
(Storage management system according to one embodiment of the present invention)
1A and 1B are views showing a refrigerator 2 provided with a stored matter management system according to one embodiment of the present invention, in which FIG. 1A is a front view in a state where the door 6 is open, and FIG. It is sectional drawing which shows the AA cross section of a). First, with reference to FIG. 1, the arrangement of the detection unit 10 and the photographing device 20 constituting the stored object management system according to one embodiment of the present invention, particularly the system installed in the storage room 4 of the refrigerator 2, and the arrangement thereof. The function will be explained. Although FIG. 1 shows a refrigerator refrigerating room as a storage room, the present invention is not limited to this, and can be applied to any storage space provided in the refrigerator such as a freezing room and a vegetable room.

<Detector>
As shown in FIG. 1, two retroreflective photoelectric sensors 12 and 14 and a retroreflective surface 16 are provided at the front portion indicated by an arrow B in the storage chamber 4 of the refrigerator 2. The retroreflective surface 16 is a reflective surface having a characteristic that the light incident from the light source returns in the direction of the light source. In the retroreflective type photoelectric sensors 12 and 14, the light projecting unit and the light receiving unit are arranged adjacent to each other so that the light emitted from the light emitting unit is reflected by the retroreflective surface and the reflected light is received by the light receiving unit. It has become. As a result, as will be described later with reference to FIGS. 2 and 3, a detection unit 10 for detecting the presence / absence of the detected object in the front portion of the storage chamber 4 and the angle of the existing detected object with respect to the photoelectric sensors 12 and 14 is configured. doing.

More specifically, of the four inner walls of the front part of the storage chamber 4 (see arrow B), two photoelectric sensors 12 and 14 are arranged on one surface, and retroreflective surfaces 16 are arranged on the remaining three surfaces. Has been done. In the example shown in FIG. 1, the photoelectric sensors 12 and 14 are arranged at the upper ends and the lower ends of the front portion (see arrow B) of the storage chamber 4 on the side surface of the door 6 without the hinge portion.

The retroreflective surface 16 is arranged on the remaining three surfaces of the storage chamber 4 (see arrow B) where the photoelectric sensors 12 and 14 are not arranged, that is, on the upper surface, the lower surface, and the side surface having the hinge portion of the door 6. There is. Examples of the retroreflective surface include those using the refraction of glass beads and those using the reflection of prisms. For example, the retroreflective surface 16 can be provided by attaching a retroreflective sheet to the three front surfaces of the storage chamber 4 or attaching a retroreflective plate.

The arrangement of the photoelectric sensors 12 and 14 and the retroreflective surface 16 is not limited to the case shown in FIG. 1. For example, the photoelectric sensors 12 and 14 are installed on the upper surface and the retroreflective surface 16 is arranged on the remaining three surfaces. You can also do it. The photoelectric sensors 12 and 14 and the retroreflective surface 16 can be arranged on any other surface. It is also possible to arrange two or more arbitrary number of photoelectric sensors on a plurality of surfaces.

In the present embodiment, since the operable angle range (that is, the directivity angle) of the photoelectric sensors 12 and 14 is 90 degrees or more, the photoelectric sensors 12 and 14 can cover the entire opening position of the storage chamber 4. That is, the light from the photoelectric sensors 12 and 14 reaches the entire region of the retroreflective surface 16 arranged on the three front surfaces (see arrow B) of the storage chamber 4, and the reflected light reaches the photoelectric sensors 12 and 14. It is supposed to go back.
Thereby, the presence / absence of the detected object on the virtual plane 60 of the front portion (see arrow B) of the storage chamber 4 shown in FIG. 1B and the angle of the detected object with respect to the photoelectric sensors 12 and 14 can be detected. ..

With the above arrangement, the photoelectric sensors 12 and 14 and the retroreflective surface 16 are appropriately arranged according to the operable angle range of the photoelectric sensors 12 and 14, thereby covering the entire opening position of the storage chamber 4. A reasonable arrangement of possible detection units 10 can be realized.

FIG. 2 is a schematic view showing that a detected object is detected by a detection unit 10 including a retroreflective photoelectric sensor 12 (or 14) and a retroreflective surface 16. FIG. 3 is a schematic view showing that the angle of the detected object with respect to the photoelectric sensor is detected by the position of the photoelectric sensor 12 (or 14) at the light receiving portion 12b (or 14b) where the amount of incident light changes due to shading by the detected object. Is.

As shown in FIG. 2A, if there is no detected object, the light emitted from the light projecting unit 12a (14a) of the photoelectric sensor 12 (14) hits the retroreflective surface 16 and the retroreflective surface 16 The reflected light from is returned to the photoelectric sensor 12 (14) and received by the light receiving unit 12b (14b). On the other hand, as shown in FIG. 2B, if the detected object 50 is present, the light emitted from the light projecting unit 12a (14a) of the photoelectric sensor 12 (14) is blocked by the detected object 50. Therefore, it does not reach the retroreflective surface 16. Even if a part of the light reaches the retroreflective surface 16, the reflected light from the retroreflective surface 16 is blocked by the detection object 50 and does not reach the light receiving portion 12b (14b) of the photoelectric sensor 12 (14). .. The light emitted from the light projecting unit 12a (14a) of the photoelectric sensor 12 (14) hits the detection object 50 and is diffusely reflected. A part of the reflected light due to diffused reflection reaches the light receiving portion 12b (14b) of the photoelectric sensor 12 (14), but the amount of the light is extremely small as compared with the case shown in FIG. 2A.

Amount As described above, the presence or absence of the detected object can be detected by changing the amount of light incident on the light receiving portion 12b (14b) of the photoelectric sensor 12 (14) due to shading by the detected object.

Further, as shown in FIG. 3, the position where the amount of light in the light receiving portion 12b (14b) of the photoelectric sensor 12 (14) changes differs depending on the angle of the detected object with respect to the photoelectric sensor 12 (14). Photoelectric conversion elements are arranged in a matrix in the light receiving unit 12b (14b) shown in FIG. 3, and the position of a region where a change in the amount of light has occurred can be accurately detected.

The angle of the detected object 50 with respect to the photoelectric sensor 12 (14), that is, in the case shown in FIG. 3, a line connecting the region where the amount of light changes in the light receiving unit 12b (14b) and the detected object (detection object center) 50 , Here, let θ be the angle formed by the light receiving portion 12b (14b) and the line parallel to the light receiving surface. The region where the light amount change occurs when the photoelectric sensor 12 (14) has a large angle θ2 is located below the region where the light amount change occurs when the photoelectric sensor 12 (14) has a small angle θ1. ..

That is, the angle θ of the detection object 50 with respect to the photoelectric sensor 12 (14) can be detected by the position of the light receiving unit 12b (14b) in which the amount of light incident by the detection object 50 has changed. As a result, the angle θ of the detected object 50 with respect to the photoelectric sensor 12 (14) can be reliably detected.
The difference in position in the light receiving unit 12b (14b) due to the difference in angle θ is very small compared to the size of the photoelectric sensor 12 (14). Therefore, when expressing the angle of the detection object 50 with respect to the photoelectric sensor 12 (14) (see FIG. 5), between the line connecting the detection object 50 and a predetermined point of the photoelectric sensor 12 (14) and the reference line. There is no problem by showing the angle.

In the present embodiment, the light projecting units 12a (14a) of the photoelectric sensors 12 and 14 emit light in the near infrared region. As a result, it is not easily affected by ambient light, and excellent detection performance can be expected. However, the present invention is not limited to the case where the light projecting unit emits light in the near infrared region, and a light projecting unit that emits light in the visible light region can also be used. For example, an LED light source can be used for the light projecting unit 12a (14a), but a laser light source accompanied by a light diffusing means can also be used.

As the photoelectric sensors 12 and 14, any regression reflection type photoelectric sensor supplied on the market can be used, or a light emitting unit and a light receiving unit created exclusively for this system can be used in combination.

<Shooting device>
A photographing device 20 is provided in the front portion (see arrow B) of the storage chamber 4 of the refrigerator 2. In the example shown in FIG. 1, the photographing device 20 is arranged under the photoelectric sensor 12 on the side surface of the door 6 of the front portion (see arrow B) of the storage chamber 4 without a hinge portion. With this arrangement, the imaging device 20 can acquire image data of the detected object existing at an arbitrary position in the front portion (see arrow B) of the storage chamber 4 of the refrigerator 2. In addition, interference with the light used for detecting the photoelectric sensors 12 and 14 can be avoided.

As the photographing device 20, for example, a device using a photodiode such as a CMOS image sensor or a CCD image sensor can be used. The photographing device 20 may acquire a still image or may acquire a moving image. When acquiring a moving image, an image at a predetermined timing of the moving image can be taken out and used as a still image.

The photographing by the photographing apparatus 20 may continue the photographing at all times. When continuous shooting is continued, moving images may be taken, or still images may be taken continuously at regular intervals. It is also possible to start photographing when the photoelectric sensors 12 and 14 detect the presence of the detected object and end the photographing when the presence of the detected object is no longer detected. Further, the photographing can be started when the door 6 for opening and closing the storage chamber 4 is opened, and ended when the door is closed. Shooting can be started and ended at any other timing.

<Explanation of stored items management system>
FIG. 4 is a block diagram showing an outline of a stored object management system according to one embodiment of the present invention. In the storage management system 40, two photoelectric sensors 12 and 14 forming a detection unit 10 together with the retroreflective surface 16, a photographing device 20, and a control unit in which the photoelectric sensors 12 and 14 and the photographing device 20 are electrically connected are electrically connected. 30 and are provided. The photoelectric sensors 12 and 14 include light emitting units 12a and 14a and light receiving units 12b and 14b, respectively. The control unit 30 can be provided exclusively for the stored item management system, the control device of the refrigerator 2 can perform the function of the control unit, and the remote server can function as the control unit by using an internet line or the like. It can also be fulfilled.

FIG. 5 is a schematic view showing that the position of the detected object 50 on the virtual plane 60 at the front of the storage chamber 4 is calculated based on the angles detected by the two photoelectric sensors 12 and 14. A method of calculating the position of the detected object 50 on the virtual plane 60 by the control unit 30 of the storage management system 40 will be described with reference to FIG.
In FIG. 5, the angle of the detection object 50 with respect to the photoelectric sensors 12 and 14 means the angle formed by the line connecting the photoelectric sensors 12 and 14 and the detection object 50 with the reference line. The reference line in FIG. 5 is a line connecting the two photoelectric sensors 12 and 14, which coincides with a line parallel to the light receiving surface of the light receiving unit 12b (14b) shown in FIG.

Let L be the distance between the two photoelectric sensors 12 and 14. Let α be the angle formed by the line connecting the photoelectric sensor 12 and the detection object 50 with the reference line, and let β be the angle formed by the line connecting the photoelectric sensor 14 and the detection object 50 with the reference line. From the angles α and β detected by the two photoelectric sensors 12 and 14, the two-dimensional position of the detected object 50 on the virtual plane 60 at the front of the storage chamber 4 can be calculated by a triangulation method. That is, the intersection of the straight line extended from the photoelectric sensor 12 at the elevation angle α and the straight line extended from the photoelectric sensor 14 at the elevation angle β is the position of the detection object 50.

Basically, the two-dimensional position of the detected object can be determined by using two photoelectric sensors, but the two-dimensional position of the detected object can be determined in the same manner even when three or more photoelectric sensors are used. be able to. In this case, the intersections of the straight lines extended at the elevation angle detected by the photoelectric sensor may not match to some extent. At that time, the average position of a plurality of intersections can be set as the position of the detected object. In FIG. 5, the photoelectric sensor is arranged on one of the four front surfaces of the storage chamber 4, but the present invention is not limited to this, and two or more photoelectric sensors are arranged on two or more surfaces. Even when they are arranged, the position of the detected object can be determined in the same manner.

The control unit 30 can determine the type of the detected object based on the image data captured by the imaging device 20. Therefore, any image analysis technique can be used.
The type of the detected object to be determined includes the determination of whether or not the stored object is present in the user's hand, and the determination of the type of the stored object held in the user's hand. That is, the detected object may be only the user's hand, or may be the user's hand and the stored object held in the hand. When analyzing the acquired image data at an arbitrary timing between the timing when the detection of the detected object is started and the timing when the detection is finished, only the arm or wrist of the user may be identified. The information identified in this way can be stored in the storage device of the system.

The types of stored items to be judged include the types of vegetables or fruits (for example, carrots, Chinese cabbage, mandarins, bananas, etc.), the types of fish (for example, ordinary fish, octopus, shrimp, etc.), and the types of meat (beef, pork, etc.). (Chicken, etc.) can be exemplified. In addition, food in a translucent container (for example, 6 eggs in a transparent pack) can also be identified by the container (transparent pack), the type of food (eggs), and the number (6 eggs). it can. Food in non-translucent containers and packaging can be stored by adding additional information such as barcode reading, voice input, and input from mobile terminals.
However, the present invention is not limited to these, and any other type can be determined based on the stored data.

If the size of the stored items is small, the stored items may be hidden in the user's hand and the stored items may not be visible. To deal with this, multiple imaging devices are placed at different positions and viewed from different angles. It is also conceivable to be able to determine the type of stored items by taking a picture.

<Control processing by the control unit>
FIG. 6 is a flowchart showing a main routine showing an example of control processing for managing stored items. The control process by the control unit 30 of the storage management system 40 will be described in detail with reference to FIG.

In FIG. 6, first, it is determined whether or not the detected object 50 is detected in the front portion of the storage chamber 4 of the refrigerator 2 by the photoelectric sensors 12 and 14 (step S2). If it is determined by this determination that the detected object 50 has not been detected (NO), this determination process is repeated as it is. That is, it is in a standby state until the detected object 50 is detected.

If it is determined in step S2 that the detection object 50 is detected (YES) by the photoelectric sensors 12 and 14, then the photoelectric sensors 12 and 14 of the detected objects detected by the photoelectric sensors 12 and 14 are referred to. The angles α and β (see FIG. 5) are acquired (step 4). Then, as shown in FIG. 5, the two-dimensional position of the detected object 50 on the virtual plane 60 is calculated by the method of triangulation (step S6).

Next, shooting by the photographing device 20 is started (step S8), and image data immediately after the detection of the detected object 50 is acquired (step S10). Then, the image analysis program is started to determine the type of the detected object (step S12). The determination of the type of the detected object includes the determination of whether or not the stored object is present in the user's hand, and the determination of the type of the stored object held in the user's hand.

As a result of the determination process in step S12, it is determined whether or not there is a stored item in the hand (step S14). If it is determined by this judgment that the stored items exist (YES), it is assumed that the stored items are stored in the storage chamber 4. Therefore, the position of the detected object calculated in step S6 is determined as the storage position of the stored object, and is stored in the storage device of the control unit 30 together with the type of stored object (for example, carrot) determined in the determination process of step S12 (for example, carrot). Step S18), the process proceeds to step S20. If it is determined in step S14 that there is no stored item (NO), it is determined that only the hand of the user who has nothing exists, and the process proceeds to step S20 as it is.

In step S20, the photoelectric sensors 12 and 14 determine whether or not the detection of the detected object 50 is completed, that is, whether or not the user's hand is no longer present in the front portion of the storage chamber 4 (step S20). If it is determined by this determination that the detection of the detected object 50 has not been completed (NO), this determination process is repeated as it is. That is, it is in a standby state until the detection of the detected object 50 is completed.
If it is determined in step S20 that the detection of the detected object 50 is completed (YES), then the image taken by the photographing device 20 is terminated (step S22), and the image immediately before the detection of the detected object 50 is completed. Acquire data (step S24). Then, the image analysis program is started to determine the type of the detected object (step S26).

As a result of the determination process in step S26, it is determined whether or not there is a stored item in the hand (step S28). If it is determined in this judgment that the stored items exist (YES), it is assumed that the stored items stored in the storage chamber 4 are taken out. Therefore, it is stored in the storage device of the control unit 30 based on the position of the detected object calculated in step S6, that is, the position of taking out the stored object, and the type of stored object (for example, Chinese cabbage) determined in the determination process of step S26. The data of the stored item corresponding to the stored storage position is deleted (step S30), and this subroutine is terminated. If it is determined in step S28 that there is no stored item (NO), it is determined that only the hand of the user who has nothing exists, and this subroutine is terminated as it is.

As described above, since the position and type of the stored items can be detected at the opening position of the storage room, there is no problem even if the stored items are stacked one above the other, and the stored items in the storage room are stored. Regardless of the storage location or how to place the stored items, the type of stored items and the storage position in the storage room can be reliably grasped.

In particular, when the stored object is present in the user's hand immediately after the detection of the presence of the detected object 50 is started and the stored object is not present in the user's hand immediately before the detection of the presence of the detected object is completed. It is determined that the stored items are stored in the storage room, and the storage position and the type of the stored items are stored. On the other hand, if there is no stored item in the user's hand immediately after starting the detection of the presence of the detected object 50, and there is a stored item in the user's hand immediately before the detection of the presence of the detected object is completed. Is determined that the stored item has been taken out of the storage room, and the data of the stored item at the corresponding storage position is deleted.

By such control processing, it is possible to surely obtain important information for the management of the stored items with a simple device configuration such as the photoelectric sensors 12 and 14 and the photographing device 20.

If the user tries to store the stored items, but does not put the stored items in the storage room and withdraws while holding the stored items as they are, the control process of step S18 is performed to store the stored items. Since the storage position and type are stored and the stored data of the stored items is deleted by the control process in step S30, accurate management of the stored items can be realized even in such a case. In addition, in this system, even if 6 eggs in a transparent pack are stored as described above and one of them is taken out, 5 eggs in the transparent pack remain. You can know exactly what you are doing.

In the control process shown in FIG. 6, the photographing apparatus 20 starts photographing based on the photoelectric sensors 12 and 14 starting to detect the presence of the detected object 50. As a result, it is possible to efficiently and surely acquire image data when storing or taking out the stored items.

Further, based on the fact that the photoelectric sensors 12 and 14 have finished detecting the presence of the detected object 50, the photographing device 20 ends the photographing. As a result, it is possible to efficiently acquire image data when storing or taking out the stored items.

However, as described above, the photographing by the photographing apparatus is not limited to the case where the photographing is started when the presence of the detected object is detected and ends when the presence is no longer detected. For example, you may be shooting all the time, you can start shooting when the door that opens and closes the storage room opens, end it when it closes, or at any other time. You can also start and end shooting.

The control process for determining the type of the detected object based on the image data is not limited to only when the presence of the detected object is detected or when the existence is no longer detected, and can be performed at any timing during that period. For example, when the arm or wrist of the user is identified by this determination, it can be determined that the user is in the process of placing or removing the stored object in the storage chamber.

Although the embodiments and embodiments of the present invention have been described, the disclosed contents may be changed in the details of the configuration, and the invention in which the embodiments and the combinations and orders of the elements in the embodiments are changed are requested. It can be realized without deviating from the scope and ideas of.

2 Refrigerator 4 Storage room 6 Door 10 Detection unit 12 Photoelectric sensor 12a Light projecting unit 12b Light receiving unit 14 Photoelectric sensor 14a Light projecting unit 14b Light receiving unit 16 Retroreflective surface 20 Imaging device 30 Control unit 40 Storage device management system 50 Detection device 60 Virtual Plane

Claims (5)

Includes two or more retroreflective photoelectric sensors and retroreflective surfaces located in the front of the refrigerator storage chamber, the presence or absence of a detection object in the front of the storage chamber, and the photoelectric sensor of the detection object in existence. A detector that detects the angle with respect to
The imaging device installed at the front of the storage room and
A control unit that performs control processing based on data from the detection unit and the imaging device, and
With
The control unit
Based on the angles detected by the two or more photoelectric sensors, the position of the detected object on the virtual plane at the front of the storage chamber is calculated.
A stored object management system characterized in that the type of the detected object is determined based on the image data photographed by the photographing device.

Claim 1 is characterized in that two or more of the photoelectric sensors are arranged on one of the four open-side surfaces of the inner wall of the storage chamber, and the retroreflective surface is arranged on the remaining three surfaces. Storage management system described in.
The stored object management system according to claim 1, wherein the photographing apparatus starts photographing based on the photoelectric sensor starting to detect the presence of the detected object.
The stored object management system according to claim 3, wherein the imaging device terminates imaging based on the photoelectric sensor ending detection of the presence of the detected object.
The control unit compares the image data immediately after the photoelectric sensor starts detecting the presence of the detected object with the image data immediately before the photoelectric sensor finishes detecting the presence of the detected object, and stores the image data. The stored item management system according to any one of claims 1 to 4, wherein it is determined whether an item is stored or a stored item is taken out.