JP2019138622A - Storing device - Google Patents

Abstract

To provide a storing device that can improve quality of an image inside a storing device, which is provided to a user, and to provide an image processing device, an image processing method and a program.SOLUTION: A storing device includes an opening door, a storing chamber and a generation part. The storing chamber stores an object. The generation part, when at least one of an opening operation and a closing operation is conducted on the opening door, varies the physical relationship with the storing chamber in accordance with the opening operation and the closing operation and synthesizes a plurality of imaged images that are acquired by imaging by an imaging part for imaging a plurality of images of the storing chamber.SELECTED DRAWING: Figure 5

Description

  Embodiments described herein relate generally to a storage device.

  2. Description of the Related Art Conventionally, in a storage device such as a refrigerator, when a door (a hinged door) is opened and closed, the inside of the storage device is imaged by an imaging device such as a camera attached to the door, and an image obtained by the imaging is presented to the user It has been known.

JP 2001-317858 A

  However, the conventional technique has a problem that an image including an area to be presented to the user cannot be obtained depending on, for example, the angle of the door when opening and closing.

  The problem to be solved by the present invention is to provide a storage device capable of improving the quality of images in the storage device presented to the user.

  The storage device according to the embodiment includes a hinged door, a storage room, and an imaging unit. The storage room stores objects. The imaging unit is arranged to be inclined in the normal direction of the door surface of the hinged door, and images the storage room. The imaging unit starts imaging when the angle between the front of the storage room and the hinged door changes in the increasing direction, stops imaging when the amount of change in the increasing direction of the angle falls below a threshold value, and then changes in the decreasing direction of the angle When the amount exceeds the threshold value, imaging is started again, and when the amount of change in the angle decreasing direction is equal to or less than the threshold value, imaging is stopped.

The front view of the refrigerator of embodiment. The top view of the refrigerator of embodiment. The top view of the refrigerator of embodiment. The figure which shows an example of the captured image corresponding to each camera position of embodiment. 1 is a diagram illustrating an example of a configuration of an image processing apparatus according to an embodiment. The figure for demonstrating the angle of the hinged door of embodiment. The figure for demonstrating the correction | amendment with respect to the captured image of embodiment. The figure for demonstrating the correction | amendment with respect to the captured image of embodiment. The figure which shows an example of the 1st synthesized image of embodiment. The schematic diagram which shows an example of the 1st synthetic | combination image and insufficient area | region of embodiment. The conceptual diagram which shows the production | generation method of the 2nd synthesized image of embodiment. 1 is a diagram illustrating an example of a configuration of an image processing apparatus according to an embodiment. The schematic diagram for demonstrating the 1st reliability of embodiment.

  Hereinafter, embodiments of a storage device, an image processing device, an image processing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. In the following description, a refrigerator will be described as an example of a storage device, but the present invention is not limited to this.

(First embodiment)
FIG. 1 is a front view of the refrigerator 1 of the present embodiment. As shown in FIG. 1, the refrigerator 1 includes a hinged door 10 and a main body 20.

  The "folding door" refers to one end (one of the side edges of the door) fixed with a hinge or the like, and the other end (door) with the fixed portion as an axis. The door (door) which moves so that the other side edge of the both side edges of (1) may draw an arc.

  As shown in FIG. 1, the main body 20 has a storage chamber 21 for storing an object. FIG. 1 illustrates the state after the opening operation of the hinged door 10 is performed so that the entire entrance of the storage chamber 21 is completely exposed in a front view. Here, although illustration is omitted, the hinged door 10 is provided with a gripping part for the user to open and close the hinged door 10. In the example of FIG. 1, the hinged door 10 is configured with a single door, but is not limited thereto, and may be, for example, a double hinged door configured with two doors, for example, a type that opens up and down. It may be a door. Moreover, in the example of FIG. 1, the door pocket 11 for accommodating containers, such as a PET bottle, is provided in the rear surface (surface by the side of the storage chamber 21) of the hinged door 10. As shown in FIG.

  In the example of FIG. 1, the hinged door 10 has a camera in which the positional relationship with the storage chamber 21 changes according to the opening or closing operation of the hinged door 10 and images a plurality of storage chambers 21 (images are continuously captured). 30 is attached. In this specification, “imaging” refers to converting an image of a subject (imaging target) imaged by an optical system such as a lens into an electrical signal. The camera 30 only needs to have a function of capturing a moving image, and can be constituted by, for example, a visible light camera or an infrared camera using a CCD or a CMOS. In this example, it can be considered that the camera 30 corresponds to the “imaging unit” in the claims. In this example, the camera 30 is externally attached to the hinged door 10. However, the present invention is not limited to this. For example, the camera 30 may be incorporated (embedded) in the hinged door 10.

  In the present embodiment, imaging by the camera 30 is performed when the hinged door 10 is opened and closed. Moreover, it is preferable to determine the installation location of the camera 30, the optical axis direction of the camera 30 at the time of installation, the angle of view, and the like according to the range of the storage chamber 21 presented to the user. For example, when presenting at least the entire entrance of the storage chamber 21, the optical axis direction and angle of the camera 30 can be determined so that at least the entire entrance of the storage chamber 21 is imaged when the hinged door 10 is opened and closed. preferable. Further, in the present embodiment, a case where the number of cameras 30 is one will be described as an example. However, the present invention is not limited to this. For example, the range of the storage chamber 21 presented to the user using a plurality of cameras 30 is used. It is also possible to take a form of imaging.

  FIG. 2 is a plan view of the refrigerator 1 as viewed from above. FIG. 2 shows an example in which the camera 30 is installed at a position about 2/3 of the total length of the upper edge of the door from the base (part corresponding to the rotation axis of the door) above the hinged door 10. ing. In the example of FIG. 2, the camera 30 is installed with the camera optical axis directed in the direction of an angle θ with respect to the normal direction of the door surface (front surface or rear surface of the door).

  FIG. 3 is a plan view when the refrigerator 1 is viewed from above. When the opening operation of the hinged door 10 is performed until the entire entrance of the storage chamber 21 is completely exposed in a front view, the imaging by the camera 30 is performed. It shows how it is done. In the example of FIG. 3, when the opening operation is performed, the camera 30 captures an image while moving from position P1 → position P2 → position P3. The camera 30 images the right side of the entrance of the storage chamber 21 at the position P1, images the vicinity of the center of the entrance of the storage chamber 21 at the position P2, and images the left side of the storage chamber 21 at the position P3. Until the end, the entire entrance of the storage chamber 21 is imaged.

  FIG. 4A shows a captured image corresponding to the position P1 (an image obtained by capturing with the camera 30 at the position P1). FIG. 4B shows a captured image corresponding to the position P2. Further, (c) of FIG. 4 represents a captured image corresponding to the position P3.

  FIG. 5 is a diagram illustrating an example of the configuration of the image processing apparatus 100 mounted on the refrigerator 1. Here, a case where the image processing apparatus 100 is mounted on the refrigerator 1 will be described as an example. However, the present invention is not limited to this, and the image processing apparatus 100 may be provided outside the refrigerator 1, for example.

  As illustrated in FIG. 5, the image processing apparatus 100 includes a camera control unit 110, an acquisition unit 120, a generation unit 125, and a display control unit 160.

  The camera control unit 110 controls the start or end of imaging by the camera 30 based on the signal from the sensor 40. In this example, the sensor 40 is an angle sensor that detects the angle φ of the hinged door 10 at a predetermined cycle. As the angle sensor constituting the sensor 40, various known configurations can be used. In this example, as shown in FIG. 6, the angle φ represents an angle formed by the front surface of the main body 20 and the hinged door 10. If the angle φ is “0 degrees”, the hinged door 10 is in a closed state, and if the angle φ is “90 degrees”, the hinged door 10 is opened so that the entire entrance of the storage chamber 21 is exposed in front view. Can be considered a state. Each time the sensor 40 according to the present embodiment detects the angle φ, a signal indicating the detected angle φ (which may be referred to as an “angle signal” in the following description) is acquired by the camera control unit 110 and the acquisition described later. To the unit 120.

  Returning to FIG. 5, the description of the camera control unit 110 will be continued. As described above, in the present embodiment, imaging by the camera 30 is performed when the hinged door 10 is opened and closed. More specifically, it is as follows. For example, when the angle φ indicated by the angle signal acquired from the sensor 40 in the predetermined cycle changes in an increasing direction from “0 degree”, the camera control unit 110 determines that the opening operation has started, and the camera 30 Control to start imaging. After performing the control to start imaging by the camera 30, the camera control unit 110 completes the opening operation when the amount of change in the increasing direction of the angle φ indicated by the angle signal from the sensor 40 is equal to or less than the threshold value. And control to end the imaging by the camera 30 is performed. Thereafter, when the amount of change in the decreasing direction of the angle φ indicated by the angle signal from the sensor 40 exceeds the threshold value, the camera control unit 110 determines that the closing operation has started and performs control to start imaging by the camera 30. . After performing the control to start imaging by the camera 30, the camera control unit 110 determines that the closing operation is completed when the angle φ indicated by the angle signal from the sensor 40 indicates “0 degree”. Control to end imaging by the camera 30 is performed.

  Next, the acquisition unit 120 illustrated in FIG. 5 will be described. The acquisition unit 120 acquires the captured image obtained by the imaging and the imaging time indicating the time when the imaging is performed from the camera 30 every time the imaging by the camera 30 is performed, and sends the captured image to the detection unit 131 described later. Supply. In this example, the acquisition unit 120 acquires the angle signal from the sensor 40 at the predetermined cycle and supplies the angle signal to the detection unit 131 described later.

  For example, the above-described camera control unit 110 is not provided, and imaging by the camera 30 is always performed, and the acquisition unit 120 performs an opening operation or a closing operation of the hinged door 10 based on an angle signal from the sensor 40. Only when it is determined that it is being performed, the captured image and the captured time may be acquired from the camera 30.

  Next, the generation unit 125 illustrated in FIG. 5 will be described. The generation unit 125 synthesizes a plurality of captured images obtained by imaging with the camera 30 when at least one of the opening operation and the closing operation of the hinged door 10 is performed. Specific contents will be described below. As illustrated in FIG. 5, the generation unit 125 includes a first generation unit 130, a storage unit 140, and a second generation unit 150.

  When at least one of the opening operation and the closing operation of the hinged door 10 is performed, the first generation unit 130 includes a plurality of objects so that the same subject included in the plurality of captured images obtained by imaging with the camera 30 overlaps. A first synthesized image (panoramic image) obtained by synthesizing the captured images is generated. This will be specifically described below. In the present embodiment, the first generation unit 130 includes a detection unit 131, a correction unit 132, and a synthesis unit 133.

  The detection unit 131 detects the relative position of the camera 30 with respect to the storage chamber 21. In this example, the detection unit 131 can be considered to have a function corresponding to the “position detection unit” in the claims. More specifically, it is as follows. In this example, the detection unit 131 stores installation information such as a predetermined mounting position of the camera 30 (in this example, the position of the camera 30 attached to the hinged door 10) and a direction, and the installation information such as the camera 30 or a memory (not illustrated). From an external device). For example, the detection unit 131 can also acquire environment information indicating an imaging environment such as brightness and shutter speed during imaging from the camera 30. Further, for example, the detection unit 131 can also acquire the specifications of the camera 30 such as the resolution and angle of view of the camera 30 from the camera 30 or a memory (not shown) (which may be an external device).

  For example, the detection unit 131 defines a coordinate system with the base position of the hinged door 10 as the origin, the angle φ indicated by the angle signal supplied from the acquisition unit 120 described above, and the attachment position of the camera 30 indicated by the installation information described above. Thus, the coordinate value of the camera 30 in the coordinate system can be calculated, and the calculated coordinate value can be detected as the position of the camera 30 (the relative position of the camera 30 with respect to the storage chamber 21). In the present embodiment, the detection unit 131 detects the position of the camera 30 at the time of imaging (more specifically, based on information (captured image, imaging time and angle signal) supplied from the acquisition unit 120, the above-described installation information, and the like. In other words, a plurality of camera positions corresponding to a plurality of imaging times are detected. Then, the detection unit 131 supplies detection information in which a captured image obtained by imaging at the position is associated with a plurality of detected positions (position of the camera 30 at the time of imaging) to the correction unit 132 described later. To do.

  For example, the detection unit 131 detects (determines) the start and end of the opening operation of the hinged door 10 or the start and end of the closing operation based on the angle signal supplied from the acquisition unit 120 at the predetermined period. You can also. For example, when the opening operation is performed (every time a series of operations from the start to the end of the opening operation is completed), the detection unit 131 sends the detection information corresponding to the opening operation to the correction unit 132 described later. It can also be supplied. Further, for example, when the closing operation is performed, the detection unit 131 indicates the above-described detection information corresponding to the closing operation, which will be described later, every time a series of operations from the start to the end of the closing operation is completed. 132 can also be supplied.

  For example, the detection unit 131 may be configured to also function as the acquisition unit 120 described above. In other words, the acquisition unit 120 may not be provided.

  Further, the method for detecting the position of the camera 30 is not limited to the above, and various methods are conceivable. For example, a gyroscope, GPS, or the like can be attached to the camera 30, and the position of the camera 30 can be obtained based on information obtained by the gyroscope, GPS, or the like. In addition, for example, a marker or the like may be arranged in the imaging range by the camera 30 and the position of the camera 30 may be obtained from the position and size of the marker in the captured image. You may obtain | require the position of the camera 30 by methods other than the method illustrated above. In short, the detection unit 131 only needs to have a function of detecting the position of the camera 30 (the position of the camera 30 when imaging is performed).

  The correction unit 132 is a reference indicating a reference position among the positions of the camera 30 that change with time (changes with time) when at least one of the opening operation and the closing operation of the hinged door 10 is performed. Correction for aligning the scale of the subject included in the captured image corresponding to each position is performed on the captured image corresponding to a position other than the position. In the present embodiment, the correction unit 132 refers to the detection information supplied from the detection unit 131, identifies a position other than the reference position among a plurality of positions included in the detection information, and corresponds to the identified position. The attached captured image is corrected. This will be specifically described below.

  As an example, the opening operation shown in FIG. 3 is assumed. In this example, in a series of opening operations, the position of the camera 30 changes with time from P1 → P2 → P3. Therefore, in the detection information corresponding to the opening operation, the images at the positions P1 and P1 are captured. The obtained captured image (hereinafter referred to as “captured image Ip1”) is associated, and the position P2 and the captured image obtained by imaging at the position P2 (hereinafter referred to as “captured image Ip2”). Are associated with each other, and the position P3 is associated with a captured image obtained by imaging at the position P3 (hereinafter referred to as “captured image Ip3”). In the following description, when the captured image Ip1, the captured image Ip2, and the captured image Ip3 are not distinguished from each other, they may be simply referred to as “captured image Ip”.

  Here, when the same object inside the storage chamber 21 is imaged at the position P1, the position P2, and the position P3, the distance to the object varies depending on the position, and therefore included in each captured image Ip. The size of the object is also different. Therefore, it is difficult to simply connect the captured image Ip1, the captured image Ip2, and the captured image Ip3. Therefore, in the present embodiment, a reference viewpoint position (reference position) is set, and a captured image corresponding to a position other than the reference position is corrected. The reference position is not limited as long as it is not an unobservable place such as the back side of the refrigerator 1 as long as the storage chamber 21 can be observed. However, since it is not preferable that a first composite image (panoramic image) described later is greatly different from the scenery to be presented to the user, in this embodiment, the optical axis of the camera 30 is orthogonal to the entrance surface of the storage chamber 21. The position (P2 shown in FIG. 3) of the camera 30 at the time is set as a reference position.

  For example, as a correction of the captured image corresponding to the position P1, as shown in FIG. 7, the distance L from the entrance of the storage chamber 21 is equal to the position P2, and the optical axis of the camera 30 at the position P1 and the storage chamber 21 It can also be corrected as if the image was taken at a position P1 ′ where the optical axes of the camera 30 intersect at the intersection c1 with the entrance. As shown in FIG. 8, the correction can use a simple geometric transformation such as an affine transformation. The correction is not limited to the above-described method, and any correction is possible as long as the scales of the objects shown in the captured images corresponding to the respective positions are equal. The correction unit 132 corrects the captured image corresponding to each of the positions P1 and P3 other than the reference position P2 as described above. Then, the correction unit 132 corrects the captured image (in the following description, the corrected captured image may be referred to as a “corrected image”) and a reference image (this example) indicating the captured image corresponding to the reference position. Then, the captured image Ip2) is supplied to the composition unit 133 described later.

  The combining unit 133 illustrated in FIG. 5 combines the reference image and the corrected image supplied from the correcting unit 132 to generate a first combined image. In this example, the synthesizing unit 133 compares the reference image with each of a plurality of correction images included in the correction image group (in this example, the correction image corresponding to the position P1 and the correction image corresponding to the position P3). The same range, that is, the region where the same object is imaged is superimposed. As a result, as shown in FIG. 9, a first composite image in which the image area is expanded beyond the range of the reference image is generated. At this time, if the images are synthesized in order from an image close to the reference position, a first synthesized image with higher image quality can be easily generated. In this embodiment, the images are combined after being corrected, but a method of correcting the images while combining them is also conceivable. For example, each image (a captured image corresponding to a position other than the reference position) may be deformed so as to optimally overlap the reference image. Specifically, while trying various deformations, searching for the optimal overlapping deformation with respect to the reference image, determining the optimal overlapping deformation, and then combining them. It can be considered that the method in the present embodiment described above is a method for determining this optimal deformation using the position of the camera 30.

  The first generation unit 130 described above generates a first composite image when the opening operation or the closing operation of the hinged door 10 is performed (every time the opening operation or the closing operation of the hinged door 10 is performed), and the storage unit 140, and the detection information used to generate the first composite image and the first composite image is supplied to a second generation unit 150 described later. The storage unit 140 stores the first composite image generated by the first generation unit 130 in time series.

  Next, the 2nd production | generation part 150 shown in FIG. 5 is demonstrated. When the first composite image generated by the first generation unit 130 described above does not include a part of the target area indicating the area to be presented to the user in the storage room 21, the second generation unit 150 A part of the target region is complemented using the first composite image, and a second composite image to be presented to the user is generated. This will be specifically described below. In this example, the target area is the entire area of the entrance of the storage chamber 21.

  For example, when the hinged door 10 cannot be opened until the entire entrance of the storage chamber 21 is completely exposed and the entire entrance of the storage chamber 21 cannot be imaged, this opening operation (hereinafter referred to as “non-fully open operation”). The first composite image (the first composite image generated when the non-fully-opening operation is performed) corresponding to the case may not include a part of the entire entrance of the storage chamber 21 (not shown). It becomes. Similarly, for example, when a closing operation (hereinafter, referred to as “insufficient closing operation”) is performed with the position of the hinged door 10 not opened until the entire entrance of the storage chamber 21 is completely exposed. The first composite image corresponding to the under-close operation (the first composite image generated when the under-close operation is performed) is an image that does not include a part of the entire entrance of the storage chamber 21.

  Here, the hinged door 10 is opened until the entire entrance of the storage chamber 21 is completely exposed based on the position of the camera 30 that changes over time in the non-fully opening operation or the under-closing operation described above (the position of the camera 30 at the time of imaging). Out of the range of the first composite image corresponding to the non-full-open operation or the under-close operation among the image regions of the first composite image generated in this case (in the following description, sometimes referred to as “entire view region”). (Hereinafter, sometimes referred to as “insufficient region”).

  In the present embodiment, when the second generation unit 150 acquires the first composite image and the detection information from the first generation unit 130 (every time it is acquired), the position of the camera 30 included in the detection information. From this, it is determined whether or not the first composite image has a full view area. In other words, the second generation unit 150 does not perform an opening operation corresponding to the first composite image acquired from the first generation unit 130 from the position of the camera 30 included in the detection information acquired from the first generation unit 130. It can be considered that it is determined whether or not the fully-opening operation or the closing operation corresponding to the first composite image is an under-closing operation.

  The second generation unit 150 determines that the first composite image acquired from the first generation unit 130 (which may be referred to as “the latest first composite image” in the following description) does not have a full view area. In this case, the above-described insufficient region is obtained from the position of the camera 30 included in the detection information (detection information acquired from the first generation unit 130) used for generating the latest first composite image. FIG. 10 is a schematic diagram showing an example of the latest first composite image and the lacking area in this case.

  Then, the second generation unit 150 includes a past scene area from the past first composite image (the first composite image generated before the latest first composite image) stored in the storage unit 140. First composite image (preferably, a past first composite image having a panoramic view area and corresponding to a time closest to the current time) is selected from among the selected past first composite images. By complementing the area corresponding to the shortage area with respect to the latest first synthesized image, a second synthesized image in which the image area is extended to the outside of the range of the latest first synthesized image is generated. FIG. 11 is a conceptual diagram showing a method for generating a second composite image in this case. The second composite image generated in this way is an image presented to the user.

  Even if the first composite image acquired from the first generation unit 130 has a full view area, for example, when an object to be taken in or out of the user's hand or the storage chamber 21 enters the imaging range of the camera 30 during imaging, It is also assumed that a part of the entire entrance of the storage room 21 is hidden. In this case, for example, the second generation unit 150 includes a plurality of positions (a position of the camera 30 at the time of imaging) included in the detection information acquired from the first generation unit 130, and a plurality of positions corresponding to the plurality of positions on a one-to-one basis. A first complement target area that indicates an area in which an object existing in front of the entrance of the storage room 21 is imaged in the entire view area by estimating the distance to the subject using a captured image and the like using a method such as motion stereo. Can be requested. Then, the second generation unit 150 selects the past first composite image in which the entire entrance of the storage chamber 21 is imaged from the past first composite images stored in the storage unit 140, and the selected past The second synthesized image can also be generated by complementing the area corresponding to the first complement target area described above in the first synthesized image with respect to the latest first synthesized image.

  Even if the first composite image acquired from the first generation unit 130 has a full view area, a part of the entire entrance of the storage chamber 21 is not captured because the moving speed of the camera 30 at the time of opening and closing is high. The case is also assumed. In this case, for example, the second generation unit 150 has not captured a part of the entire entrance of the storage chamber 21 in the entire view area based on the plurality of positions included in the detection information acquired from the first generation unit 130. A second complementing target region indicating a region (which can be considered as a frame dropping region) can be obtained. Then, the second generation unit 150 selects the past first composite image in which the entire entrance of the storage chamber 21 is imaged from the past first composite images stored in the storage unit 140, and the selected past The second synthesized image can be generated by complementing the latest first synthesized image with the area corresponding to the above-described second complementing target area in the first synthesized image.

  In short, the second generation unit 150 is a target region (in this example, the entrance of the storage chamber 20) in which the first composite image generated by the first generation unit 130 described above indicates the region to be presented to the user in the storage chamber 21. In the case where a part of the entire region is not included, a part of the target area may be complemented using the past first composite image, and a second composite image to be presented to the user may be generated.

  Next, the display control unit 160 shown in FIG. 5 will be described. The display control unit 160 performs control to display the second composite image generated by the second generation unit 150 on a display unit 50 (which may be configured with a liquid crystal display or the like) 50 that displays the image. In this example, the display control unit 160 performs control to display the generated second composite image on the display unit 50 every time the second composite image is generated by the second generation unit 150. The displayed second composite image is updated to the latest second composite image. Moreover, in this example, although the display part 50 is provided in the front surface of the hinged door 10, it is not restricted to this, The installation location of the display part 50 is arbitrary. For example, the display unit 50 may be provided outside the refrigerator 1. For example, the display unit 50 may be mounted on an external device (for example, a TV, a PC, a mobile phone terminal, or the like) that can be connected (communicable) with the image processing apparatus 100.

  In addition, the timing which displays a 2nd synthesized image is arbitrary, The form which always displays may be sufficient and the form which displays according to the request | requirement from a user may be sufficient. For example, the user can request the display of the second composite image via a button operation, voice, mail, remote control or the like. Further, for example, the second composite image may be displayed at regular time intervals.

  The image processing apparatus 100 described above has a hardware configuration including a CPU (Central Processing Unit), a ROM, a RAM, a communication I / F device, and the like. The function of each unit described above (camera control unit 110, acquisition unit 120, first generation unit 130 (detection unit 131, correction unit 132, synthesis unit 133), second generation unit 150, display control unit 160) This is realized by expanding and executing the program stored in the RAM on the RAM. In addition, the present invention is not limited to this, and at least a part of the functions of each unit described above can be realized by a dedicated hardware circuit (for example, a semiconductor integrated circuit). Further, for example, the storage unit 140 described above can be realized by a ROM, an external storage device (for example, HDD), or the like.

  Further, the program executed by the image processing apparatus 100 described above may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program executed by the image processing apparatus 100 described above may be provided or distributed via a network such as the Internet. The program executed by the image processing apparatus 100 described above may be provided by being incorporated in advance in a non-volatile recording medium such as a ROM.

  In the prior art, for example, even when the user's hand enters the imaging range at the same time as the door is opened or closed, or even when the door moving speed is high, an image including an area to be presented to the user cannot be obtained. Occur. That is, the conventional technique has a problem that the quality of the image in the storage device presented to the user is low. On the other hand, as described above, in the present embodiment, when the opening operation or the closing operation of the hinged door 10 is performed, by combining a plurality of captured images obtained by imaging with the camera 30, The entire region (in this embodiment, the entire entrance of the storage chamber 21) can be expressed. More specifically, in this embodiment, when the opening operation or the closing operation of the hinged door 10 is performed, a plurality of imaging is performed so that the same subject included in the plurality of imaging images obtained by imaging with the camera 30 overlaps. A first composite image is generated by combining the images. When the generated first composite image does not include a part of the target area indicating the area to be presented to the user in the storage room 21, a part of the target area is used using the past first composite image. To generate a second composite image for presentation to the user. Therefore, for example, even if the user does not open the hinged door 10 until the entire entrance of the storage chamber 21 is completely exposed, the second composite image in which the entire entrance of the storage chamber 21 is captured for the user. Can be presented. That is, according to this embodiment, the advantageous effect that the quality of the image in the refrigerator 1 presented to the user can be improved can be achieved.

(Second Embodiment)
Next, a second embodiment will be described. Description of parts common to the first embodiment described above will be omitted as appropriate.

  FIG. 12 is a diagram illustrating an example of the configuration of the image processing apparatus 200 according to the second embodiment. As illustrated in FIG. 12, the image processing apparatus 200 further includes a setting unit 170.

  In the present embodiment, the detection unit 131 has a function of detecting the relative speed of the camera 30 with respect to the storage chamber 21 in addition to the function described in the first embodiment. In this example, it can be considered that the detection unit 131 has a function corresponding to the “speed detection unit” in the claims. In the following description, the relative speed of the camera with respect to the storage chamber 21 may be referred to as “the moving speed of the camera 30” or “the moving speed”. In this example, the detection unit 131 obtains a moving speed corresponding to the position of the camera 30 at the time of imaging. For example, the detection unit 131 uses the position of the camera 30 detected in the same manner as in the first embodiment described above (a plurality of camera positions corresponding to a plurality of imaging times one to one) to detect the camera 30 at the time of imaging. The moving speed corresponding to the position can also be obtained. For example, a gyroscope, an accelerometer, or the like is mounted on the camera 30 and the moving speed can be obtained using signals output from these instruments, or can be obtained by other methods. Absent.

  For example, the detection unit 131 generates detection information corresponding to the opening operation when the opening door 10 is opened, and generates detection information corresponding to the closing operation when the closing door 10 is closed. can do. The detection information in this example is configured by information in which a captured image and a moving speed are associated with each other for each of a plurality of detected positions (positions of the camera 30 at the time of imaging).

  In the present embodiment, the first generation unit 130 generates a first composite image when the opening operation or the closing operation of the hinged door 10 is performed, and uses the detection information used for generating the first composite image as the first information. It is supplied to the setting unit 170 together with one composite image. When the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130 (each time it is acquired), the setting unit 170 performs the first composite image based on the acquired first composite image and the detection information. For each of a plurality of areas, a first reliability indicating a lower value is set as the moving speed when a captured image corresponding to the area is obtained is set. This will be specifically described below.

In the following description, the first reliability is a value in the range of 0 or more and 1 or less, and “1” represents the highest reliability and “0” represents the lowest reliability. Further, the moving speed of the camera 30 at the position p of the camera 30 at the time of imaging is denoted as Vp. Here, the captured image when the camera 30 is stationary (Vp = 0) is an image with no blur if the object (subject) is in focus. Bokeh is an aspect of image degradation that occurs in the process of obtaining an image by imaging, and refers to an event that occurs when a group of rays to be focused (condensed) at one point spreads on the image plane. On the other hand, as the moving speed Vp of the camera 30 increases, motion blur is seen in the captured image, so that the image quality decreases and the image becomes less reliable. That is, the first reliability decreases as the moving speed Vp of the camera 30 increases (see FIG. 13). The first reliability can be expressed by the following formula 1, for example.

In the above formula 1, Sp represents the first reliability, and Vth represents the speed serving as a threshold value. This is equivalent to determining that the captured image at that time is a highly reliable image if the moving speed Vp is equal to or less than a certain speed Vth. Vth may be set according to the frame rate, shutter speed, or the like of the camera 30, and Vth can be set to a large value when the frame rate is high or when the shutter speed is fast. The calculation method of the first reliability Sp is not limited to the method described above, and the first reliability Sp can be designed to decrease as the moving speed Vp increases. The first reliability Sp can also be expressed by the following formula 2, for example.

  In the above, the blur of the image is indirectly evaluated from the moving speed of the camera 30. However, the present invention is not limited to this. For example, the blur amount in the image is directly measured, and the reliability indicating a lower value is calculated as the blur amount is larger. It may be a form to do. In other words, this reliability can be considered to be synonymous with the first reliability indicating a lower value as the moving speed is larger (the blur amount is larger).

  In the present embodiment, when the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130, the setting unit 170 refers to the acquired detection information for each of the plurality of captured images included in the detection information. In addition, a first reliability Sp corresponding to the moving speed Vp associated with the captured image is calculated and assigned to the captured image. Thereby, the first reliability Sp corresponding to the moving speed Vp associated with the captured image is assigned to each of the plurality of captured images that are the basis of the first composite image acquired from the first generation unit 130. And the setting part 170 acquired from the 1st production | generation part 130 with the detection information used for the production | generation of the said 1st synthetic | combination image among the 1st synthetic | combination images acquired from the 1st production | generation part 130 (the said 1st synthetic image). The plurality of areas corresponding to the plurality of captured images included in the detection information) are identified, and the first reliability Sp assigned to the captured image corresponding to the area is set for each of the identified areas. To do.

  As described above, for each of a plurality of regions in the first composite image acquired from the first generation unit 130 (a plurality of regions corresponding to a plurality of captured images that are the basis of the first composite image). The first reliability Sp indicating a lower value is set as the moving speed of the camera 30 when the captured image corresponding to the region is obtained is higher.

  Of the plurality of regions corresponding to the plurality of captured images that are the basis of the first composite image in the first composite image acquired from the first generation unit 130, with respect to portions that overlap each other, for example, It is also possible to set the first reliability Sp1 of the part by multiplying the first reliability Sp1 of each of a plurality of overlapping regions (for example, a product obtained by weighting each first reliability Sp1). For example, by calculating an average value of the first reliability Sp1 of each of a plurality of regions overlapping each other, the first reliability Sp1 of the part can be set.

  For example, the concept of the distance from the camera 30 to the imaging target can be added, and the first reliability Sp1 can be assigned to each pixel in the captured image. The distance to the imaging target may be directly measured using a stereo camera or a distance meter, or may be estimated from the captured image by a method such as motion stereo. In addition, when the distance to the imaging target is not directly used, there is a method of using the distance from the camera 30 to the entrance of the storage chamber 21.

Here, let La be the distance (distance to the entrance of the storage chamber 21) in the real space of an arbitrary pixel a in the captured image Ip corresponding to the position p of the camera 30 at the time of imaging. As La is larger, the movement of the subject in the captured image becomes smaller and blurring is less likely to occur. Therefore, the region is designed so that the reliability is higher in a region where La is larger. On the other hand, blurring is more likely to occur as La is smaller. Therefore, it is preferable to design such that reliability is lower as La is smaller. If the reliability using the distance corresponding to the pixel a is SpL (a), this reliability SpL (a) can be expressed by, for example, the following Expression 3.

In the above formula 3, Lth is an arbitrary positive constant, and the distance beyond this distance is less likely to cause blur. The reliability SpL (a) may be calculated by a method other than the above. From the reliability SpL (a) and the reliability calculated from the moving speed corresponding to the captured image including the pixel a (for example, the reliability calculated by the above formula 1 or 2) SpV, the pixel a The corresponding first reliability Sp1 (a) can also be calculated. The first reliability Sp1 (a) can be expressed by, for example, the following formula 4.

  In addition to the above formula 4, for example, a method such as obtaining the sum of SpL (a) and SpV and normalizing can be considered, and other methods may be used. Then, using the first reliability Sp1 (a) assigned to each pixel included in each of the plurality of captured images that are the basis of the first composite image, the first confidence for each pixel in the first composite image. The degree Sp1 can also be obtained.

  When setting the first reliability for each of the plurality of regions in the first composite image acquired from the first generation unit 130 (every time), the setting unit 170 sets the first reliability. The composite image is stored in the storage unit 140, and the first composite image in which the first reliability is set and the detection information used for generating the first composite image are supplied to the second generation unit 150. The storage unit 140 stores the first composite image in which the first reliability is set in time series.

  Similar to the first embodiment described above, the second generation unit 150 uses the past first composite image when the first composite image acquired from the setting unit 170 does not include the entire entrance of the storage chamber 21. A second composite image is generated by complementing the above-described insufficient region. Furthermore, the second generation unit 150 of the present embodiment has the first reliability set for each of the first composite image (latest first composite image) acquired from the setting unit 170 and the past first composite image. Based on the above, a second composite image is generated. More specifically, the second generation unit 150 has a first reliability higher than a first reliability of a corresponding region in the past first composite image among a plurality of regions in the latest first composite image. A region showing a lower value is replaced with a corresponding region in the past first composite image. Thereby, the advantageous effect that the image quality of the second composite image can be further improved can be achieved.

(Third embodiment)
Next, a third embodiment will be described. Description of parts common to the above-described embodiments will be omitted as appropriate. The basic configuration of the image processing apparatus according to the third embodiment is the same as that of the image processing apparatus 200 according to the second embodiment (similar to the configuration of FIG. 12), and thus detailed illustration is omitted.

  In the present embodiment, the setting unit 170 increases the value for each of a plurality of regions in the first composite image as the time (image capturing time) when the captured image corresponding to the region is obtained is closer to the current time. The second reliability shown is further set, and the second generation unit 150 sets the first reliability based on the first reliability and the second reliability set for each of the first composite image and the past first composite image. The second embodiment is different from the second embodiment in that two composite images are generated. The specific contents of the third embodiment will be described below.

  As in the above-described embodiments, when the opening operation of the hinged door 10 is performed, the detection unit 131 generates detection information corresponding to the opening operation, and when the opening operation of the hinged door 10 is performed, the detection unit 131 performs the closing operation. Detection information corresponding to the operation is generated. The detection information in this example is configured by information in which a captured image, a moving speed, and an imaging time are associated with each other for each of a plurality of detected positions (positions of the camera 30 at the time of imaging).

  Similarly to the second embodiment described above, the first generation unit 130 generates a first composite image when the opening operation or the closing operation of the hinged door 10 is performed, and is used to generate the first composite image. The detected information is supplied to the setting unit 170 together with the first composite image.

  When the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130, the setting unit 170, for each of a plurality of regions in the first composite image, based on the acquired first composite image and the detection information. In addition, the first reliability indicating a lower value is set as the moving speed when the captured image corresponding to the region is obtained is higher, and the imaging time when the captured image corresponding to the region is obtained is currently The second reliability indicating a higher value as the time is closer is set, and the third reliability defined according to the set first reliability and second reliability is set. The first reliability setting method is the same as that in the second embodiment.

  The second reliability is a value within the range of 0 or more and 1 or less, and “1” represents that the reliability is the highest (represents that the imaging time is closest to the current time), “ “0” represents the lowest reliability. In this example, when the difference between the imaging time and the current time is greater than or equal to the threshold, the second reliability is set to “0”.

  In the present embodiment, when the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130, the setting unit 170 refers to the acquired detection information for each of the plurality of captured images included in the detection information. In addition, a second reliability indicating a lower value as the difference between the imaging time associated with the captured image and the current time is larger is calculated and assigned to the captured image. Thereby, the second reliability corresponding to the imaging time associated with the captured image is assigned to each of the plurality of captured images that are the basis of the first composite image acquired from the first generation unit 130. Then, the setting unit 170 has a plurality of one-to-one correspondences with the plurality of captured images included in the detection information used for generating the first composite image among the first composite images acquired from the first generation unit 130. The second reliability assigned to the captured image corresponding to the area is set for each of the specified areas. As described above, for each of a plurality of regions in the first composite image acquired from the first generation unit 130 (a plurality of regions corresponding to a plurality of captured images that are the basis of the first composite image). The second reliability indicating a lower value is set as the imaging time when the captured image corresponding to the region is obtained is closer to the current time.

  Then, the setting unit 170 performs a plurality of regions in the first composite image acquired from the first generation unit 130 (a plurality of regions corresponding to a plurality of captured images that are the basis of the first composite image). In addition, a third reliability defined according to the first reliability and the second reliability set in the area is calculated, and the calculated third reliability is set in the area. In the present embodiment, the setting unit 170 obtains the third reliability by multiplying the first reliability and the second reliability. However, the setting unit 170 is not limited to this, and for example, each of the first reliability and the second reliability is determined. The third reliability may be obtained by weighted multiplication, for example, the third reliability may be obtained by calculating a weighted average value of the first reliability and the second reliability, The third reliability may be obtained by normalizing the sum of the first reliability and the second reliability to a value in the range of 0 to 1.

  When setting the third reliability for each of the plurality of regions in the first composite image acquired from the first generation unit 130, the setting unit 170 sets the first composite image with the third reliability, The detection information used for generating the first composite image is stored in the storage unit 140 in association with the detection information, and is supplied to the second generation unit 150.

  In this example, the difference between the imaging time and the current time included in the detection information associated with the past first composite image stored in the storage unit 140 increases with the passage of time. Every time the setting unit 170 stores the first composite image with the third reliability set in the storage unit 140, the difference between the imaging time included in the detection information associated with the past first composite image and the current time. Accordingly, it is possible to update the third reliability set for each region in the past first composite image. If the difference between the imaging time and the current time is greater than or equal to the threshold, the third reliability can be set to “0”.

  Similar to the first embodiment described above, the second generation unit 150 uses the past first composite image when the first composite image acquired from the setting unit 170 does not include the entire entrance of the storage chamber 21. A second composite image is generated by complementing the above-described insufficient region. Furthermore, the second generation unit 150 of the present embodiment has a third reliability set for each of the first composite image (latest first composite image) acquired from the setting unit 170 and the past first composite image. Based on the above, a second composite image is generated. More specifically, the second generation unit 150 has a third reliability higher than a third reliability of a corresponding region in the past first composite image among the plurality of regions in the latest first composite image. A region showing a lower value is replaced with a corresponding region in the past first composite image.

  In other words, in the present embodiment, in addition to complementing the shortage area, the second composite image is generated by comprehensively considering the deterioration of the image quality and the imaging time, thereby improving the image quality of the second composite image. The advantageous effect of being able to be achieved can be achieved.

  As mentioned above, although embodiment of this invention was described, each above-mentioned embodiment was shown as an example and is not intending limiting the range of invention. The above novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. The above novel embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

(Modification)
Hereinafter, modified examples will be described.

(1) Modification 1
In each of the embodiments described above, the first generation unit 130 generates the first composite image when the opening operation or the closing operation of the hinged door 10 is performed. May be a form in which the first composite image is generated when the opening operation is performed and the first composite image is not generated even when the closing operation is performed. Further, for example, the first generation unit 130 may generate the first composite image when the closing operation is performed, and may not generate the first composite image even when the opening operation is performed.

  Further, for example, when both the opening operation and the closing operation are performed, the first generation unit 130 (a series of operations from the start of the opening operation to the end of the closing operation (hereinafter referred to as “opening / closing operation”)). The first composite image may be generated every time the image is performed. In short, the 1st production | generation part 130 should just be a form which produces | generates a 1st synthesized image, when at least one of the opening operation | movement of the hinged door 10 and a closing operation | movement is performed.

  Here, for example, it is assumed that the first composite image is generated when the above-described opening / closing operation is performed. In this embodiment, for example, the position of the camera 30 at the time of imaging in the opening operation (referred to as “first position” for convenience of description) and the position of the camera 30 at the time of imaging in the closing operation (for convenience of description “second” (Referred to as “overlapping area” for convenience of explanation) is imaged at each of the first position and the second position. In such a case, it is necessary to select an image used for generating the first composite image in units of frames or regions. As the selection method, an area where the overlapping area is imaged in the captured image corresponding to the first position and an area where the overlapping area is imaged in the captured image corresponding to the second position are selected. It is preferable to compare and select the one with higher reliability. For example, as in the third embodiment described above, the moving speed (blur amount) at the time of imaging and the reliability (third reliability) according to the imaging time are set to the captured image corresponding to the first position and the second It is also possible to select a region with higher reliability by setting each captured image corresponding to the position.

  In this example, even in the same imaging range, the value of the information is higher in a captured image with a later imaging time. Specifically, a general method of using the refrigerator 1 is a flow of (1) opening the hinged door 10, (2) carrying in / out the stored items, and (3) closing the hinged door 10, Since the state of the storage chamber 21 changes before and after 2), the state at the time of (3) is the information that the user really wants to know. In other words, it is preferable that the captured image with the later imaging time has a higher value and is designed to have high reliability even when there is image quality degradation such as a slight blurring of the image. The same imaging range can be estimated by calculating the imaging range from the position of the camera 30, a method of calculating a similar region by comparing the captured images, a method of using a marker, or the like. But it can be estimated.

(2) Modification 2
In each of the embodiments described above, the camera 30 is provided on the hinged door 10, but is not limited thereto, and for example, the camera 30 may be provided on the main body 20. In this embodiment, the camera 30 changes the positional relationship with the door pocket 11 (see FIG. 1) according to the opening or closing operation of the hinged door 10, and continuously images the door pocket 11. That is, in this embodiment, it can be considered that the door pocket 11 corresponds to the “storage chamber” in the claims. Also in this form, the processing relating to the generation of each of the first composite image and the second composite image can be considered in the same manner as in the above-described embodiments. This is because the movement of the camera 30 to image the storage room 21 and the imaging of the hinged door 10 to which the camera 30 moves stationary are the same from the viewpoint of relative movement. If the camera 30 (stationary camera 30) attached to the main body 20 is observed from above the moving hinge 10, the camera 30 is observed as moving, and from the observer on the hinge 10, It is impossible to distinguish which is moving. By the way, the position of the camera 30 (the position of the camera 30 at the time of imaging) included in the detection information in this case is a position on the coordinate system that changes as the hinged door 10 moves. By doing so, it is possible to consider in the same manner as the position of the camera 30 in each of the above-described embodiments. The method for estimating the position of the camera 30 is basically the same as that in each of the embodiments described above. However, it is impossible to attach a gyroscope or GPS instrument to the camera 30, but it is possible to estimate the position of the camera 30 by attaching a gyroscope or GPS instrument to the hinged door 10.

  In short, the camera 30 only needs to have a configuration in which the positional relationship with the storage chamber (the storage chamber 21 or the door pocket 11) changes according to the opening or closing operation of the hinged door 10 and a plurality of storage chambers are imaged. In addition, the first generation unit 130 is configured so that the same subject included in the plurality of captured images obtained by imaging with the camera 30 overlaps when at least one of the opening operation and the closing operation of the hinged door is performed. Any form may be used as long as the first composite image is generated by combining the captured images. In addition, when the first composite image does not include a part of the target area indicating the area to be presented to the user in the storage room, the second generation unit 150 uses the first composite image in the past to generate the target area. Any form may be used as long as the second composite image is generated to complement a part of the image and present it to the user.

(3) Modification 3
In each of the above-described embodiments, the refrigerator 1 has been described as an example of the “storage device” in the claims, but is not limited thereto. For example, it may be a cupboard having a hinged door, a cabinet having a hinged door (a shelf for organizing documents), a refrigeration machine having a hinged door, and the like.

  In addition, each above embodiment and each modification can be combined arbitrarily.

DESCRIPTION OF SYMBOLS 1 Refrigerator 10 Folding door 11 Door pocket 20 Main body part 21 Storage room 30 Camera 40 Sensor 50 Display part 100 Image processing apparatus 110 Camera control part 120 Acquisition part 130 1st generation part 140 Storage part 150 2nd generation part 160 Display control part 170 Setting Part

Claims (5)

With hinged doors,
A storage room for storing objects;
An image capturing unit that is disposed to be inclined in the normal direction of the door surface of the hinged door and that images the storage room;
The imaging unit
Imaging starts when the angle between the front surface of the storage chamber and the hinged door changes in the increasing direction, stops imaging when the amount of change in the increasing direction of the angle falls below a threshold, and then changes in the decreasing direction of the angle When the amount exceeds the threshold value, the imaging is started again, and when the amount of change in the decreasing direction of the angle is equal to or less than the threshold value, the imaging is stopped.
Storage device. The imaging axis of the imaging unit is inclined toward the rotation axis of the hinged door,
The storage device according to claim 1. The imaging unit captures an image of a portion of the storage room that is away from the rotation axis of the hinged door as the hinged door is opened.
The storage device according to claim 1 or 2. The imaging unit is installed so as to image the entire storage chamber from the start to the end of the opening operation of the hinged door,
The storage device according to any one of claims 1 to 3. A generation unit that combines a plurality of captured images obtained by imaging by the imaging unit;
The storage device according to any one of claims 1 to 4.

Images