WO2022038799A1 - 冷蔵庫内の画像データ処理方法および冷蔵庫 - Google Patents

冷蔵庫内の画像データ処理方法および冷蔵庫 Download PDF

Info

Publication number
WO2022038799A1
WO2022038799A1 PCT/JP2020/044607 JP2020044607W WO2022038799A1 WO 2022038799 A1 WO2022038799 A1 WO 2022038799A1 JP 2020044607 W JP2020044607 W JP 2020044607W WO 2022038799 A1 WO2022038799 A1 WO 2022038799A1
Authority
WO
WIPO (PCT)
Prior art keywords
refrigerator
image data
storage
image
door
Prior art date
Application number
PCT/JP2020/044607
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
幸太郎 高橋
真司 上野
将 矢野
貴志 藤元
康之 上甲
Original Assignee
日立グローバルライフソリューションズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立グローバルライフソリューションズ株式会社 filed Critical 日立グローバルライフソリューションズ株式会社
Priority to CN202080018909.3A priority Critical patent/CN114514405B/zh
Priority to CN202310993504.7A priority patent/CN116857878A/zh
Publication of WO2022038799A1 publication Critical patent/WO2022038799A1/ja

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D13/00—Stationary devices, e.g. cold-rooms
    • F25D13/02—Stationary devices, e.g. cold-rooms with several cooling compartments, e.g. refrigerated locker systems
    • F25D13/04—Stationary devices, e.g. cold-rooms with several cooling compartments, e.g. refrigerated locker systems the compartments being at different temperatures
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D23/00—General constructional features
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D23/00—General constructional features
    • F25D23/02—Doors; Covers
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D23/00—General constructional features
    • F25D23/12—Arrangements of compartments additional to cooling compartments; Combinations of refrigerators with other equipment, e.g. stove
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D27/00—Lighting arrangements
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2500/00—Problems to be solved
    • F25D2500/06—Stock management
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2700/00—Means for sensing or measuring; Sensors therefor
    • F25D2700/02—Sensors detecting door opening

Definitions

  • the present invention relates to an image data processing method in a refrigerator and a refrigerator.
  • a refrigerator has been proposed in which the inside of the refrigerator is photographed with a camera for the purpose of inventory management in the refrigerator.
  • a plurality of storage rooms divided into at least one photographing range, a plurality of internal cameras for photographing at least one of the photographing ranges, or a plurality of photographing ranges are photographed. It is equipped with at least one camera in the refrigerator and a control device that stores images in a plurality of shooting ranges in a storage device and communicates with an operation display device that displays the images. " Then, in the operation display device, an operation of instructing the movement of the image in the refrigerator compartment is performed by "touching the finger and moving it left or right (so-called swipe operation or flick operation performed on the touch panel of the smartphone or tablet terminal).
  • the control device displays the image of the door pocket side of the door located in the opposite direction to the refrigerating room in the observation state where the user observes the refrigerating room from the front side of the refrigerator in combination with the image of the refrigerating room.
  • Patent Document 1 requires the user to perform a swipe operation or a flick operation in the operation display device in order to simultaneously view the refrigerator compartment and the door pocket of the door.
  • the image data processing method in the refrigerator of the present invention made in consideration of the above circumstances is a refrigerator that processes image data captured by one or more image pickup devices including a plurality of food storage areas in the imaging range in total.
  • image data processing methods there is a cutting process (for example, S392) in which a plurality of parts including a food storage area are cut out from one or two or more image data captured by an imaging device.
  • a compositing process for example, S396) for creating one image data using the part of the above, and an output process (for example, S40) for outputting one image data synthesized by the compositing process are included in the execution. It is a feature.
  • Other aspects of the invention will be described in embodiments described below.
  • FIG. 1 is a front view showing the refrigerator 100 according to the first embodiment.
  • FIG. 2A is a side view showing the refrigerator 100 according to the first embodiment.
  • FIG. 2B is a side view when the image pickup apparatus 30 according to the first embodiment is housed.
  • FIG. 3 is a front view showing a state in which the door of the refrigerator 100 according to the first embodiment is opened.
  • FIG. 4 is a top view showing a state in which the door of the refrigerator 100 according to the first embodiment is opened.
  • the 6-door refrigerator 100 will be described as an example, but the description is not limited to the 6-door refrigerator 100.
  • the heat insulating box body (box body 10) of the refrigerator 100 is in the order of the refrigerating room 1, the ice making room 2 and the upper freezing room 3, the vegetable room 4, and the lower freezing room 5 arranged on the left and right from the top.
  • the refrigerator 100 includes a door that opens and closes the opening of each storage room. These doors open and close the openings of the refrigerating room 1, the left and right rotary refrigerating room doors 1a and 1b, and the openings of the ice making room 2, the upper freezing room 3, the vegetable room 4, and the lower freezing room 5.
  • the positions of the vegetable compartment 4 and the lower freezing chamber 5 may be changed vertically.
  • the ice making chamber 2, the upper freezing chamber 3 and the lower freezing chamber 5 are collectively referred to as a freezing chamber 7.
  • Door hinges (reference numerals omitted) are provided at the upper and lower parts of the refrigerator compartment 1 in order to fix the refrigerator compartment doors 1a and 1b to the refrigerator 100. As shown in FIGS. 3 and 4, when the refrigerating chamber doors 1a and 1b are opened and closed, door storage portions 11a (door storage portions a) and 11b (door storage portions b) are provided.
  • the refrigerating room 1 is a refrigerating storage room in which the inside of the refrigerator is set to a refrigerating temperature range (0 ° C. or higher), for example, about 4 ° C. on average.
  • the freezing chamber 7 is a freezing storage chamber in which the inside of the refrigerator is set to a freezing temperature range (less than 0 ° C.), for example, about ⁇ 18 ° C. on average.
  • the vegetable compartment 4 is a refrigerated storage chamber having a refrigerated temperature range of, for example, about 6 ° C. on average, and is a refrigerated storage chamber that suppresses drying of food.
  • an image pickup device 30 for photographing the inside of the refrigerator is arranged in a substantially central portion of the outer upper part of the storage chamber. That is, it is provided above the box body (insulated box body) of the refrigerator 100, substantially in the center.
  • the lens 34 of the image pickup apparatus 30 (see FIGS. 65A and 65B) is in front of the opening edge surface 10a of the storage chamber, preferably from the refrigerating chamber doors 1a and 1b (doors). Is also located in front. That is, the image pickup device 30 is provided outside the storage room, and the situation inside the refrigerator including the door storage portions 11a and 11b can be imaged from a bird's-eye view.
  • the image pickup device 30 has a mechanism that slides back and forth to move (a mechanism that slides in a direction perpendicular to the opening of the storage chamber), and can move back and forth across the front surface of the door of the refrigerator 100.
  • a mechanism that slides back and forth to move a mechanism that slides in a direction perpendicular to the opening of the storage chamber
  • the depth dimension is not affected at the time of packing.
  • the number of refrigerators loaded on the transport truck does not decrease, and even after installation, the image pickup device 30 can be stored and the image pickup range can be blindfolded when the image pickup device 30 is not used.
  • the blindfold can be configured in consideration of the user who does not like the fact that the image pickup area of the camera is always within his / her own living range.
  • the refrigerating room door 1b is provided with a shooting button 39 (a shooting command receiving unit that receives a shooting command from the user) that is used when manually shooting with the image pickup device 30.
  • the photographing button 39 may be provided in the refrigerating room door 1a, the box body, or another area of the door.
  • FIG. 5A is a front view showing the image pickup apparatus 30 according to the first embodiment.
  • FIG. 5B is a side view showing the image pickup apparatus 30 according to the first embodiment.
  • FIG. 5C is a side sectional view (AA sectional view of FIG. 5A) showing the image pickup apparatus 30 according to the first embodiment.
  • FIG. 6 is a perspective view showing the image pickup apparatus 30 according to the first embodiment.
  • the image pickup apparatus 30 is composed of a case portion 31 and a lid portion 32, and has a camera 33 (imaging section) and a lens 34 which is a fisheye lens in front of the case portion 31. As shown in FIG. 6, the lens 34 is surrounded by a light-shielding wall 35. A plurality of lighting units 38 are provided in the vicinity of the light-shielding wall 35.
  • control SoC 41 and the wireless LAN unit 42 are stored inside the case unit 31.
  • the control SoC41 System on a chip integrates the functions of a general microcontroller such as a processor core on one chip of an integrated circuit, as well as the functions for application purposes, and cooperates with the system. It is an integrated circuit designed to function as an integrated circuit.
  • the control SoC41 is an example of an integrated circuit, and can be replaced by another integrated circuit or the like.
  • the image pickup device 30 provided outside the storage chamber images the door storage portions 11a and 11b and the inside of the storage chamber from a bird's-eye view.
  • the features of the image pickup apparatus 30 will be further described.
  • the camera 33 used in the image pickup apparatus 30 can shoot a wide angle of 180 degrees or more, and by having a fisheye lens, the entire interior of the refrigerator can be visually recognized when the door is opened.
  • the lens 34 for the camera of the image pickup apparatus 30 is located in front of the front of the refrigerator door, the state inside the refrigerator is photographed even when only the drawer portion 6 of the vegetable compartment 4 and the lower freezing chamber 5 is opened. can do.
  • the image pickup apparatus 30 is provided with a lighting unit 38, and a highly visible image can be obtained by taking a picture using the lighting unit 38 outside the refrigerator, which can be adjusted to an illuminance suitable for shooting. Since the lighting provided in the refrigerator has a relatively high illuminance, the stored items are strongly reflected when taken with a camera, and the taken image tends to be overexposed. Therefore, in the present embodiment, at the time of shooting, the lighting inside the refrigerator is turned off and the lighting unit 38 outside the refrigerator is turned on for shooting. Further, the lighting of the lighting unit 38 is directional so as to illuminate the entire interior, the door storage units 11a and 11b when the door is opened, and the second storage room (for example, the vegetable room 4 and the lower freezing room 5). Have.
  • the image pickup device 30 has the light-shielding wall 35, the direct light of the illumination unit 38 does not enter the shooting range, so that it is possible to obtain a highly visible image with suppressed reflection and overexposure.
  • the installation position of the illumination can be made as close as possible to the lens 34, so that the outer shape can be kept compact. Further, it is possible to obtain the effect of preventing scratches on the lens 34.
  • the image pickup device 30 includes a communication unit (wireless LAN unit 42), and image data transmitted from the communication unit is, for example, via a home wireless router (for example, a home router 53 (see FIG. 7)). Is supplied to the server 54 (see FIG. 7). Details will be described later.
  • a communication unit wireless LAN unit 42
  • image data transmitted from the communication unit is, for example, via a home wireless router (for example, a home router 53 (see FIG. 7)). Is supplied to the server 54 (see FIG. 7). Details will be described later.
  • a communication circuit for transmitting power supply and a shooting command is connected to the refrigerator main body side.
  • the refrigerator itself is powered by a commercial power source or the like.
  • the power supply of the image pickup apparatus 30 can be received from a commercial power source via the refrigerator body.
  • the image pickup device 30 may be driven by a rechargeable battery.
  • FIG. 7 is a configuration diagram showing an imaging system of the refrigerator 100 according to the first embodiment.
  • door sensors (not shown) that detect the open / closed state of the refrigerating room doors 1a and 1b, the ice making room door 2a, the upper freezing room door 3a, the vegetable room door 4a, and the lower freezing room door 5a, respectively (not shown). Etc. are also provided.
  • a main microcomputer 52 (processing device) equipped with a memory such as a CPU (Central Processing Unit), a ROM (Read Only Memory) and a RAM (Random Access Memory), which are a part of a control device, an interface circuit, etc. , Control device, control unit) are arranged.
  • the main microcomputer 52 controls an imaging system that captures images in the refrigerator based on information from the door sensor or the photographing button 39.
  • the main microcomputer 52 also supplies power to the image pickup apparatus 30.
  • the main microcomputer 52 When the main microcomputer 52 detects a predetermined angle or more based on information such as the door opening angle from the door sensor when automatic shooting is set, the main microcomputer 52 transmits a shooting command to the control SoC41.
  • the control SoC 41 issues a shooting instruction to the camera 33 and acquires unprocessed image data from the camera 33.
  • the control SoC41 performs image processing on the unprocessed image data, for example, as shown in FIG.
  • the main microcomputer 52 detects that the shooting button 39 has been operated during the manual shooting setting, the main microcomputer 52 transmits a shooting command to the control SoC 41. Subsequent processing is the same as when automatic shooting is set.
  • the image pickup device 30 is provided with a wireless LAN unit 42 that can be connected to the home router 53.
  • the image data in the refrigerator processed by the control SoC 41 can be transmitted to the server 54 via the home router 53.
  • the server 54 transmits the image data in the refrigerator in response to the image display request in the refrigerator from a mobile device such as a smartphone 55 owned by the user, a personal computer, or the like. As a result, the user can grasp the situation in the refrigerator.
  • FIG. 8 is a flowchart showing a process at the time of manual shooting setting according to the first embodiment. It will be described with reference to FIG. 7 as appropriate.
  • the main microcomputer 52 determines whether or not the door is open (door open?) By the door sensor. (S32) If there is a door opening (S32, Yes), the process proceeds to S33. On the other hand, when the door is not opened (S32, No), the main microcomputer 52 returns to S32.
  • the main microcomputer 52 determines whether or not the shooting button 39 is operated (pressed), and if the shooting button 39 is pressed (S33, Yes), the process proceeds to S34 and the shooting button 39 is not pressed. In the case (S33, No), the process returns to S32.
  • the main microcomputer 52 issues a shooting command to the control SoC41 to turn off the lighting inside the refrigerator and turn on the lighting outside the refrigerator (lighting unit 38) (S35).
  • the control SoC 41 commands the camera 33 to take a picture (S36).
  • the main microcomputer 52 turns on the lighting inside the refrigerator and turns off the lighting outside the refrigerator (illumination unit 38) (S37).
  • the camera 33 transmits unprocessed image data to the control SoC41 (S38), and the control SoC41 performs in-house image correction processing (image processing) (S39). Details of the image processing will be described later with reference to FIG.
  • the control SoC 41 performs image output processing (image update) on the server 54 via the wireless LAN unit 42 and the home router 53 (S40).
  • image update image update
  • the server 54 notifies the user's mobile device or the like (S41). The above is the process when manual shooting is set.
  • FIG. 9 is a flowchart showing a process at the time of automatic shooting setting according to the first embodiment. It will be described in a timely manner with reference to FIG. 7. The same process as in FIG. 8 will be omitted.
  • the setting from the setting device (not shown) is the camera automatic shooting setting (camera automatic setting ON) (S31A)
  • the main microcomputer 52 determines whether or not the door is opened by the door sensor (door open?). If the determination is made (S32) and the door is open (S32, Yes), the process proceeds to S33A. On the other hand, when the door is not opened (S32, No), the main microcomputer 52 returns to S32.
  • the main microcomputer 52 determines whether or not the door opening angle is equal to or greater than a predetermined angle (for example, the door angle is 85 degrees or more), and the door opening angle is equal to or greater than the predetermined angle (S33A, Yes). , S34, and when the door opening angle is less than a predetermined angle (S33A, No), the process returns to S32.
  • S34 to S41 are the same as in FIG.
  • S33A in the case of the refrigerating room 1 having double doors, if the opening angle of at least one of the refrigerating room doors 1a and 1b is a predetermined angle (S33A, Yes), the process proceeds to S34.
  • a predetermined angle for example, the door angle is 85 degrees or more
  • 3A It has a detection unit (for example, installed in the drawer unit 6) that detects when the drawer distance of the refrigerator 100 exceeds a certain level, and the detection unit detects the operation of the drawer and automatically takes a picture.
  • the imaging range of the vegetable room or the freezing room can be kept substantially constant.
  • a shooting button (for example, a shooting button 39) is provided on the door of the refrigerator, and when the user presses the shooting button, the lighting inside the refrigerator is turned off and the lighting of the image pickup apparatus 30 is turned on before shooting. Therefore, it is easy to recognize the timing of shooting.
  • the captured image is subjected to image correction processing, and the distorted image is corrected by the wide-angle camera, so that an image close to the user's line of sight can be provided.
  • 6A It has an application service that allows the user to confirm the image via a mobile phone or the like, and the image is a storage room in which the image taken by the detection unit provided in the door and the drawer is. It is possible to automatically determine whether the image is an image of.
  • the refrigerator 100 of the present embodiment has a storage chamber with an open front (for example, a refrigerating chamber 1) and a rotatable storage portion (for example, a door storage portion 11a) provided on the front surface of the storage chamber.
  • a door having 11b) for example, refrigerating room doors 1a and 1b
  • a door opening angle detecting means for detecting the opening angle of the door
  • lighting for detecting that the door has been opened by the door opening detecting means to illuminate the storage room.
  • the device an image pickup device (for example, an image pickup device 30) provided in the upper center of the outside of the storage chamber, a fisheye lens with a wide angle field (for example, a lens 34), and a lighting device (for example, an illumination unit) that lights up at the time of shooting. 38) and a communication means (for example, a wireless LAN unit 42) for transmitting captured image data.
  • the refrigerator 100 has a control device (for example, a main microcomputer 52) that issues a shooting command to an image pickup device when a predetermined opening angle is detected by the door opening angle detecting means.
  • the image taken by the image pickup device with the door open is a single shot image due to the action of the fisheye lens in the wide-angle field of view, and not only the storage room but also the entire door storage part is taken in the shooting range.
  • the vegetable room (for example, vegetable room 4) and the freezing room (for example, the lower freezing room 5) of the drawer storage section can also be included in this shooting range, and each storage room can be set as in Patent Document 1. It is not necessary to provide a plurality of image pickup devices for each, and there is no need for a complicated and expensive drive device or drive control for changing the shooting range.
  • the image taken by the image pickup device is sufficient to grasp the state of the stored items in the refrigerator as it is, but the image is curved due to the action of the fisheye lens, and the object close to the image pickup device is more distorted. .. Further, since the visual field image is from above the refrigerator, there is room for improvement in terms of visibility, unlike the positional relationship in which the image is arranged in the visual field from the front of the refrigerator, which is usually observed by the user.
  • a distorted curved image taken by an image pickup device 30 having a fisheye lens having a wide-angle field of view can be viewed from the front where a user usually observes the refrigerator. It is possible to provide an image in the refrigerator in a state close to the positional relationship of the field of view of.
  • FIG. 10 is a flowchart showing the image processing (S39) according to the first embodiment. Images of each process will be described with reference to FIGS. 11 to 16.
  • FIG. 11 is an image showing the inside of the refrigerator before image processing (the storage c as the refrigerator compartment 1, the door storage portion a as the door storage portion 11a, and the door storage portion b as the door storage portion 11b).
  • FIG. 12 is an image showing the cutting process of each storage portion.
  • FIG. 13 is an image showing the distortion correction processing of each storage portion.
  • FIG. 14 is an image showing the rotation processing of each storage unit.
  • FIG. 15 is an image showing the keystone correction processing of each image.
  • FIG. 16 is an image showing a composition process of each image.
  • the main microcomputer 52 inputs the above-mentioned photographed image (see FIG. 11) (S391), and performs a fan-shaped cutting process (cutting process) so as to cover the range of each storage unit (S392). Specifically, as shown in FIG. 12, the main microcomputer 52 cuts out the internal image of the refrigerator compartment 1 and the images of the door storage portions 11a and 11b, for example, in a fan shape (see FIG. 12). ..
  • the main microcomputer 52 performs distortion correction processing (distortion correction processing) of the image of each storage portion curved by the action of the fisheye lens (S393). Specifically, as shown in FIG. 13, the main microcomputer 52 performs distortion correction processing for each of the internal image of the refrigerating chamber 1 and the curved images of the door storage portions 11a and 11b.
  • Various known correction algorithms can be used for this distortion correction process, and for example, the distortion can be corrected by executing a calibration process in which parameters are set in advance (for example, before shipping the refrigerator).
  • the main microcomputer 52 performs an image rotation process of rotating the direction of each image of each storage unit that has been subjected to the distortion correction process so as to have a field of view from the front, which is the observation state of the user (S394).
  • the main microcomputer 52 has the door storage portion 11a on the left side and the door storage portion 11b on the right side so as to have a field of view from the front, which is the observation state of the user. It is arranged.
  • the main microcomputer 52 performs distortion correction processing (trapezoidal correction processing) on a rectangular image that is distorted into a trapezoidal shape in the depth direction (S395). Specifically, as shown in FIG. 15, the main microcomputer 52 performs trapezoidal correction processing on the internal image of the refrigerating chamber 1 and the images of the door storage portions 11a and 11b.
  • the perspective projection conversion and its complementary processing such as the nearest neighbor method and the Lanczos method can be executed to convert an image of a storage portion having a substantially trapezoidal shape or a fan shape into a substantially rectangular shape.
  • the main microcomputer 52 performs image composition processing on the image created in S395 and creates it as one image (S396).
  • This image is displayed on the application of the smartphone 55 (see FIG. 7).
  • the embodiment shows only one example, and each step may be changed back and forth as appropriate.
  • the image data processing method in the refrigerator is an image data processing method in the refrigerator that processes image data captured by an image pickup device provided with a wide-angle lens including a plurality of food storage areas in the image pickup range.
  • the cutting process By the cutting process, at least a part of the area other than the food storage area, for example, the area where the wall surface of the refrigerator or the floor surface outside the refrigerator is shown without showing the food, is removed from the input photographed image (FIG. 11). can.
  • the combined image By synthesizing the cut-out images of each food storage area, the combined image (FIG. 16) is the food storage area displayed on the entire display screen of the smartphone 55, for example, on which this image is displayed. The ratio can be increased.
  • the cutting process is executed on one image captured by one imaging device 30, but two or more imaging devices are prepared, two or more images are captured, and each image is captured. Each portion including the food storage area included in the image data may be cut out.
  • the present embodiment is characterized in that the ratio of the area occupied by the food storage area to the entire image data is larger in the image after the composition processing than in the image data taken by the wide-angle lens. For example, when FIG. 11 and FIG. 16 are compared, the area where the food is reflected becomes larger and can be easily seen by the user.
  • the present embodiment is characterized in that the separation distance of the plurality of parts cut out by the cutting process is shortened after the synthesis process.
  • the distance between the door storage portion a, the storage c, and the door storage portion b shown in FIG. 16 is the distance between the door storage portion a, the storage c, and the door storage portion b shown in FIG. It can be smaller than the distance and easier for the user to see.
  • a plurality of storage areas separated from each other at an actual distance such as a door storage unit and a storage unit and two door storage units are output as one image, a swipe operation or a flick can be performed by the user as in Patent Document 1. It is possible to make the whole view easier without having to perform an operation to switch the displayed image.
  • the two door storage portions are opened by rotating the door hinges arranged at different positions as rotation axes and in opposite directions, the actual distance is relatively large in the refrigerator.
  • the refrigerating room 1 has been described with the example of a storage room with double doors, but it can also be applied to a storage room with single doors.
  • Second Embodiment In the first embodiment, the image composition processing of the refrigerating chamber 1 (first storage chamber) has been described, but the present invention is not limited to this. In the second embodiment, the image composition processing including the refrigerating room 1 (first storage room) and the vegetable room 4 and the like (second storage room) will be described.
  • FIG. 17 is a side view showing a state in which the drawer portion 6 of the refrigerator 100 according to the second embodiment is pulled out.
  • FIG. 18 is an image showing a cutting process of the drawer portion 6 of the refrigerator according to the second embodiment.
  • FIG. 19 is an image showing an enlargement process of the drawer portion 6 of the refrigerator according to the second embodiment.
  • the image pickup device 30 for photographing the inside of the refrigerator can image the inside of the drawer portion 6. Therefore, the stored state can be grasped from the image of the drawer portion 6.
  • FIG. 20 is an image showing a composition process of each image including the drawer portion 6 according to the second embodiment.
  • the image of the drawer portion 6 (second storage d) is also synthesized as compared with FIG.
  • the user can grasp the storage state including the refrigerating room 1 (first storage room) and the vegetable room 4 and the like (second storage room) at once.
  • FIG. 16 is an image synthesized from the captured image shown in FIG.
  • the refrigerator 100 has at least a first storage chamber that opens in the double doors and a second storage chamber that opens in the front-rear direction, and the cutting process is characterized by being the first storage chamber and the second storage chamber. Further, the refrigerator 100 has at least a first storage chamber that opens in the double doors and a second storage chamber that opens in the front-rear direction, the first storage chamber has a door storage portion, and the cutting process is performed in the first storage chamber. It may be a refrigerator, a door storage unit, and a second storage room. In the cutting process, when the opening of the second storage chamber is detected, the second storage chamber can be cut out.
  • FIG. 21 is another example of an image showing a composition process of each image including the drawer portion 6 according to the second embodiment. Since the display screen of the smartphone 55 on which an image is expected to be displayed is often rectangular, as shown in FIG. 21, the door storage portion a so that the combined image becomes rectangular as a whole. , B positions can also be moved from the side of the storage c. In the example of FIG. 21, the door storage portions a and b are moved to the opposite side of the second storage d from the storage c.
  • the cutting process is performed with at least one door storage unit, the inside of the first storage room, and the second storage room as food storage areas, respectively.
  • at least one door storage portion and the second storage chamber are arranged so as to be located on opposite sides of each other with the inside of the first storage chamber interposed therebetween.
  • FIG. 22A is an explanatory diagram showing a normal state of the image pickup apparatus 30.
  • FIG. 22B is an explanatory diagram showing a state in which the image pickup apparatus 30 is being stored.
  • FIG. 22C is an explanatory diagram showing a state when the image pickup apparatus 30 is stored.
  • the refrigerator 100 moves a box body forming a storage chamber with an open front, an image pickup device 30 provided outside the storage chamber, and the image pickup device 30 in a direction at least perpendicular to the opening.
  • a support portion 45 (hinge portion) that can support the refrigerator 45 (hinge portion) is provided.
  • the tip portion of the image pickup apparatus 30 having the camera 33 is configured to be folded in half from the support portion 45 and does not affect the depth dimension at the time of packing.
  • the load quantity of the image is not reduced, and even after the installation, the image pickup device 30 can be stored and the image pickup range can be blindfolded when the image pickup device 30 is not used.
  • the image pickup device 30 enables revolution (moving in a circular locus around the hinge portion; revolution around the axis) in the direction of looking into the inside of the refrigerator. Since the box body exists in the circular locus, the revolving angle is less than 360 ° when observed from the direction orthogonal to the revolving plane.
  • a drive unit capable of applying a force in the revolution direction may be attached to the image pickup device 30 and controlled so as to look into the inside of the refrigerator or the like at the time of image pickup.
  • the image pickup device 30 can be positioned within the movement locus of the door at the time of image pickup.
  • the present invention is not limited to the first embodiment and the second embodiment described above, and includes various modifications.
  • the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations.
  • the photographing button 39 of FIG. 3 may be provided on the refrigerating room door 1a or may be provided on the upper surface of the drawer portion 6.
  • a plurality of image pickup devices used for photographing may be prepared, and the image data of the food storage area included in each captured image may be used for compositing processing into one image.
  • a curved image taken by an image pickup device having a fisheye lens (wide-angle lens) provided outside the storage chamber is distorted, cut out, and combined, and one image is output.
  • An image pickup device having a wide-angle lens such as a fisheye lens provided outside the storage chamber captures a bird's-eye view of the inside of the storage chamber including the door storage section from above, corrects distortion of curved images, and cuts out each storage section.
  • image composition after processing and these image processing it is possible to provide the situation in the refrigerator in a state close to the user's line of sight with one image, and it becomes easy to recognize the situation in the refrigerator.
  • it is advantageous in terms of cost because it is not necessary to provide a large number of image pickup devices or an image pickup device having a complicated variable control, and the lens is damaged due to the stored items being close to the lens, the shooting range is obstructed, and the like. Can solve the problem of.
  • the user can check the latest status in the refrigerator with an image even when he / she is out, which can lead to unnecessary shopping and reduction of food loss.
  • Refrigerator room (1st storage room) 1a, 1b Refrigerator door 2 Ice making room (second storage room) 2a Ice making room door 3 Upper freezing room (second storage room) 3a Upper freezing room door 4 Vegetable room (second storage room) 4a Vegetable room door 5 Lower freezing room (second storage room) 5a Lower freezer door 6

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Combustion & Propulsion (AREA)
  • Physics & Mathematics (AREA)
  • Mechanical Engineering (AREA)
  • Thermal Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
  • Devices That Are Associated With Refrigeration Equipment (AREA)
PCT/JP2020/044607 2020-08-20 2020-12-01 冷蔵庫内の画像データ処理方法および冷蔵庫 WO2022038799A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080018909.3A CN114514405B (zh) 2020-08-20 2020-12-01 冷藏库内的图像数据处理方法和冷藏库
CN202310993504.7A CN116857878A (zh) 2020-08-20 2020-12-01 贮藏库

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-139594 2020-08-20
JP2020139594A JP2022035347A (ja) 2020-08-20 2020-08-20 冷蔵庫内の画像データ処理方法および冷蔵庫

Publications (1)

Publication Number Publication Date
WO2022038799A1 true WO2022038799A1 (ja) 2022-02-24

Family

ID=80322891

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/044607 WO2022038799A1 (ja) 2020-08-20 2020-12-01 冷蔵庫内の画像データ処理方法および冷蔵庫

Country Status (3)

Country Link
JP (1) JP2022035347A (zh)
CN (2) CN116857878A (zh)
WO (1) WO2022038799A1 (zh)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017161210A (ja) * 2016-03-07 2017-09-14 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． 冷蔵庫
JP2019219110A (ja) * 2018-06-20 2019-12-26 三菱電機株式会社 冷蔵庫

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003042626A (ja) * 2001-07-26 2003-02-13 Toshiba Corp 冷蔵庫
JP6402353B2 (ja) * 2013-03-29 2018-10-10 パナソニックＩｐマネジメント株式会社 冷蔵庫及び冷蔵庫システム
CN105222521B (zh) * 2013-04-23 2018-01-09 Lg电子株式会社 冰箱及其控制方法
JP2015014401A (ja) * 2013-07-04 2015-01-22 日立アプライアンス株式会社 冷蔵庫
US10365034B2 (en) * 2013-10-18 2019-07-30 Lg Electronics Inc. Refrigerator and control method for the same
JP6296908B2 (ja) * 2014-06-09 2018-03-20 三菱電機株式会社 冷蔵庫及びネットワークシステム
JP6323279B2 (ja) * 2014-09-22 2018-05-16 三菱電機株式会社 保冷保温庫、及び、その管理システム
JP6799958B2 (ja) * 2016-07-20 2020-12-16 三菱電機株式会社 冷蔵庫
CN108302855A (zh) * 2017-01-13 2018-07-20 富泰华工业（深圳）有限公司 冰箱及其摄像系统
JP7012262B2 (ja) * 2017-10-06 2022-01-28 パナソニックＩｐマネジメント株式会社 冷蔵庫

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017161210A (ja) * 2016-03-07 2017-09-14 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． 冷蔵庫
JP2019219110A (ja) * 2018-06-20 2019-12-26 三菱電機株式会社 冷蔵庫

Also Published As

Publication number Publication date
JP2023068052A (ja) 2023-05-16
JP2022035347A (ja) 2022-03-04
CN114514405A (zh) 2022-05-17
CN114514405B (zh) 2023-08-15
CN116857878A (zh) 2023-10-10

Similar Documents

Publication Publication Date Title
JP2021050909A (ja) 通知方法、システム
JP7500285B2 (ja) 冷蔵庫および出力装置
WO2016162956A1 (ja) 冷蔵庫
JP2014070796A (ja) 庫内撮影装置および冷蔵庫
EP2843336A1 (en) Refrigerator and refrigerator system for monitoring food
JP6322803B2 (ja) 冷蔵庫及び冷蔵庫システム
JP6167330B2 (ja) 冷蔵庫及び冷蔵庫システム
JP6167331B2 (ja) 冷蔵庫及び冷蔵庫システム
JP2000055533A (ja) 冷却貯蔵庫及びその管理システム
JP6361014B2 (ja) 冷蔵庫及び冷蔵庫システム
JP6485678B2 (ja) 冷蔵庫及び冷蔵庫システム
JP6167289B2 (ja) 冷蔵庫及びその冷蔵庫システム
WO2022038799A1 (ja) 冷蔵庫内の画像データ処理方法および冷蔵庫
JP6681555B2 (ja) 冷蔵庫
JP7513788B2 (ja) 冷蔵庫
JP6167288B2 (ja) 冷蔵庫
JP6500251B2 (ja) 冷蔵庫
JP6212700B2 (ja) 冷蔵庫及びその冷蔵庫システム
JP6671002B2 (ja) 冷蔵庫
JP6167329B2 (ja) 冷蔵庫
JP6145701B2 (ja) 冷蔵庫
JP6167328B2 (ja) 冷蔵庫
JP6114916B2 (ja) 冷蔵庫
JP2021096066A (ja) 冷蔵庫、システム
JP6145702B2 (ja) 冷蔵庫

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20950364

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20950364

Country of ref document: EP

Kind code of ref document: A1