WO2023036099A1 - 制冷电器中的库存管理系统 - Google Patents

制冷电器中的库存管理系统 Download PDF

Info

Publication number
WO2023036099A1
WO2023036099A1 PCT/CN2022/117153 CN2022117153W WO2023036099A1 WO 2023036099 A1 WO2023036099 A1 WO 2023036099A1 CN 2022117153 W CN2022117153 W CN 2022117153W WO 2023036099 A1 WO2023036099 A1 WO 2023036099A1
Authority
WO
WIPO (PCT)
Prior art keywords
confidence score
refrigeration appliance
image
determining
adjusted
Prior art date
Application number
PCT/CN2022/117153
Other languages
English (en)
French (fr)
Inventor
施罗德·迈克尔·古德曼
Original Assignee
海尔智家股份有限公司
青岛海尔电冰箱有限公司
海尔美国电器解决方案有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 海尔智家股份有限公司, 青岛海尔电冰箱有限公司, 海尔美国电器解决方案有限公司 filed Critical 海尔智家股份有限公司
Priority to CN202280060761.9A priority Critical patent/CN117999449A/zh
Publication of WO2023036099A1 publication Critical patent/WO2023036099A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2500/00—Problems to be solved
    • F25D2500/06—Stock management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/68—Food, e.g. fruit or vegetables

Definitions

  • the present invention relates generally to refrigeration appliances, and more particularly to an inventory management system and method of operating an inventory management system in a refrigeration appliance.
  • Refrigerated appliances typically include a cabinet defining a refrigerated compartment for receiving food for storage. Additionally, the refrigerated appliance includes one or more doors that are rotatably hinged to the cabinet to allow selective access to food products stored in the refrigerated compartment.
  • the refrigerating appliance may also include various storage components installed in the refrigerating compartment and designed to facilitate storing food therein. Such storage components may include shelves, boxes, shelves, or drawers that receive food items within the refrigerated compartment and aid in organizing and arranging the food items.
  • some appliances include cameras for monitoring food as it is added to or removed from the refrigeration appliance.
  • conventional camera systems may have difficulty identifying specific objects, differentiating similar products, and accurately identifying the location of objects within a refrigerated compartment.
  • refrigeration appliances that use cameras typically rely on simple image analysis and cannot use existing knowledge about current inventory or historical inventory storage practices when identifying objects.
  • a refrigeration appliance with a system for improved inventory management would be useful. More particularly, a refrigeration appliance that includes an inventory management system capable of monitoring incoming and outgoing inventory with an increased degree of confidence would be particularly beneficial.
  • a refrigerating appliance comprising: a case body defining a refrigerating compartment; a door rotatably hinged to the case body to provide selective access to the refrigerating compartment and a camera assembly for monitoring the refrigerated compartment.
  • a controller is operatively coupled to the camera assembly and is configured to obtain images using the camera assembly, analyze the images to identify objects added to or removed from the refrigerated compartment and raw confidence associated with the identified objects degree score, determining that the raw confidence score falls below a high confidence threshold, obtaining an adjusted confidence score associated with the object based at least in part on the raw confidence score and historical inventory data, determining that the adjusted confidence score exceeds the high confidence score confidence threshold, and modifying the inventory list in response to determining that the adjusted confidence score exceeds the high confidence threshold.
  • a method of implementing inventory management within a refrigeration appliance includes a refrigerated compartment and a camera assembly configured to monitor the refrigerated compartment.
  • the method includes acquiring an image using a camera assembly, analyzing the image to identify an object being added to or removed from the refrigerated compartment and a raw confidence score associated with the identified object, determining that the raw confidence score falls to a high confidence below the high confidence threshold, obtaining an adjusted confidence score associated with the object based at least in part on the raw confidence score and historical inventory data, determining that the adjusted confidence score exceeds the high confidence threshold, and in response to determining that the adjusted confidence score
  • the inventory list is modified if the degree score exceeds a high confidence threshold.
  • FIG. 1 provides a perspective view of a refrigeration appliance according to an exemplary embodiment of the present invention.
  • FIG. 2 provides a perspective view of the exemplary refrigeration appliance of FIG. 1 , with the door of the fresh food compartment shown in an open position to reveal the inventory management system, according to an exemplary embodiment of the present invention.
  • FIG. 3 provides a method for operating the example inventory management system of FIG. 2 in accordance with an example embodiment of the present invention.
  • FIG. 4 provides an image obtained using a camera of the exemplary inventory management system of FIG. 2 in accordance with an exemplary embodiment of the present invention.
  • FIG. 5 provides a flowchart of an exemplary process for implementing an inventory management method in a refrigeration appliance according to an exemplary embodiment of the present invention.
  • upstream refers to where the fluid flow is coming from, while “downstream” refers to the direction the fluid flow is going.
  • upstream refers to where the fluid flow is coming from
  • downstream refers to the direction the fluid flow is going.
  • includes and “including” are intended to be inclusive in a manner similar to the term “comprising”.
  • the term “or” is generally intended to be inclusive (ie, "A or B” is intended to mean “A or B or both”).
  • Approximate language is used to modify any quantitative representation that is amenable to variation without resulting in a change in the basic function to which it is related. Accordingly, a value modified by terms such as “about,” “approximately,” and “approximately” is not to be limited to the precise value specified. In at least some cases, the approximate language may correspond to the precision of the instrument used to measure the value. For example, approximate language may mean within a 10% margin.
  • FIG. 1 provides a perspective view of an exemplary refrigeration appliance 100
  • FIG. 2 illustrates some refrigeration appliances 100 with doors in open positions.
  • the refrigeration appliance 100 generally defines a vertical V, a lateral L and a lateral T, each of which is perpendicular to each other such that an orthogonal coordinate system is generally defined.
  • the cooling appliance 100 includes a housing 102 that is generally configured to house and/or support various components of the cooling appliance 100 and may also define one or more interior chambers or compartments of the cooling appliance 100 .
  • Room the terms “casing,” “housing,” and the like are generally intended to refer to the outer frame or support structure of refrigeration appliance 100, including, for example, any suitable number, type, or and constructed support structures, such as systems of elongated support members, multiple interconnected panels, or some combination thereof.
  • the box body 102 does not necessarily need to be enclosed, and may be an open structure that simply includes various components supporting the refrigeration appliance 100 . Rather, the case 102 may enclose some or all of the interior of the case 102 .
  • the tank 102 may have any suitable size, shape and configuration while remaining within the scope of the present invention.
  • the box 102 generally extends along a vertical V between a top 104 and a bottom 106, and along a lateral L between a first side 108 (e.g., the left side when viewed from the front as in FIG. 1 ) and a second side. Extending between two sides 110 (eg, the right side as viewed from the front in FIG. 1 ) and along a transverse direction T between a front portion 112 and a rear portion 114 .
  • terms such as “left”, “right”, “front”, “rear”, “top” or “bottom” are used with reference to the viewing angle from which a user approaches the cabinet 102 .
  • Case 102 defines a refrigerated compartment for receiving food for storage.
  • the cabinet 102 defines a fresh food compartment 122 disposed at or adjacent to the top 104 of the cabinet 102 and a freezer compartment 124 disposed at or adjacent to the bottom 106 of the cabinet 102 .
  • the refrigeration appliance 100 is generally called a bottom-mounted refrigerator.
  • the benefits of the present invention apply to other types and styles of cooling appliances, such as ceiling mounted cooling appliances, side by side cooling appliances, or single door cooling appliances.
  • aspects of the invention may also be applicable to other electrical appliances. Accordingly, the descriptions set forth herein are for example purposes only and are not intended to be limited in any way to any particular appliance or configuration.
  • Refrigerator door 128 is rotatably hinged to the edge of cabinet 102 for selective access to fresh food compartment 122 .
  • a freezer door 130 is disposed below the refrigerator door 128 to selectively enter the freezer compartment 124 .
  • Freezer door 130 is coupled to a freezer drawer (not shown) slidably mounted within freezer compartment 124 .
  • refrigeration door 128 forms a seal over front opening 132 defined by bin 102 (eg, extending in a plane defined by vertical V and lateral L).
  • a user may place items within the fresh food compartment 122 through the front opening 132 and may then close the refrigerator door 128 to facilitate climate control.
  • Refrigerator door 128 and freezer door 130 are shown in a closed position in FIG. 1 .
  • FIG. 1 Those skilled in the art will understand that other chamber and door configurations are possible and within the scope of the present invention.
  • FIG. 2 provides a perspective view of refrigeration appliance 100 shown with refrigeration door 128 in an open position.
  • various storage components are mounted within the food preservation compartment 122 to facilitate storage of food products therein, as will be understood by those skilled in the art.
  • storage components may include boxes 134 and shelves 136 . Each of these storage components is used to receive food products (eg, beverages or/or solid food products) and can assist in organizing these food products.
  • the box 134 can be mounted on the refrigerator door 128 or can be slid into a receiving space in the fresh food compartment 122 .
  • the storage components shown are for illustrative purposes only and that other storage components may be used and may have different sizes, shapes, and configurations.
  • Dispensing assembly 140 is typically used to dispense liquid water and/or ice. While an exemplary dispensing assembly 140 has been illustrated and described herein, it should be understood that various changes and modifications may be made to the dispensing assembly 140 while remaining within the scope of the invention.
  • the dispensing assembly 140 and its various components may be at least partially disposed within a dispenser recess 142 defined on one of the refrigeration doors 128 .
  • a dispenser recess 142 is defined on the front 112 of the refrigeration appliance 100 such that a user may operate the dispensing assembly 140 without opening the refrigeration door 128 .
  • the dispenser recess 142 is provided at a predetermined height, which is convenient for a user to take ice and enables the user to take ice without bending over.
  • the dispenser recess 142 is disposed at approximately the level of the user's chest.
  • Dispensing assembly 140 includes an ice dispenser 144 that includes a drain 146 for discharging ice from dispensing assembly 140 .
  • An actuation mechanism 148 shown as a paddle, is mounted below the drain opening 146 to operate the ice dispenser 144 or the water dispenser.
  • any suitable actuation mechanism may be used to operate ice dispenser 144 .
  • ice dispenser 144 may include a sensor (such as an ultrasonic sensor) or a button instead of a paddle.
  • Drain 146 and actuation mechanism 148 are external parts of ice dispenser 144 and are mounted in dispenser recess 142 .
  • refrigerator door 128 may define an ice bin compartment 150 (FIG. 2) that houses an ice maker and ice storage bin (not shown) configured to supply ice to a dispensing device recess 142.
  • a control panel 152 is provided to control the mode of operation.
  • the control panel 152 includes one or more inputs 154 for selection, such as a knob, button, touch screen interface, etc., such as a water dispense button and an ice dispense button, for selecting a desired mode of operation, such as crushed or non-crushed ice .
  • input 154 may be used to specify a fill volume or method of operating dispense assembly 140 .
  • the input 154 may communicate with a processing device or controller 156 . Signals generated in controller 156 operate refrigeration appliance 100 and dispensing assembly 140 in response to selector input 154 .
  • a display 158 such as an indicator light or a screen, may be provided on the control panel 152 . Display 158 may be in communication with controller 156 and may display information in response to signals from controller 156 .
  • processing device may refer to one or more microprocessors or semiconductor devices, and is not necessarily limited to a single element.
  • the processing device may be programmed to operate the refrigeration appliance 100 , the dispensing assembly 140 , and other components of the refrigeration appliance 100 .
  • the processing device may include or be associated with one or more storage elements (eg, persistent storage media).
  • the storage element comprises an electrically erasable programmable read-only memory (EEPROM).
  • EEPROM electrically erasable programmable read-only memory
  • a memory element may store information accessible to a processing device, including instructions executable by the processing device.
  • the instructions may be software or any collection of instructions and/or data which, when executed by the processing means, cause the processing means to perform operations.
  • external communication system 170 is used to allow interaction, data transfer, and other communications between refrigeration appliance 100 and one or more external devices.
  • the communication may be used to provide and receive operating parameters, user instructions or notifications, performance characteristics, user preferences, or any other suitable information for improved performance of the refrigeration appliance 100 .
  • external communication system 170 may be used to communicate data or other information to enhance the performance of one or more external devices or appliances and/or to improve user interaction with such devices.
  • external communication system 170 allows controller 156 of cooling appliance 100 to communicate with a separate device external to cooling appliance 100 , generally referred to herein as external device 172 . As described in more detail below, these communications may be facilitated using wired or wireless connections, such as via network 174 .
  • external device 172 may be any suitable device separate from refrigeration appliance 100 that is configured to provide and/or receive communications, information, data, or commands to a user.
  • the external device 172 may be, for example, a personal phone, smart phone, tablet, laptop or personal computer, wearable device, smart home system, or another mobile or remote device.
  • remote server 176 may communicate with refrigeration appliance 100 and/or external device 172 over network 174 .
  • the remote server 176 may be a cloud-based server 176 and thus be located at a remote location, such as in a separate state, country, or the like.
  • external device 172 may communicate with remote server 176 over network 174, such as the Internet, to send/receive data or information, provide user input, receive user notifications or instructions, interact with or control refrigeration appliance 100 wait.
  • external device 172 and remote server 176 may communicate with refrigeration appliance 100 to communicate similar information.
  • remote server 176 may be configured to receive and analyze images obtained by camera assembly 190, for example, to facilitate inventory analysis.
  • refrigeration appliance 100 may be performed using any type of wired or wireless connection and using any suitable type of communication network, provided below A non-limiting example of .
  • external device 172 may communicate directly or indirectly with refrigeration appliance 100 via any suitable wired or wireless communication connection or interface (eg, network 174 ).
  • network 174 may include one or more of a local area network (LAN), wide area network (WAN), personal area network (PAN), the Internet, a cellular network, any other suitable short-range or long-range wireless network, and the like.
  • any suitable communication means or protocol may be used, such as via radio, laser, infrared, Ethernet type devices and interfaces, etc.) to send communications.
  • Such communications may use various communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, Secure HTTP, SSL) .
  • TCP/IP Transmission Control Protocol/IP
  • HTTP HyperText Transfer Protocol
  • SMTP Simple Transfer Protocol
  • FTP FTP
  • encodings or formats e.g., HTML, XML
  • protection schemes e.g., VPN, Secure HTTP, SSL
  • An external communication system 170 according to an exemplary embodiment of the present invention is described herein.
  • the exemplary functions and configurations of the external communication system 170 provided herein are used as examples only in order to facilitate describing aspects of the present invention.
  • System configurations may vary, other communication means may be used to communicate directly or indirectly with one or more associated appliances, other communication protocols and procedures may be implemented, etc. Such changes and modifications are considered to be within the scope of the present invention.
  • the refrigeration appliance 100 may also include an inventory management system 180 that is generally configured to monitor one or more chambers of the refrigeration appliance 100 to monitor the addition or removal of inventory. More specifically, as described in more detail below, the inventory management system 180 may include a plurality of sensors, cameras, or other detection devices for monitoring the fresh food compartment 122 to detect items placed in or removed from the fresh food compartment 122. (eg, generally identified by reference numeral 182). In this regard, inventory management system 180 may use data from each of these devices to obtain a complete representation or knowledge of the identity, location, and/or other qualitative or quantitative characteristics of objects 182 within food storage compartment 122 . Although inventory management system 180 is described herein as monitoring fresh food compartment 122 to detect objects 182, it should be understood that aspects of the invention may be used to monitor objects or items in any other suitable appliance, compartment, or the like.
  • inventory management system 180 may include a camera assembly 190 that is typically configured and used to obtain images of refrigeration appliance 100 during operation.
  • camera assembly 190 includes one or more cameras 192 mounted to bin 102 , refrigerator door 128 or otherwise positioned within view of fresh food compartment 122 .
  • camera assembly 190 is described herein as being used to monitor fresh food compartment 122 of refrigeration appliance 100 , it should be understood that aspects of the invention may be used to monitor any other suitable area of any other suitable appliance, such as freezer compartment 124 .
  • camera 192 of camera assembly 190 is mounted to case 102 at front opening 132 of food preservation compartment 122 and is oriented to have a The field of view in chamber 122.
  • the camera assembly 190 may include a plurality of cameras 192 disposed within the housing 102 , wherein each of the plurality of cameras 192 has a Specify the monitoring area or range.
  • the field of view of each camera 192 may be limited or focused on a particular area within the fresh food compartment 122 .
  • the inventory management system 182 may include a plurality of cameras 192 that may be mounted to the side walls of the fresh food compartment 122 and may be spaced apart along the vertical V to cover different monitoring zones.
  • each camera 192 it is preferable to locate each camera 192 close to the front opening 132 of the fresh food compartment 122 and to orient each camera 192 so that the field of view points toward the fresh food compartment 122 . In this way, privacy concerns associated with obtaining images of the user of the refrigeration appliance 100 may be mitigated or avoided entirely.
  • the camera assembly 190 may be used to facilitate the inventory management process of the refrigeration appliance 100 .
  • various cameras 192 may be positioned at openings of the fresh food compartment 122 to monitor food items (generally identified as objects 182 ) being added to or removed from the fresh food compartment 122 .
  • each camera 192 may be oriented in any other suitable manner for monitoring any other suitable area in or around the refrigeration appliance 100 .
  • camera assembly 190 may include any suitable number, type, size and configuration of cameras 192 for obtaining images of any suitable zone or area within or about refrigeration appliance 100, according to alternative embodiments.
  • each camera 192 may include features for adjusting the field of view and/or orientation.
  • the images obtained by the camera assembly 190 may vary in number, frequency, angle, resolution, detail, etc., in order to enhance the clarity of certain areas around or within the cooling appliance 100 .
  • the controller 156 may be configured to illuminate the refrigerated compartment with one or more light sources prior to obtaining an image.
  • controller 156 (or any other suitable dedicated controller) of refrigeration appliance 100 may be communicatively coupled to camera assembly 190 and may be programmed or used to analyze images obtained by camera assembly 190, for example, to identify Items added to or removed from refrigeration appliance 100 are described in detail below.
  • controller 136 may be operatively coupled to camera assembly 190 for analyzing one or more images obtained by camera assembly 190 to extract useful information about objects 182 located within fresh food compartment 122 .
  • images obtained by camera assembly 190 may be used to extract barcodes, identify products, monitor movement of products, or obtain other product information related to object 182 .
  • the analysis can be performed locally (eg, on controller 156 ), or can be sent to a remote server (eg, remote server 176 via external communication network 170 ) for analysis.
  • a remote server eg, remote server 176 via external communication network 170
  • Such analysis is intended to facilitate inventory management, for example by identifying food products that are added to or removed from the refrigerated compartment.
  • Method 200 may be used to operate camera assembly 190, or any other suitable camera assembly for monitoring appliance operation or inventory.
  • controller 156 may be used to implement method 200 .
  • the exemplary method 200 is discussed herein merely to describe exemplary aspects of the invention, and is not intended to be limiting.
  • the method 200 includes: at step 210 , using a camera assembly to obtain an image of a refrigerated compartment of a refrigerated appliance.
  • the camera assembly 190 of the refrigeration appliance 100 may obtain an image 300 (such as shown in FIG. 4 ) inside the fresh food compartment 122 , and the image 300 may include a plurality of objects 182 in its field of view.
  • camera assembly 190 of refrigeration appliance 100 may obtain one or more images (eg, image 300 ) of fresh food compartment 122 , freezer compartment 124 , or any other zone or area in or around refrigeration appliance 100 .
  • camera 192 is directed downward from the top center of bin 102 and has a field of view covering the width of food preservation compartment 122 (eg, as shown in the photograph of FIG. 4 ). Also, the field of view may be centered on the front opening 132 at the front of the cabinet 102 where, for example, the refrigeration door 128 is seated against the front of the cabinet 102 . In this manner, the field of view of camera 192 and the resulting image obtained may capture any motion or movement of objects entering and/or exiting fresh food compartment 122 . Images obtained by camera assembly 190 may include one or more still images, one or more video clips, or any other suitable type and number of images suitable for identifying food items (e.g., generally identified by reference numeral 182) or inventory analysis. image.
  • images obtained by camera assembly 190 may include one or more still images, one or more video clips, or any other suitable type and number of images suitable for identifying food items (e.g., generally identified by reference numeral 182) or inventory analysis. image.
  • camera assembly 190 may acquire images under any suitable trigger, such as a time-based imaging schedule in which camera assembly 190 periodically images and monitors fresh food compartment 122 .
  • the camera assembly 190 may periodically capture low resolution images until motion is detected (eg, via image differentiation of the low resolution images), at which point one or more high resolution images may be obtained.
  • refrigeration appliance 100 may include one or more motion sensors (eg, optical, acoustic, electromagnetic, etc.) that A motion sensor is triggered, and camera assembly 190 may be operatively coupled to such motion sensor to obtain images of object 182 during such movement.
  • the refrigeration appliance 100 may include a door switch that detects when the refrigeration door 128 is opened, at which point the camera assembly 190 may begin to acquire one or more images.
  • image 300 may be acquired continuously or periodically while refrigeration door 128 is open.
  • obtaining the image 300 may include determining that the refrigeration door 128 of the refrigeration appliance 100 is open, and capturing the image at a set frame rate while the refrigeration door 128 is open.
  • the motion of the food item between image frames may be used to determine whether the food item 182 was removed from or added to the fresh food compartment 122 .
  • the images obtained by camera assembly 190 may vary in number, frequency, angle, resolution, detail, etc., in order to enhance the clarity of food item 182 .
  • controller 156 may be configured to illuminate a refrigerator light (not shown) while image 300 is being acquired. Other suitable triggers are possible and within the scope of the present invention.
  • Step 220 includes analyzing image 300 using a machine learning image recognition process to identify objects that were added to or removed from the refrigerated compartment and raw confidence scores (generally identified by reference numeral 310 ) associated with the identified objects. ). It should be understood that the analysis may utilize any suitable image analysis technique, image decomposition, image segmentation, image processing, and the like. This analysis may be performed entirely by controller 156, may be offloaded to a remote server for analysis, may be analyzed with user assistance (eg, via control panel 152), or may be analyzed in any other suitable manner. According to an exemplary embodiment of the present invention, the analysis performed at step 220 may include a machine learning image recognition process.
  • this image analysis may use any suitable image processing technique, image recognition process, or the like.
  • image analysis and the like may be used generally to refer to any suitable method of observation, analysis, image decomposition, feature extraction, image classification, etc. of one or more images, videos, or other visual representations of an object.
  • this image analysis may include the implementation of image processing techniques, image recognition techniques, or any suitable combination thereof.
  • image analysis may use any suitable image analysis software or algorithm to continuously or periodically monitor moving objects within fresh food compartment 122 . It should be appreciated that this image analysis or processing may be performed locally (eg, by controller 156 ) or remotely (eg, by offloading the image data to a remote server or network, eg, remote server 176 ).
  • analysis of the one or more images may include implementing image processing algorithms.
  • image processing and the like are generally intended to refer to any suitable method or algorithm for analyzing images that does not rely on artificial intelligence or machine learning techniques (e.g., in contrast to the machine learning image recognition process described below ).
  • image processing algorithms may rely on image differentiation, such as a pixel-by-pixel comparison of two consecutive images. This comparison can help identify substantial differences between sequentially acquired images, for example, to identify movement, the presence of a particular object, the presence of a particular condition, and the like.
  • one or more reference images may be obtained when certain conditions exist, and these reference images may be stored for future comparison with images obtained during operation of the appliance. The similarity and/or difference between the reference image and the obtained image can be used to extract useful information for improving the performance of the electrical appliance.
  • image differentiation may be used to determine when a pixel-level motion metric passes a predetermined motion threshold.
  • Processing algorithms may also include measures for isolating or eliminating noise in image comparisons, eg, due to image resolution, data transmission errors, inconsistent lighting, or other imaging errors. By removing this noise, image processing algorithms can improve accurate object detection, avoid false object detections, and isolate important objects, regions, or patterns within the image. Additionally or alternatively, the image processing algorithm may use other suitable techniques for identifying or identifying particular items or objects, such as edge matching, divide and conquer search, grayscale matching, histograms of receptive field responses, or another suitable example. process (eg, executed at controller 156 based on one or more captured images from one or more cameras). Other image processing techniques are possible and within the scope of the present invention.
  • image analysis may also include the use of artificial intelligence ("AI"), such as machine learning image recognition processes, neural network classification modules, any other suitable artificial intelligence (AI) techniques, and/or any other suitable image analysis techniques, examples of which are described in more detail below.
  • AI artificial intelligence
  • the various exemplary image analysis or evaluation procedures described below may be used independently, collectively, or interchangeably to extract detailed information about the images being analyzed to facilitate performance of one or more of the methods described herein or Improve appliance operation in other ways.
  • any suitable number and combination of image processing, image recognition, or other image analysis techniques may be used to obtain accurate analysis of the acquired images.
  • the image recognition process may use any suitable artificial intelligence technique, eg, any suitable machine learning technique, or eg, any suitable deep learning technique.
  • the image recognition process may include implementing a form of image recognition known as region-based convolutional neural network ("R-CNN") image recognition.
  • R-CNN may involve taking an input image and extracting region proposals including potential objects or regions of the image.
  • a "region proposal" may be one or more regions in an image that are likely to belong to a particular object, or may include adjacent regions that share common pixel properties.
  • a convolutional neural network is then used to compute features from the region proposals, and the extracted features will then be used to determine the classification of each specific region.
  • the image segmentation process may be used together with R-CNN image recognition.
  • image segmentation creates pixel-based masks for individual objects in an image and provides a more detailed or fine-grained understanding of various objects within a given image.
  • image segmentation can involve dividing the image into segments (e.g., into groups of pixels containing similar attributes), which Segments can be analyzed independently or in parallel to obtain a more detailed representation of one or more objects in the image. This may be referred to in this paper as "Mask R-CNN" etc., as opposed to the regular R-CNN architecture.
  • Mask R-CNN can be based on Fast R-CNN which is slightly different from R-CNN.
  • R-CNN first applies a convolutional neural network (“CNN”), which is then assigned to region proposals on covn5 feature maps, rather than initially segmented for region proposals.
  • CNN convolutional neural network
  • standard CNNs may be used to obtain, identify or detect any other qualitative or quantitative data related to one or more objects or regions within one or more images.
  • the K-means algorithm can be used.
  • the image recognition process may use any other suitable neural network process while remaining within the scope of the present invention.
  • the step of analyzing the one or more images may include using a deep belief network (“DBN”) image recognition process.
  • the DBN image recognition process can often consist of stacking many separate unsupervised networks that use the hidden layers of each network as input to the next layer.
  • the step of analyzing the one or more images may include implementing a deep neural network (“DNN”) image recognition process, which typically involves the use of neural networks with multiple layers between Network-Inspired Computing Systems).
  • DNN deep neural network
  • Other suitable image recognition processes, neural network processes, artificial intelligence analysis techniques, and combinations of the above or other known methods may be used while remaining within the scope of the present invention.
  • the image recognition process may include detecting certain conditions based on a comparison of initial conditions, which may rely on image subtraction techniques, image stacking techniques, image stitching, and the like. For example, subtracting images can be used to train a neural network with multiple classes for future comparison and image classification.
  • the machine learning image recognition model may be actively trained by the appliance with new images, may be provided with training data from the manufacturer or from another remote source, or may be trained in any other suitable manner.
  • the image recognition process relies at least in part on a neural network trained with multiple images of appliances in different configurations, subjected to different conditions, or interacting in different ways.
  • This training data can be stored locally or remotely, and can be transmitted to a remote server for use in training other appliances and models.
  • image processing and machine learning image recognition processes can be used together to facilitate improved image analysis, object detection, or to extract other useful qualitative or quantitative information from one or more images that can be used to improve the operation or performance of an appliance data or information.
  • the methods described herein may use any or all of these techniques interchangeably to improve the image analysis process and promote improved appliance performance and consumer satisfaction.
  • the image processing algorithms and machine learning image recognition processes described herein are exemplary only and are not intended to limit the scope of the invention in any way.
  • one or more images may be obtained by camera assembly 190 during performance of method 200 .
  • the image analysis performed at step 220 may identify number of objects 182 within image 300, eg, based on training of a machine learning model using similar food items 182 (eg, apples or oranges as exemplified herein).
  • the machine learning image recognition process may also provide raw confidence scores (eg, as generally identified by reference numeral 310 for each object 182 recognized in FIG. 4 ).
  • raw confidence score 310 may generally represent the probability that an object has been properly identified by a machine learning model, eg, expressed as a confidence percentage, where 100% corresponds to a full confidence.
  • raw confidence score 310 may be a direct output of a machine-learned image recognition model when analyzing image 300 and may be based on any suitable characteristic of object 182 being monitored or tracked.
  • each food object 182 may have identifiable characteristics, such as stems, discolorations, blemishes, or other characteristics that may be identifiable and associated with that particular object 182 (eg, similar to the object's fingerprint).
  • Machine learning image recognition models can identify individual objects based on their specific fingerprints and can use identifiable features from other images to improve the accuracy of object identification. It is worth noting, however, that historical data related to item storage and inventory analysis can be used to refine the raw confidence score 310 to facilitate improved inventory management.
  • aspects of the invention described below may involve method steps for improving the raw confidence score 310, particularly when it falls below a suitable confidence threshold.
  • step 230 may include determining that raw confidence score 310 has fallen below a high confidence threshold.
  • This high confidence threshold may be predetermined and programmed into the controller 156 as a threshold beyond which object identification will result in a change to the refrigerator inventory record or list.
  • step 230 may include determining that the original confidence score is above a low confidence threshold, eg, such that object 182 is identified for which the confidence score associated is not suitably high enough to warrant an inventory change.
  • Step 240 generally includes obtaining an adjusted confidence score (eg, generally identified by reference numeral 312 in FIG. 4 ) associated with the object based at least in part on raw confidence score 310 and historical inventory data.
  • an adjusted confidence score eg, generally identified by reference numeral 312 in FIG. 4
  • the terms "historical inventory data” and the like are generally intended to refer to any data or information related to the storage of items or food objects within the refrigeration appliance 100 . According to an exemplary embodiment, this historical inventory data may be used to generate adjusted confidence score 312 based on historical practice associated with refrigeration appliance 100 .
  • exemplary historical inventory data will be described herein, it should be understood that the data described is exemplary only, and is used to facilitate discussion of various aspects of the invention, not to limit the scope of the invention.
  • historical inventory data may include information related to common food items or objects 182 stored in various spatial regions (eg, generally identified by reference numeral 320 in FIG. 4 ).
  • the historical inventory data can be broken down into object classes and subclasses.
  • item classes may include vegetables (e.g., including subcategories such as peas, carrots, corn, etc.), fruits (e.g., including subcategories such as berries, cut fruit, etc.), prepared meals (e.g., including subcategories such as lasagna, pizza, microwave meals, etc.), dessert (e.g., including subcategories such as ice cream, baubles, candy, etc.), meat (e.g., including subcategories such as chicken, beef, pork etc.) and other classes.
  • vegetables e.g., including subcategories such as peas, carrots, corn, etc.
  • fruits e.g., including subcategories such as berries, cut fruit, etc.
  • prepared meals e.g., including subcategories such as lasagna, pizza, microwave meals, etc.
  • dessert e.g., including subcategories such as ice cream, baubles, candy, etc.
  • meat e.g., including subcategories
  • historical inventory data may include a location associated with each of the various objects 182 that may be stored within the refrigeration appliance 100 .
  • the refrigerated compartment 122 may be broken down into a plurality of spatial regions 320, and the frequency with which a particular object 182 is stored within a corresponding spatial region 320 may be recorded in historical inventory data.
  • the spatial region 320 being accessed is the lower left box 134 of the refrigerated compartment 122 .
  • historical inventory data may include a list of objects that are frequently placed within spatial region 320 (eg, in lower left box 134 ).
  • the object list may include object classifications as described above, as well as the number of occurrences that the object was placed in the particular spatial region 320, the frequency of such placement, or the likelihood that the particular object 182 will be placed in the particular spatial region 320 any other relevant data.
  • adjusted confidence score 312 may be determined by controller 156 based on raw confidence score 310 and historical inventory data. For example, if raw confidence score 310 is relatively low, but historical inventory data indicates that the identified object is very likely stored in specified spatial region 320, adjusted confidence score 312 may be increased relative to raw confidence score 310. Conversely, if an object is identified as having a relatively low raw confidence score 310, and historical inventory data indicates that such objects are not typically stored in the spatial region 320, the adjusted confidence score may be reduced relative to the raw confidence score 310 312.
  • step 240 may include identifying the spatial region 320 within the refrigerated compartment 122 where the object was added or removed, determining that the object's food type is the same as a common food type stored in the spatial region 320, and increasing the raw confidence score to Adjusted confidence score.
  • step 220 of analyzing image 300 using a machine learning image recognition model has identified that fruit is placed in lower left box 134 . Specifically, this step identifies an apple with 37% confidence and two oranges with 36% and 37% confidence, respectively. Additionally, historical inventory data indicates that apples are very often stored in this space region 320 , while oranges are rarely or never stored in the space region 320 . Accordingly, the original confidence score 310 may be boosted (e.g., from 37% to 44%) for objects identified as apples, while the original confidence score 310 associated with objects identified as oranges may be decreased (e.g., respectively 5% reduction).
  • step 250 may include determining that the adjusted confidence score exceeds a high confidence threshold.
  • step 240 may generally be used to increase confidence in objects identified in step 220, and thus may facilitate improved inventory management.
  • method 200 may include asking the user of refrigeration appliance 100 for the identity of the subject, or taking no further action and/or not inventorying the appliance. Make changes.
  • Step 260 includes modifying the inventory list in response to determining that the adjusted confidence score exceeds a high confidence threshold.
  • the controller 156 can monitor and track the inventory within the refrigeration appliance 100 by identifying whether food products 182 have been added to or removed from the fresh food compartment 122 .
  • the controller 156 may maintain a record of food items placed in or removed from the fresh food compartment 122 .
  • the controller 156 may add an apple to the appliance inventory list.
  • refrigeration appliance 100 may require the user to input the identities of the two objects identified as oranges, eg, via control panel 152 or external device 172 .
  • method 200 may also include updating historical inventory data based on items or objects 182 added to or removed from refrigeration appliance 100 .
  • step 270 may include adjusting historical inventory data in response to determining that the adjusted confidence score exceeds a high confidence threshold.
  • the controller 156 may be programmed to make a greater adjustment to the raw confidence score 310 to achieve the adjusted confidence score 312 as apples are more frequently stored in the lower left box 134 .
  • tracking event occurrences eg, when certain objects are placed within certain spatial regions 320
  • historical inventory data can be improved and lead to more accurate adjustments to confidence scores and improved inventory analysis and management.
  • the method 200 may also include determining 310 that the raw confidence score exceeds a high confidence threshold. In this case, it may not be necessary to determine an adjusted confidence score 312 to adjust the appliance inventory. Therefore, when raw confidence score 310 is this high, controller 156 may adjust historical inventory data and modify the inventory list for refrigeration appliance 100 . Conversely, method 200 may also include determining that the object cannot be identified with any level of confidence. In such a case, method 200 may include taking no further action or asking the user for the identity of the subject, eg, via control panel 152 or external device 172 .
  • method 400 may be similar to or interchangeable with method 200 and may be implemented by controller 156 of refrigeration appliance 100 .
  • the controller 156 may determine that an image should be captured, for example, by determining that the refrigeration door 128 has been opened.
  • Step 404 may include obtaining one or more images or videos and determining whether an object was added to or removed from the refrigerated compartment 122 .
  • Step 406 may be performed at step 406, eg, to identify the object and a corresponding confidence score associated with such object identification.
  • this image analysis may be the same as or similar to the image analysis described above with respect to step 220 .
  • Step 408 may include determining that no objects are identified as recognizable from the image, and step 410 may include asking the user for object identification and/or not making changes to inventory or historical inventory data.
  • step 412 may include identifying the object, but at a low level of confidence, eg, a level of confidence not high enough to warrant a change in appliance inventory or historical inventory data.
  • Step 414 may include obtaining location statistics and attempting to adjust a confidence score associated with object recognition, eg, based on historical inventory data. If at step 416 the adjusted confidence score is still not above the high confidence threshold, method 400 may proceed to step 410 (as described above). Conversely, if the adjusted confidence score now exceeds the high confidence threshold, step 418 may include updating appliance inventory and/or adjusting historical inventory data.
  • Step 420 may include identifying objects with a high confidence level (eg, above which the controller 156 should update the appliance inventory list and/or adjust historical inventory data).
  • Step 422 may include adjusting historical inventory data, for example, by adding locations, identified objects, and the like.
  • the camera assembly may include a camera mounted to the cabinet, placed on the bottom of the door, or otherwise configured to record activity within the one or more refrigerated compartments.
  • the appliance controller can record frequent storage locations for different types of items. These historical statistics can be used to improve future object recognition and inventory management, for example, by adjusting visual recognition confidence based on storage location.
  • a camera assembly may be used to monitor items being added to and/or removed from different areas within the refrigerated compartment.
  • the refrigerated compartment may be programmatically divided into multiple spatial regions, may be divided into quadrants for each level, or may be divided into any other suitable virtual zone, space or region.
  • occurrences of the item type or more generally of the class of items may be added to the region's history.
  • Statistics about common item placements can affect object classification and overall confidence in inventory changes.
  • a low confidence inventory change may be promoted to a high confidence change if the item is a common item type for a storage location.
  • the identified object is at high confidence, then the location or object occurrence is added to the statistics for that location and the inventory is updated based on that information.
  • the identified object is at low confidence, then the location statistics or confidence values for common items that increase the location will be obtained.
  • the inventory can be updated if the confidence level is high. Conversely, if the confidence level is low, the controller may ask the user to identify the item or not make a change to the inventory.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • Mechanical Engineering (AREA)
  • Thermal Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

一种制冷电器（100），包括：箱体（102），该箱体（102）限定制冷间室（122）；门体（128），该门体（128）可旋转地铰接到箱体（102），以提供选择性达到制冷间室（122）的途径；以及库存管理系统（180），该库存管理系统（180）安装在制冷间室（122）内，用于监测放置在制冷间室（122）内的对象（182）。库存管理系统（180）包括相机组件（190），该相机组件（190）在食品（182）被添加到制冷间室（122）或从中取出时获得食品（182）的图像（300）。电器（100）的控制器（156）使用机器学习图像识别过程来分析图像（300），以识别对象（182）和与对象（182）识别有关的对应置信度分数。然后使用历史库存数据来调节置信度分数并在经调节的置信度分数（312）超过高置信度阈值时修改库存列表。

Description

制冷电器中的库存管理系统 技术领域
本发明总体涉及制冷电器，更具体地涉及制冷电器中的库存管理系统和操作库存管理系统的方法。
背景技术
制冷电器通常包括箱体，该箱体限定用于接收食品以便储存的制冷间室。另外，制冷电器包括一个或多个门体，这些门体可旋转地铰接到箱体，以允许选择性地接近制冷间室中储存的食品。制冷电器还可以包括安装在制冷间室内并且设计成便于在其中储存食品的各种储存部件。这种储存部件可以包括在制冷间室内接收食品并且辅助组织和布置这些食品的搁架、盒、层架或抽屉。
特别地，经常期望具有存在于制冷电器内的物品的更新的库存，例如以便于重新排序，确保食物新鲜或避免变质等。由此，可能期望监测添加到制冷电器或从制冷电器取出的食品，并且获得与这些食品的存在、数量或质量有关的其他信息。某些传统的制冷电器具有用于监测制冷电器中的食品的系统。然而，这种系统通常需要用户交互，例如，通过控制面板进行的关于添加或取出的食品的直接输入。
作为对比，某些电器包括用于在食品被添加到制冷电器或从制冷电器取出时监测食品的相机。然而，传统的相机系统可能难以识别特定对象、区分类似产品以及精确地识别制冷间室内的对象的位置。而且，使用相机的制冷电器通常依赖于简单的图像分析，并且在识别对象时不能使用与当前库存或历史库存储存实践相关的现有知识。
因此，具有用于改进库存管理的系统的制冷电器将是有用的。更特别地，包括能够以提高的置信度监测进入和离开库存的库存管理系统的制冷电器将是特别有益的。
发明内容
本发明的各个方面以及优点将会在下文的描述中进行阐述，或者是通过描述可以显而易见的，或者是可以通过实施本发明而学到。
在一个示例性实施方式中，提供了一种制冷电器，包括：箱体，该箱体限定制冷间室；门体，该门体可旋转地铰接到箱体，以提供选择性到达制冷间室的途径； 以及相机组件，该相机组件用于监测制冷间室。控制器可操作地联接到相机组件，并且被配置为：使用相机组件获得图像，分析图像以识别被添加到制冷间室或从制冷间室取出的对象以及与被识别的对象相关联的原始置信度分数，确定原始置信度分数降到高置信度阈值以下，至少部分地基于原始置信度分数和历史库存数据获得与对象相关联的经调节的置信度分数，确定经调节的置信度分数超过高置信度阈值，并且响应于确定经调节的置信度分数超过高置信度阈值而修改库存列表。
在另一示例性实施方式中，提供了一种在制冷电器内实施库存管理的方法。该制冷电器包括制冷间室和设置成用于监测制冷间室的相机组件。方法包括：使用相机组件获得图像，分析图像以识别被添加到制冷间室或从制冷间室取出的对象以及与被识别的对象相关联的原始置信度分数，确定原始置信度分数降到高置信度阈值以下，至少部分地基于原始置信度分数和历史库存数据获得与对象相关联的经调节的置信度分数，确定经调节的置信度分数超过高置信度阈值，以及响应于确定经调节的置信度分数超过高置信度阈值而修改库存列表。
参照下文的描述以及所附权利要求，本发明的这些和其它的特征、方面以及优点将变得更容易理解。结合在本说明书中并且构成本说明书一部分的附图显示了本发明的实施方式并且与描述一起用于对本发明的原理进行解释。
附图说明
参照附图，说明书中阐述了面向本领域普通技术人员的本发明的完整公开，这种公开使得本领域普通技术人员能够实现本发明，包括本发明的最佳实施例。
图1提供了根据本发明的示例性实施方式的制冷电器的立体图。
图2提供了根据本发明的示例性实施方式的图1的示例性制冷电器的立体图，其中食物保鲜室的门体被示出为处于打开位置以露出库存管理系统。
图3提供了根据本发明的示例性实施方式的用于操作图2的示例性库存管理系统的方法。
图4提供了根据本发明的示例性实施方式的使用图2的示例性库存管理系统的相机获得的图像。
图5提供了根据本发明的示例性实施方式的用于在制冷电器中实施库存管理方法的示例性过程的流程图。
附图标记在本说明书和附图中的重复使用旨在表示本发明的相同或相似的特征或元件。
具体实施方式
现在将详细地参照本发明的实施方式，其中的一个或多个示例示于附图中。每个示例都以对发明进行解释的方式给出，并不对本发明构成限制。实际上，对于本领域技术人员而言显而易见的是，能够在不偏离本发明的范围或者精神的前提下对本发明进行多种改型和变型。例如，作为一个实施方式的一部分示出或者进行描述的特征能够用于另一个实施方式，从而产生又一个实施方式。因此，期望的是，本发明覆盖落入所附权利要求及其等同形式的范围内的这些改型以及变型。
如本文所用的，术语“第一”、“第二”和“第三”可以互换使用以将一个部件与另一个部件区分开，并且这些术语并不旨在表示各个部件的位置或重要性。术语“上游”和“下游”是指相对于流体通路中的流体流动的相对方向。例如，“上游”是指流体流动的来向，而“下游”是指流体流动的去向。术语“包括(includes)”和“包括(including)”旨在以类似于术语“包括(comprising)”的方式为包括的。类似地，术语“或”通常旨在是包括的(即，“A或B”旨在意指“A或B或两者”)。
如本文在整个说明书和权利要求书中使用的近似语言被应用于修饰任何定量表示，该定量表示可容许在不导致其相关的基本功能改变的情况下变化。因此，由诸如“大约”、“近似”以及“大致”的术语修饰的值不限于所指定的精确值。在至少一些情况下，近似语言可对应于用于测量值的仪器的精度。例如，近似语言可以指在10％的裕度内。
现在参见附图，将描述根据本发明的示例性方面的示例性电器。具体地，图1提供了示例性制冷电器100的立体图，图2示例了一些门体处于打开位置的制冷电器100。如图示例，制冷电器100通常限定竖向V、侧向L和横向T，竖向V、侧向L和横向T中的每一个相互垂直，使得大体限定正交坐标系。
根据示例性实施方式，制冷电器100包括箱体102，该箱体102通常用于容纳和/或支撑制冷电器100的各种部件，并且还可限定制冷电器100的一个或多个内部腔室或间室。在这点上，如本文所用的，术语“箱体”、“壳体”等通常旨在指制冷电器100的外框架或支撑结构，例如，包括由任何合适的材料形成的任何合适数量、类型和构造的支撑结构，诸如细长支撑构件、多个互连面板或其一些组合的系统。应当理解，箱体102不一定需要围合，可以是简单地包括支撑制冷电器100的各种元件的开放结构。相反，箱体102可以包围箱体102内部的一些或所有部分。应当理解，箱体102可具有任何合适的尺寸、形状和构造，均在本发明的范围内。
如图示例，箱体102通常沿着竖向V在顶部104与底部106之间延伸，沿着侧向L在第一侧108(例如，如图1中从前方观察时的左侧)与第二侧110(例如，如图1中从前方观察时的右侧)之间延伸，并且沿着横向T在前部112与后部114之间延伸。一般而言，诸如“左”、“右”、“前”、“后”、“顶部”或“底部”的术语是参考用户接近箱体102的视角来使用的。
箱体102限定用于接收食品以便储存的制冷间室。特别地，箱体102限定设置在箱体102的顶部104处或与其相邻设置的食物保鲜室122和布置在箱体102的底部106处或与其相邻布置的冷冻室124。由此可见，制冷电器100通常被称为底置式冰箱。然而，认识到，本发明的益处适用于其他类型和样式的制冷电器，例如，顶置式制冷电器、对开门式制冷电器或单门制冷电器。而且，本发明的方面也可以适用于其他电器。因此，本文阐述的描述仅出于示例目的，而无意于在任何方面限制任何特定的电器或配置。
冷藏门体128可旋转地铰接到箱体102的边缘，以便选择性地进入食物保鲜室122。另外，在冷藏门体128的下方布置冷冻门体130，以便选择性地进入冷冻室124。冷冻门体130联接至可滑动地安装在冷冻室124内的冷冻抽屉(未示出)。通常，冷藏门体128在由箱体102限定的前开口132(例如，在由竖向V和侧向L限定的平面内延伸)上形成密封。在这点上，当冷藏门体128打开时，用户可以通过前开口132将物品放置在食物保鲜室122内，然后可以关闭冷藏门体128以便于气候控制。冷藏门体128和冷冻门体130在图1中被示出为处于关闭位置。本领域技术人员将理解，其它腔室和门体构造是可行的，并且在本发明的范围内。
图2提供了在冷藏门体128处于打开位置的情况下示出的制冷电器100的立体图。如图2所示，如本领域技术人员将理解的，各种储存部件被安装在食物保鲜室122内，以促进食品在其中的储存。特别地，储存部件可以包括盒134和层架136。这些储存部件中的每一个用于接收食品(例如，饮料或/或固体食品)，并且可以辅助组织这些食品。如图所示，盒134可以安装在冷藏门体128上或者可以滑入食物保鲜室122中的容纳空间中。应当理解，所示的储存部件仅用于说明的目的，并且可以使用其它储存部件，并且其它储存部件可以具有不同的尺寸、形状以及构造。
再次参见图1，将描述根据本发明的示例性实施方式的分配组件140。虽然将说明并描述分配组件140的几个不同的示例性实施方式，但类似的附图标记可用于指代类似的部件和特征。分配组件140通常用于分配液态水和/或冰。虽然在本文中说明并描述了示例性分配组件140，但应当理解，可以在保持在本发明的范围内的同时 对分配组件140进行各种变更和修改。
分配组件140及其各种部件可以至少部分地设置在限定于冷藏门体128中的一个上的分配器凹部142内。在这点上，分配器凹部142限定在制冷电器100的前部112上，使得用户可以在不打开冷藏门体128的情况下操作分配组件140。另外，分配器凹部142设置在预定高度处，该预定高度方便用户取冰，并且使得用户能够在不需要弯腰的情况下取冰。在示例性实施方式中，分配器凹部142设置在接近用户的胸部水平的位置处。
分配组件140包括冰分配器144，该冰分配器144包括用于从分配组件140排出冰的排放口146。被示出为拨片的致动机构148安装在排放口146下方，以便操作冰分配器144或水分配器。在可选示例性实施方式中，可以使用任意合适的致动机构来操作冰分配器144。例如，冰分配器144可以包括传感器(诸如超声传感器)或按钮，而不是拨片。排放口146和致动机构148是冰分配器144的外部零件，并且安装在分配器凹部142中。与之相比，冷藏门体128可以限定容纳制冰机和储冰盒(未示出)的冰盒室150(图2)，该制冰机和储冰盒被构造成将冰供应至分配器凹部142。
设置控制面板152，以便控制操作模式。例如，控制面板152包括一个或多个用于选择的输入154，诸如旋钮、按钮、触摸屏界面等，诸如水分配按钮和冰分配按钮，用于选择期望的操作模式，诸如碎冰或非碎冰。另外，输入154可以用于指定填充容积或操作分配组件140的方法。在这点上，输入154可以与处理装置或控制器156通信。在控制器156中生成的信号响应于选择器的输入154而操作制冷电器100和分配组件140。另外，可以在控制面板152上设置显示器158，诸如指示灯或屏幕。显示器158可以与控制器156通信，并且可以响应于来自控制器156的信号而显示信息。
如本文中使用的，“处理装置”或“控制器”可以指一个或多个微处理器或半导体装置，并且不必限于单个元件。处理装置可以被编程为操作制冷电器100、分配组件140以及制冷电器100的其他部件。处理装置可以包括一个或多个存储元件(例如，永久存储介质)或与其关联。在一些这种实施方式中，存储元件包括电可擦可编程只读存储器(EEPROM)。通常，存储元件可以存储处理装置可访问的信息，包括可以由处理装置执行的指令。可选地，指令可以是软件或指令和/或数据的任意集合，该软件或指令和/或数据的任意集合在由处理装置执行时，使得处理装置执行操作。
仍然参见图1，将描述根据本发明的示例性实施方式的外部通信系统170的示意图。通常，外部通信系统170用于允许制冷电器100与一个或多个外部装置之间的 交互、数据传送和其他通信。例如，该通信可以用于提供和接收操作参数、用户指令或通知、性能特性、用户偏好或用于制冷电器100的改进性能的任何其它合适的信息。另外，应当理解，外部通信系统170可用于传送数据或其它信息，以提高一个或多个外部装置或电器的性能和/或改进与这种装置的用户交互。
例如，外部通信系统170允许制冷电器100的控制器156与制冷电器100外部的独立装置进行通信，该独立装置在本文中通常被称为外部装置172。如以下更详细描述的，这些通信可以使用有线或无线连接(诸如经由网络174)来促进。通常，外部装置172可以是与制冷电器100分开的任何合适的装置，该装置被配置为向用户提供和/或从用户接收通信、信息、数据或命令。在这点上，外部装置172可以是例如个人电话、智能电话、平板电脑、膝上型或个人计算机、可穿戴装置、智能家庭系统或者另一移动或远程装置。
另外，远程服务器176可以通过网络174与制冷电器100和/或外部装置172通信。在这点上，例如，远程服务器176可以是基于云的服务器176，由此位于远处位置，诸如在单独的州、国家等。根据示例性实施方式，外部装置172可通过网络174(诸如因特网)与远程服务器176通信，以发送/接收数据或信息、提供用户输入、接收用户通知或指令、与制冷电器100交互或控制制冷电器等。另外，外部装置172和远程服务器176可以与制冷电器100通信以传送类似的信息。根据示例性实施方式，远程服务器176可被配置为接收和分析由相机组件190获得的图像，例如以便于库存分析。
通常，可以使用任何类型的有线或无线连接并且使用任何合适类型的通信网络来进行制冷电器100、外部装置172、远程服务器176和/或其它用户装置或电器之间的通信，下面提供了通信网络的非限制性示例。例如，外部装置172可以通过任何合适的有线或无线通信连接或接口(例如网络174)与制冷电器100直接或间接通信。例如，网络174可以包括局域网(LAN)、广域网(WAN)、个域网(PAN)、因特网、蜂窝网络、任何其他合适的短程或远程无线网络等中的一个或多个。另外，可以使用任何合适的通信装置或协议(诸如经由
Figure PCTCN2022117153-appb-000001
无线电、激光、红外、以太网类型的装置和接口等)来发送通信。另外，这种通信可以使用各种通信协议(例如，TCP/IP、HTTP、SMTP、FTP)、编码或格式(例如，HTML、XML)和/或保护方案(例如，VPN、安全HTTP、SSL)。
本文描述了根据本发明的示例性实施方式的外部通信系统170。然而，应当理解，本文提供的外部通信系统170的示例性功能和配置仅用作示例，以便于描述本发明 的各方面。系统配置可以变化，其他通信装置可以用于直接或间接地与一个或多个关联的电器通信，可以实施其他通信协议和步骤等。这些变化和修改被认为在本发明的范围内。
现在一般参见图2，制冷电器100还可以包括库存管理系统180，该库存管理系统180通常配置为监测制冷电器100的一个或多个腔室，以监测库存的添加或取出。更具体地，如以下更详细地描述的，库存管理系统180可以包括多个传感器、相机或其他检测装置，其用于监测食物保鲜室122，以检测设置在食物保鲜室122中或从其取出的对象(例如，通常由附图标记182标识)。在这点上，库存管理系统180可以使用来自这些装置中的每一个的数据来获得食物保鲜室122内的对象182的身份、位置和/或其它定性或定量特性的完整表示或知识。尽管库存管理系统180在本文中被描述为监测食物保鲜室122，以便检测对象182，但是应当理解，本发明的各方面可以用于监测任何其他合适的电器、腔室等中的对象或物品。
如图2示意性所示，库存管理系统180可以包括相机组件190，该相机组件190通常设置和用于在运行期间获得制冷电器100的图像。具体地，根据所示例的实施方式，相机组件190包括一个或多个相机192，这些相机192安装到箱体102、冷藏门体128或以其他方式设置在食物保鲜室122的视野内。尽管本文将相机组件190描述为用于监测制冷电器100的食物保鲜室122，但是应当理解，本发明的各方面可以用于监测任何其他合适的电器的任何其他合适的区域，例如冷冻室124。如图2中最佳示出的，相机组件190的相机192在食物保鲜室122的前开口132处安装到箱体102，并且被定向为具有被引导跨过前开口132和/或进入食物保鲜室122中的视场。
尽管在图2中示例了单个相机192，但是应当理解，相机组件190可以包括设置在箱体102内的多个相机192，其中，多个相机192中的每一个具有位于食物保鲜室122周围的指定监测区或范围。在这点上，例如，各个相机192的视场可以被限制到或聚焦在食物保鲜室122内的特定区域上。例如，库存管理系统182可以包括多个相机192，这些相机192可以安装到食物保鲜室122的侧壁并且可以沿着竖向V隔开，以覆盖不同的监测区。
然而，特别地，最好是将各个相机192设置为接近食物保鲜室122的前开口132，并且将各个相机192的方向确定为视场指向食物保鲜室122。这样，与获得制冷电器100的用户的图像有关的隐私问题可以减轻或完全避免。根据示例性实施方式，相机组件190可以用于促进制冷电器100的库存管理过程。由此可见，各个相机192可 以设置在食物保鲜室122的开口处，以监测被添加到食物保鲜室122或从其中取出的食品(通常标识为对象182)。
根据另一些实施方式，各个相机192可以以任意其他合适的方式定向成用于监测制冷电器100内或周围的任意其他合适的区域。应当理解，根据可选实施方式，相机组件190可以包括任意合适数量、类型、尺寸和配置的相机192，用于获得制冷电器100内或周围的任意合适的区或区域的图像。另外，应当理解，各个相机192可以包括用于调节视场和/或取向的特征。
应当理解，由相机组件190获得的图像可以在数量、频率、角度、分辨率、细节等方面变化，以便提高制冷电器100周围或内的特定区域的清晰度。另外，根据示例性实施方式，控制器156可以用于在获得图像之前使用一个或多个光源照亮制冷间室。特别地，制冷电器100的控制器156(或任意其他合适的专用控制器)可以通信地联接到相机组件190，并且可以被编程或用于分析由相机组件190获得的图像，例如，以便识别被添加到制冷电器100或从其取出的物品，如以下详细描述的。
通常，控制器136可以可操作地联接到相机组件190，用于分析由相机组件190获得的一个或多个图像，以提取关于位于食物保鲜室122内的对象182的有用信息。在这点上，例如，由相机组件190获得的图像可以用于提取条形码、识别产品、监测产品的运动、或获得与对象182有关的其他产品信息。特别地，该分析可以在本地(例如，在控制器156上)执行，或者可以被发送到远程服务器(例如，经由外部通信网络170的远程服务器176)以用于分析。这种分析旨在例如通过识别被添加到制冷间室或从制冷间室取出的食品来促进库存管理。
既然已经呈现了根据本发明的示例性实施方式的制冷电器100和相机组件190的结构和构造，则提供用于操作相机组件190的示例性方法200。方法200可用于操作相机组件190，或操作用于监测电器操作或库存的任意其它合适的相机组件。在这点上，例如，控制器156可以用于实施方法200。然而，应当理解，示例性方法200在本文仅讨论为描述本发明的示例性方面，而不旨在限制。
如图3所示，方法200包括：在步骤210，使用相机组件获得制冷电器的制冷间室的图像。例如，继续上述示例，制冷电器100的相机组件190可以获得食物保鲜室122内的图像300(例如如图4所示)，该图像300可以在其视场中包括多个对象182。在这点上，制冷电器100的相机组件190可以获得食物保鲜室122、冷冻室124或者制冷电器100内或周围的任何其它区或区域的一个或多个图像(例如，图像300)。
具体地，根据示例性实施方式，相机192从箱体102的顶部中心向下定向，并 且具有覆盖食物保鲜室122的宽度的视场(例如，如图4的照片所示)。而且，该视场可以以箱体102的前部处的前开口132为中心，例如，在该开口处，抵靠箱体102的前部安置冷藏门体128。这样，相机192的视场以及所获得的结果图像可以捕获对象进入和/或离开食物保鲜室122的任意运动或移动。通过相机组件190获得的图像可以包括一个或多个静止图像、一个或多个视频剪辑、或者适合于识别食品(例如，通常由附图标记182标识)或库存分析的任意其他合适类型和数量的图像。
特别地，相机组件190可以在任何合适的触发(诸如基于时间的成像时间表)下获得图像，在成像时间表中，相机组件190周期性地对食物保鲜室122进行成像和监测。根据另一些实施方式，相机组件190可以周期性地拍摄低分辨率图像，直到检测到运动(例如，经由低分辨率图像的图像区分)为止，此时可以获得一个或多个高分辨率图像。根据另一些实施方式，制冷电器100可以包括一个或多个运动传感器(例如，光学的、声学的、电磁的等)，当对象182被添加到食物保鲜室122或从中取出时，该一个或多个运动传感器被触发，并且相机组件190可以可操作地联接到这样的运动传感器，以在这样的移动期间获得对象182的图像。
根据另一些实施方式，制冷电器100可以包括门体开关，该门体开关检测冷藏门体128何时打开，在该时刻，相机组件190可以开始获得一个或多个图像。根据示例性实施方式，在冷藏门体128打开的同时，可以连续地或周期性地获得图像300。在这点上，获得图像300可以包括确定制冷电器100的冷藏门体128是打开的，并且在冷藏门体128打开的同时以设定的帧率捕获图像。
特别地，食品在图像帧之间的运动可以用于确定食品182是否从食物保鲜室122中取出或添加到其中。应当理解，由相机组件190获得的图像可以在数量、频率、角度、分辨率、细节等方面变化，以便提高食品182的清晰度。另外，根据示例性实施方式，控制器156可以用于在获得图像300的同时照亮冰箱灯(未示出)。其它合适的触发是可行的，并且在本发明的范围内。
步骤220包括使用机器学习图像识别过程来分析图像300以识别被添加到制冷间室或从制冷间室取出的对象以及与被识别的对象相关联的原始置信度分数(通常由附图标记310标识)。应当理解，该分析可以利用任意合适的图像分析技术、图像分解、图像分割、图像处理等。该分析可以完全由控制器156执行，可以卸载到远程服务器来分析，可以在用户辅助下分析(例如，经由控制面板152)，或者可以以任意其他合适的方式分析。根据本发明的示例性实施方式，在步骤220执行的分析可以包括机器学习图像识别过程。
根据示例性实施方式，该图像分析可以使用任何合适的图像处理技术、图像识别过程等。如本文所用的，术语“图像分析”等通常可以用于指代对象的一个或多个图像、视频或其他视觉表示的观察、分析、图像分解、特征提取、图像分类等的任何合适的方法。如以下更详细地解释的，该图像分析可以包括图像处理技术、图像识别技术或其任何适当组合的实施。在这点上，图像分析可以使用任何合适的图像分析软件或算法来持续地或周期性地监测食物保鲜室122内的移动对象。应当理解，该图像分析或处理可以在本地(例如，由控制器156)或远程(例如，通过将图像数据卸载到远程服务器或网络，例如，远程服务器176)执行。
具体地，对一个或多个图像的分析可以包括实施图像处理算法。如本文所用的，术语“图像处理”等通常旨在指代用于分析图像的不依赖于人工智能或机器学习技术的任何合适的方法或算法(例如，与以下描述的机器学习图像识别过程形成对比)。例如，图像处理算法可以依赖于图像区分，例如两个连续图像的逐像素比较。该比较可以帮助识别顺序获得的图像之间的实质差异，例如，以识别移动、特定对象的存在、特定条件的存在等。例如，当特定条件存在时，可以获得一个或多个参考图像，并且这些参考图像可以被存储，以用于将来与在电器运行期间获得的图像进行比较。参考图像与获得的图像之间的相似性和/或差异可以用于提取用于提高电器性能的有用信息。例如，图像区分可以用于确定像素级运动度量何时通过预定运动阈值。
处理算法还可以包括用于隔离或消除例如由于图像分辨率、数据传输误差、不一致照明或其他成像误差而产生的图像比较中的噪声的措施。通过消除这种噪声，图像处理算法可以改善准确的对象检测，避免错误的对象检测，并且隔离图像内的重要对象、区域或图案。另外或可选地，图像处理算法可以使用用于识别或标识特定物品或对象的其他合适的技术，诸如边缘匹配、分治搜索、灰度匹配、感受野响应的直方图或另一合适的例程(例如，基于来自一个或多个相机的一个或多个捕获的图像在控制器156处执行)。其它图像处理技术也是可行的，并且在本发明的范围内。
除了上述图像处理技术之外，图像分析还可以包括利用人工智能(“AI”)，诸如机器学习图像识别过程、神经网络分类模块、任何其他合适的人工智能(AI)技术和/或任何其他合适的图像分析技术，其示例将在下面更详细地描述。而且，以下描述的各个示例性图像分析或评估过程可以独立地、共同地或可互换地使用，以提取关于被分析的图像的详细信息，从而促进本文描述的一个或多个方法的执行或以其 他方式改进电器运行。根据示例性实施方式，可以使用任何合适数量的图像处理、图像识别或其他图像分析技术及其组合来获得对所获得的图像的准确分析。
在这点上，图像识别过程可以使用任意合适的人工智能技术，例如，任意合适的机器学习技术，或者例如，任意合适的深度学习技术。根据示例性实施方式，图像识别过程可以包括实施称为基于区域的卷积神经网络(“R-CNN”)图像识别的一种形式的图像识别。一般而言，R-CNN可包括取得输入图像并提取包括图像的潜在对象或区域的区域建议。在这点上，“区域建议”可以是图像中可能属于特定对象的一个或多个区域，或者可以包括共享共同像素特性的相邻区域。然后使用卷积神经网络来从区域建议计算特征，然后将使用所提取的特征来确定各个特定区域的分类。
根据另一些实施方式，可以将图像分割过程与R-CNN图像识别一起使用。通常，图像分割为图像中的各个对象创建基于像素的掩码，并且提供对给定图像内的各种对象的更详细或更精细的理解。在这点上，代替处理整个图像(即，像素的大集合，其中许多像素可能不包含有用信息)，图像分割可以涉及将图像划分为片段(例如，划分为包含类似属性的像素组)，这些片段可以独立地或并行地分析，以获得图像中的一个或多个对象的更详细表示。这在本文中可以被称为“掩码R-CNN”等，与常规的R-CNN架构相反。例如，掩码R-CNN可以基于与R-CNN略微不同的快速R-CNN。例如，R-CNN首先应用卷积神经网络(“CNN”)，然后将其分配给covn5特性图上的区域推荐，而不是初始地分割为区域推荐。另外，根据示例性实施方式，标准CNN可用于获得、识别或检测与一个或多个图像内的一个或多个对象或区域有关的任何其他定性或定量数据。另外，可以使用K均值算法。
根据另一些实施方式，图像识别过程可以使用任意其他合适的神经网络过程，同时保持在本发明的范围内。例如，分析一个或多个图像的步骤可以包括使用深度信念网络(“DBN”)图像识别过程。DBN图像识别过程通常可以包括堆叠许多单独的无监督网络，这些网络使用各个网络的隐藏层作为下一层的输入。根据另一些实施方式，分析一个或多个图像的步骤可以包括实施深度神经网络(“DNN”)图像识别过程，其通常包括使用在输入与输出之间具有多个层的神经网络(由生物神经网络启示的计算系统)。可以使用其他合适的图像识别过程、神经网络过程、人工智能分析技术以及上述或其他已知方法的组合，同时保持在本发明的范围内。
另外，应当理解，可以使用各种传送技术，但是不要求使用这样的技术。如果使用传送技术学习，则可以利用公共数据集来预训练神经网络架构，诸如VGG16/VGG19/ResNet50，然后可以利用电器特定数据集来重新训练最后一层。另外 或可选地，图像识别过程可包括基于初始条件的比较而检测某些条件，可依赖于图像减影技术、图像堆叠技术、图像拼接等。例如，减影图像可以用于训练具有多个类别的神经网络，以用于将来的比较和图像分类。
应当理解，机器学习图像识别模型可以由电器利用新图像主动训练，可以被提供有来自制造商或来自另一远程源的训练数据，或者可以以任何其它合适的方式训练。例如，根据示例性实施方式，该图像识别过程至少部分地依赖于神经网络，该神经网络利用不同配置的电器的多个图像训练、经历不同条件或以不同方式交互。该训练数据可以本地或远程地存储，并且可以被传送到远程服务器以用于训练其他电器和模型。
应当理解，图像处理和机器学习图像识别过程可以一起使用，以便于改进的图像分析、对象检测，或者从一个或多个图像中提取可以用于改进电器的运行或性能的其他有用的定性或定量数据或信息。实际上，本文描述的方法可以可互换地使用这些技术中的任何或全部来改进图像分析过程并且促进改进的电器性能和消费者满意度。本文描述的图像处理算法和机器学习图像识别过程仅是示例性的，并且不旨在以任何方式限制本发明的范围。
现在简要地参见图4，可以在方法200的实施期间由相机组件190获得一个或多个图像(例如，图像300)。如图示例，在步骤220执行的图像分析可以例如基于使用类似食品182(例如，如本文示例为苹果或桔子)对机器学习模型的训练来识别图像300内的多个对象182。如上简要注释，除了对象识别之外，机器学习图像识别过程还可以提供原始置信度分数(例如，如针对在图4中识别的各个对象182通常由附图标记310标识)。在这点上，例如，原始置信度分数310通常可以表示对象已经被机器学习模型适当识别的概率，例如，以置信度百分比表达，其中100％对应于完全置信度。
通常，原始置信度分数310可以是在分析图像300时机器学习图像识别模型的直接输出，并且可以基于被监测或跟踪的对象182的任何合适的特性。例如，各个食物对象182可以具有可识别的特征，诸如茎、变色、瑕疵、或其他可以是可识别的并且与该特定对象182相关联的特征(例如，类似于该对象的指纹)。机器学习图像识别模型可以基于各个对象的特定指纹来识别各个对象，并且可以使用来自其他图像的可识别特征来提高对象识别的准确性。然而，值得注意的是，与物品储存和库存分析有关的历史数据可以用于改进原始置信度分数310，以促进改进库存管理。因此，下面描述的本发明的方面可以涉及用于改进原始置信度分数310(特别是当它 降到合适的置信度阈值以下时)的方法步骤。
在这点上，例如，步骤230可以包括确定原始置信度分数310降到高置信度阈值以下。该高置信度阈值可以被预先确定并编程到控制器156中作为阈值，超过该阈值，对象识别将导致冰箱库存记录或列表的变化。另外，步骤230可以包括确定原始置信度分数高于低置信度阈值，例如，使得对象182被识别，与该识别相关联的置信度分数没有适当地高到足以值得库存的改变。
步骤240通常包括至少部分地基于原始置信度分数310和历史库存数据来获得与对象相关联的经调节的置信度分数(例如，在图4中通常由附图标记312标识)。如本文所用的，术语“历史库存数据”等通常旨在指代与制冷电器100内的物品或食物对象的储存有关的任何数据或信息。根据示例性实施方式，该历史库存数据可以用于基于与制冷电器100相关联的历史实践来生成经调节的置信度分数312。尽管本文将描述示例性历史库存数据，但应当理解，所述的数据仅是示例性的，并且用于便于本发明的各方面的讨论，而不限制本发明的范围。
例如，根据示例性实施方式，历史库存数据可以包括与储存在各个空间区域(例如，在图4中通常由附图标记320标识)中的常见食品或对象182有关的信息。该历史库存数据可以被分解成对象类和子类。在这点上，例如，物品类可以包括蔬菜(例如，包括诸如豌豆、胡萝卜、玉米等的子类)、水果(例如，包括诸如浆果、切好的水果等的子类)、预制的餐食(例如，包括诸如烤宽面条、比萨、微波餐食等的子类)、甜点(例如，包括诸如冰淇淋、小玩意、糖果等的子类)、肉类(例如，包括诸如鸡肉、牛肉、猪肉等的子类)和其它类。
另外，历史库存数据可以包括与可能储存在制冷电器100内的各种对象182中的每一个相关联的位置。在这点上，如上所述，制冷间室122可以被分解为多个空间区域320，并且特定对象182被储存在相应空间区域320内的频率可以被记录在历史库存数据中。例如，如图4所示，被访问的空间区域320是制冷间室122的左下盒134。因此，历史库存数据可以包括经常放置在空间区域320内(例如，在左下盒134中)的对象的列表。该对象列表可以包括如上所述的对象分类、以及该对象被放置在特定空间区域320中的发生次数、这种放置的频率、或者与特定对象182将被放置在特定空间区域320内的可能性有关的任何其他数据。
根据示例性实施方式，经调节的置信度分数312可以由控制器156基于原始置信度分数310和历史库存数据来确定。例如，如果原始置信度分数310相对低，但历史库存数据指示所识别的对象非常可能储存在指定空间区域320中，则可相对于 原始置信度分数310增加经调节的置信度分数312。相反，如果对象被识别为具有相对低的原始置信度分数310，并且历史库存数据指示这样的对象通常不储存在空间区域320中，则可以相对于原始置信度分数310降低经调节的置信度分数312。
如上所述，步骤240可以包括识别制冷间室122内的添加或取出对象的空间区域320，确定对象的食物类型与储存在空间区域320中的常见食物类型相同，以及将原始置信度分数增加到经调节的置信度分数。例如，如图4示例，使用机器学习图像识别模型分析图像300的步骤220已经识别出水果放置在左下盒134内。具体地，该步骤分别识别具有37％置信度的苹果和具有36％和37％置信度的两个桔子。另外，历史库存数据指示苹果非常经常地储存在该空间区域320中，而桔子很少或从不储存在空间区域320中。因此，可以针对被识别为苹果的对象提升原始置信度分数310(例如，从37％提升到44％)，而可以降低与被识别为桔子的对象相关联的原始置信度分数310(例如，分别降低5％)。
根据示例性实施方式，步骤250可以包括确定经调节的置信度分数超过高置信度阈值。在这点上，步骤240通常可以用于提升在步骤220中识别的对象的置信度，并且由此可以促进改进库存管理。相反，如果经调节的置信度分数仍然低于进行库存调节所必需的置信度阈值，则方法200可以包括向制冷电器100的用户询问对象的身份，或者不采取进一步的动作和/或不对电器库存进行改变。
步骤260包括响应于确定经调节的置信度分数超过高置信度阈值而修改库存列表。在这点上，通过识别食品182是否被添加到食物保鲜室122或从其中取出，控制器156可以监测和跟踪制冷电器100内的库存。例如，控制器156可以保持放置在食物保鲜室122内或从其中取出的食品的记录。继续上面关于图4的示例，控制器156可以将一个苹果添加到电器库存列表中。另外，制冷电器100可以要求用户例如经由控制面板152或外部装置172输入被识别为桔子的两个对象的身份。
除了调节实际的电器库存之外，方法200还可以包括基于添加到制冷电器100或从其中取出的物品或对象182来更新历史库存数据。在这点上，例如，步骤270可以包括响应于确定经调节的置信度分数超过高置信度阈值而调节历史库存数据。由此，由于苹果更频繁地储存在左下盒134中，控制器156可被编程为对原始置信度分数310进行更大的调节以实现经调节的置信度分数312。由此，通过跟踪事件发生(例如，当某些对象放置在某些空间区域320内时)，历史库存数据可以被改进并且导致对置信度分数的更准确的调节以及改进的库存分析和管理。
根据示例性实施方式，方法200还可以包括确定原始置信度分数310超过高置 信度阈值。在这种情况下，可能不需要确定经调节的置信度分数312来对电器库存进行调节。因此，当原始置信度分数310这样高时，控制器156可以调节历史库存数据并且修改用于制冷电器100的库存列表。相反，方法200还可以包括确定不能以任何置信度水平识别对象。在这种情况下，方法200可包括不采取进一步的动作或例如经由控制面板152或外部装置172向用户询问对象的身份。
现在简要地参见图5，将描述根据本发明的示例性实施方式的可由制冷电器100实施的库存管理方法400的示例性流程图。根据示例性实施方式，方法400可以与方法200类似或互换，并且可以由制冷电器100的控制器156实施。如图所示，在步骤402，控制器156可以确定应拍摄图像，例如，通过确定冷藏门体128已经打开。步骤404可以包括获得一个或多个图像或视频，并且确定对象是否被添加到制冷间室122或从其中取出。
每次对象被添加到制冷间室122或从其中取出时，可以在步骤406执行图像的分析，例如，以识别对象和与这种对象识别有关的对应置信度分数。例如，该图像分析可以与以上关于步骤220描述的图像分析相同或类似。步骤408可以包括确定没有对象被识别为可从图像识别，并且步骤410可以包括向用户询问对象识别和/或不对库存或历史库存数据进行改变。
相反，步骤412可以包括识别对象，但是在低置信度水平，例如不足以值得电器库存或历史库存数据的改变的置信度水平。步骤414可以包括获得位置统计数据并尝试例如基于历史库存数据来调节与对象识别相关联的置信度分数。如果在步骤416，经调节的置信度分数仍然不高于高置信度阈值，则方法400可以进行到步骤410(如上所述)。相反，如果经调节的置信度分数现在超过高置信度阈值，则步骤418可以包括更新电器库存和/或调节历史库存数据。步骤420可以包括识别高置信度水平(例如，控制器156应在高于该置信度水平时更新电器库存列表和/或调节历史库存数据)的对象。步骤422可以包括例如通过添加位置、所识别的对象等来调节历史库存数据。
图3和图5描述了为了说明和讨论的目的而以特定顺序执行的步骤。使用本文所提供的发明内容，本领域普通技术人员将理解，本文所述的任意方法的步骤可以以各种方式改编、重新排列、扩展、省略或修改，而不脱离本发明的范围。而且，虽然使用制冷电器100作为示例解释了方法200和方法400的各方面，但是应当理解，该方法可以应用于任何合适的制冷电器或期望库存管理的任何其它电器的操作。
本发明的方面涉及用于使用相机组件来促进制冷电器中的库存管理的系统和方 法。相机组件可以包括安装到箱体、放置在门体的底部上、或以任何其它方式设置为记录一个或多个制冷间室内的活动的相机。电器控制器可以记录用于不同类型的物品的经常储存位置。这些历史统计可以用于例如通过基于储存位置调节视觉识别置信度来改进未来对象识别和库存管理。
例如，相机组件可以用于监测添加到制冷间室内的不同区域和/或从其中取出的物品。制冷间室可以被编程地分成多个空间区域，可以被分成用于各个水平的象限，或者可以被分成任何其他合适的虚拟区、空间或区域。当物品被放置在特定区域中并以预定的置信度水平分类时，物品类型或更一般的物品类的出现可被添加到该区域的历史中。关于常见物品放置的统计可能影响对象分类和库存改变的总体置信度。
例如，如果物品是储存位置的常见物品类型，则可以将低置信度库存改变提升到高置信度改变。当所识别的对象处于高置信度时，那么添加位置或者将对象出现添加到该位置的统计，并且基于该信息更新库存。当所识别的对象处于低置信度时，那么将获得位置统计或增加位置的常见物品的置信度值。在该过程之后，如果置信度水平高，则可以更新库存。相反，如果置信度水平低，则控制器可以询问用户以识别物品或不对库存进行改变。
本书面描述使用示例对本发明进行了公开(其中包括最佳实施例)，并且还使本领域技术人员能够实施本发明(其中包括制造和使用任意装置或系统并且执行所包含的任意方法)。本发明的可专利范围通过权利要求进行限定，并且可以包括本领域技术人员能够想到的其它的示例。如果这种其它的示例包括与权利要求的字面语言没有区别的结构元件，或者如果这种其它的示例包括与权利要求的字面语言没有实质区别的等同结构元件，则期望这种其它的示例落入权利要求的范围中。

Claims (20)

  1. 一种制冷电器，其特征在于，包括：
    箱体，所述箱体限定制冷间室；
    门体，所述门体可旋转地铰接到所述箱体，以提供选择性地到达所述制冷间室的途径；
    相机组件，所述相机组件用于监测所述制冷间室；以及
    控制器，所述控制器可操作地联接到所述相机组件，所述控制器被配置为：
    使用所述相机组件获得图像；
    分析所述图像以识别被添加到所述制冷间室或从所述制冷间室取出的对象以及与所述被识别的对象相关联的原始置信度分数；
    确定所述原始置信度分数降到高置信度阈值以下；
    至少部分地基于所述原始置信度分数和历史库存数据获得与所述对象相关联的经调节的置信度分数；
    确定所述经调节的置信度分数超过所述高置信度阈值；并且
    响应于确定所述经调节的置信度分数超过所述高置信度阈值而修改库存列表。
  2. 根据权利要求1所述的制冷电器，其特征在于，所述控制器被配置为使用机器学习图像识别过程来分析所述图像。
  3. 根据权利要求2所述的制冷电器，其特征在于，所述机器学习图像识别过程包括卷积神经网络、基于区域的卷积神经网络、深度信念网络或深度神经网络图像识别过程中的至少一个。
  4. 根据权利要求1所述的制冷电器，其特征在于，至少部分地基于所述原始置信度分数和所述历史库存数据来获得所述经调节的置信度分数包括：
    识别所述制冷间室内的添加或取出所述对象的空间区域；
    确定所述对象的食物类型与储存在所述空间区域中的常见食物类型相同；以及
    将所述原始置信度分数增加到所述经调节的置信度分数。
  5. 根据权利要求1所述的制冷电器，其特征在于，所述控制器还被配置为：
    响应于确定所述经调节的置信度分数超过所述高置信度阈值而调节所述历史库存数据。
  6. 根据权利要求5所述的制冷电器，其特征在于，调节所述历史库存数据包括 基于所述被识别的对象将事件发生添加到所述历史库存数据。
  7. 根据权利要求1所述的制冷电器，其特征在于，所述控制器还被配置为：
    确定所述原始置信度分数超过所述高置信度阈值；并且
    响应于确定所述原始置信度分数超过所述高置信度阈值而调节所述历史库存数据。
  8. 根据权利要求7所述的制冷电器，其特征在于，所述控制器还被配置为：
    响应于确定所述原始置信度分数超过所述高置信度阈值而修改所述库存列表。
  9. 根据权利要求1所述的制冷电器，其特征在于，所述控制器还被配置为：
    确定所述对象无法被识别；并且
    向所述制冷电器的用户询问以识别所述对象。
  10. 根据权利要求1所述的制冷电器，其特征在于，修改所述库存列表包括向所述库存列表添加所述对象或从其中去除所述对象。
  11. 根据权利要求1所述的制冷电器，其特征在于，所述历史库存数据包括与储存在所述制冷间室内的可能对象的食物类型、类别、子类和位置有关的数据。
  12. 根据权利要求1所述的制冷电器，其特征在于，所述制冷间室被分成多个空间区域，并且其中，所述历史库存数据包括经常放置在所述多个空间区域中的每一个中的对象的列表。
  13. 根据权利要求1所述的制冷电器，其特征在于，所述相机组件包括：
    相机，所述相机在所述制冷间室的前开口处安装至所述箱体，所述相机被定向成具有指向所述制冷间室中的视场。
  14. 根据权利要求1所述的制冷电器，其特征在于，所述相机组件包括：
    多个相机，所述多个相机设置在所述箱体内，所述多个相机中的每个相机具有指定的监测区或区域。
  15. 根据权利要求1所述的制冷电器，其特征在于，所述控制器还被配置为：
    确定所述制冷电器的所述门体是打开的；以及
    在所述门体打开时获得所述图像。
  16. 一种在制冷电器内实施库存管理的方法，其特征在于，所述制冷电器包括制冷间室和设置成用于监测所述制冷间室的相机组件，所述方法包括：
    使用所述相机组件获得图像；
    分析所述图像以识别被添加到所述制冷间室或从所述制冷间室取出的对象以及与所述被识别的对象相关联的原始置信度分数；
    确定所述原始置信度分数降到高置信度阈值以下；
    至少部分地基于所述原始置信度分数和历史库存数据获得与所述对象相关联的经调节的置信度分数；
    确定所述经调节的置信度分数超过所述高置信度阈值；以及
    响应于确定所述经调节的置信度分数超过所述高置信度阈值而修改库存列表。
  17. 根据权利要求16所述的方法，其特征在于，分析所述图像包括使用机器学习图像识别过程，机器学习图像识别过程包括卷积神经网络、基于区域的卷积神经网络、深度信念网络或深度神经网络图像识别过程中的至少一个。
  18. 根据权利要求16所述的方法，其特征在于，至少部分地基于所述原始置信度分数和所述历史库存数据来获得所述经调节的置信度分数包括：
    识别所述制冷间室内的添加或取出所述对象的空间区域；
    确定所述对象的食物类型与储存在所述空间区域中的常见食物类型相同；以及
    将所述原始置信度分数增加到所述经调节的置信度分数。
  19. 根据权利要求16所述的方法，其特征在于，还包括：
    确定所述原始置信度分数超过所述高置信度阈值；以及
    响应于确定所述原始置信度分数超过所述高置信度阈值而调节所述历史库存数据。
  20. 根据权利要求16所述的方法，其特征在于，还包括：
    确定所述对象无法被识别；以及
    向所述制冷电器的用户询问以识别所述对象。
PCT/CN2022/117153 2021-09-08 2022-09-06 制冷电器中的库存管理系统 WO2023036099A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202280060761.9A CN117999449A (zh) 2021-09-08 2022-09-06 制冷电器中的库存管理系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US17/469,376 2021-09-08
US17/469,376 US20230076984A1 (en) 2021-09-08 2021-09-08 Inventory management system in a refrigerator appliance

Publications (1)

Publication Number Publication Date
WO2023036099A1 true WO2023036099A1 (zh) 2023-03-16

Family

ID=85386178

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/117153 WO2023036099A1 (zh) 2021-09-08 2022-09-06 制冷电器中的库存管理系统

Country Status (3)

Country Link
US (1) US20230076984A1 (zh)
CN (1) CN117999449A (zh)
WO (1) WO2023036099A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130034295A1 (en) * 2011-08-02 2013-02-07 Toyota Motor Engineering & Manufacturing North America, Inc. Object category recognition methods and robots utilizing the same
CN104110936A (zh) * 2013-04-18 2014-10-22 Lg电子株式会社 冰箱及其工作方法
CN109146074A (zh) * 2017-06-28 2019-01-04 埃森哲环球解决方案有限公司 图像对象识别
CN109712324A (zh) * 2018-12-28 2019-05-03 广东便捷神科技股份有限公司 一种自动售货机图像识别方法、售货方法及自动售货设备
CN110046566A (zh) * 2019-04-10 2019-07-23 中南大学 一种基于图形识别及物联网技术的实验室智能物料柜
US10650265B1 (en) * 2019-05-29 2020-05-12 Capital One Services, Llc Systems and methods for enhancing real-time image recognition

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130034295A1 (en) * 2011-08-02 2013-02-07 Toyota Motor Engineering & Manufacturing North America, Inc. Object category recognition methods and robots utilizing the same
CN104110936A (zh) * 2013-04-18 2014-10-22 Lg电子株式会社 冰箱及其工作方法
CN109146074A (zh) * 2017-06-28 2019-01-04 埃森哲环球解决方案有限公司 图像对象识别
CN109712324A (zh) * 2018-12-28 2019-05-03 广东便捷神科技股份有限公司 一种自动售货机图像识别方法、售货方法及自动售货设备
CN110046566A (zh) * 2019-04-10 2019-07-23 中南大学 一种基于图形识别及物联网技术的实验室智能物料柜
US10650265B1 (en) * 2019-05-29 2020-05-12 Capital One Services, Llc Systems and methods for enhancing real-time image recognition

Also Published As

Publication number Publication date
US20230076984A1 (en) 2023-03-09
CN117999449A (zh) 2024-05-07

Similar Documents

Publication Publication Date Title
WO2022105718A1 (zh) 制冷电器的库存管理系统
US11335010B2 (en) Methods for viewing and tracking stored items
US20220325946A1 (en) Selective image capture using a plurality of cameras in a refrigerator appliance
WO2021057769A1 (zh) 用于查看并跟踪所储存的物品的方法
WO2023036099A1 (zh) 制冷电器中的库存管理系统
WO2021091481A1 (en) System for object identification and content quantity estimation through use of thermal and visible spectrum images
CN113124635B (zh) 冰箱
US20220414391A1 (en) Inventory management system in a refrigerator appliance
US20230308611A1 (en) Multi-camera vision system in a refrigerator appliance
WO2023046040A1 (zh) 制冷电器中的库存管理系统
WO2023151694A1 (zh) 具有智能门体警报的制冷电器
US11796250B1 (en) Multi-camera vision system facilitating detection of door position using audio data
WO2023131202A1 (zh) 带有智能抽屉的制冷电器
US20240056555A1 (en) Adjustable camera assembly in a refrigerator appliance
CN111488831B (zh) 一种食材联想识别方法及冰箱
CN107526991B (zh) 一种冷藏装置存储物管理系统和冷藏装置
CN115704622A (zh) 一种智能冰箱和食材存取动作识别方法
US20240068742A1 (en) Gasket leak detection in a refrigerator appliance
WO2022262683A1 (zh) 制冷电器中的库存管理系统
CN108139317A (zh) 存储结构的对象识别
WO2024022401A1 (zh) 用于制冷电器的智能可调节层架
US20230375258A1 (en) Refrigerator appliances and image-based methods of detecting a door position
KR20240051573A (ko) 냉장고 및 이의 제어방법
US11747071B2 (en) Systems and methods for detecting and monitoring ice formation within an ice maker
US20230057240A1 (en) Four camera system for a refrigerator appliance

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22866562

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280060761.9

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE