WO2022105718A1 - 制冷电器的库存管理系统 - Google Patents

制冷电器的库存管理系统 Download PDF

Info

Publication number
WO2022105718A1
WO2022105718A1 PCT/CN2021/130690 CN2021130690W WO2022105718A1 WO 2022105718 A1 WO2022105718 A1 WO 2022105718A1 CN 2021130690 W CN2021130690 W CN 2021130690W WO 2022105718 A1 WO2022105718 A1 WO 2022105718A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
refrigeration appliance
anchor
identify
camera assembly
Prior art date
Application number
PCT/CN2021/130690
Other languages
English (en)
French (fr)
Inventor
弗吉尼亚 莫里斯莎拉
古德曼 施罗德迈克尔
Original Assignee
海尔智家股份有限公司
青岛海尔电冰箱有限公司
海尔美国电器解决方案有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 海尔智家股份有限公司, 青岛海尔电冰箱有限公司, 海尔美国电器解决方案有限公司 filed Critical 海尔智家股份有限公司
Publication of WO2022105718A1 publication Critical patent/WO2022105718A1/zh

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • F25D29/005—Mounting of control devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2500/00—Problems to be solved
    • F25D2500/06—Stock management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • the present invention relates generally to refrigeration appliances, and more particularly to an inventory management system for refrigeration appliances.
  • Refrigeration appliances typically include a cabinet defining a refrigerated compartment for receiving food items for storage. Additionally, the refrigeration appliance includes one or more doors rotatably hinged to the cabinet to allow selective access to food items stored in the refrigeration compartment.
  • the refrigeration appliance may also include various storage components installed within the refrigeration compartment and designed to facilitate the storage of food products therein. Such storage components may include shelves, boxes, shelves or drawers that receive food products within the refrigerated compartment and assist in organizing and arranging such food products.
  • Some conventional refrigeration appliances have systems for monitoring food as it is added to or removed from the refrigeration appliance.
  • systems typically require user interaction, eg, via direct input through a control panel regarding food items added or removed.
  • some appliances include a camera for taking images of the food as it is being added.
  • analysis of these images is very computationally intensive, requires a lot of processing power and memory, and may inaccurately identify food items.
  • a refrigeration appliance with a system for improved inventory management would be useful. More particularly, an inventory management system that includes features for monitoring food items being added to or removed from refrigeration appliances while minimizing computer resources would be particularly beneficial.
  • a refrigeration appliance includes: a cabinet defining a refrigerated compartment; a door rotatably hinged to the cabinet to provide selective access to the refrigerated compartment; a camera assembly mounted to the enclosure for monitoring the refrigerated compartment; and a controller operably coupled to the camera assembly.
  • the controller is configured to obtain the raw image using the camera component, analyze the raw image to identify the anchored object, crop the raw image to generate a reduced image around the anchored object, and analyze the reduced image to identify addition to or removal from the refrigeration compartment food.
  • a method of implementing inventory management within a refrigeration appliance includes a refrigerated compartment and a camera assembly configured to monitor the refrigerated compartment.
  • the method includes: obtaining an original image using a camera assembly; analyzing the original image to identify anchored objects; cropping the original image to generate a reduced image around the anchored object; and analyzing the reduced image to identify food items added to or removed from a refrigerated compartment .
  • FIG. 1 provides a perspective view of a refrigeration appliance according to an exemplary embodiment of the present invention.
  • FIG. 2 provides a perspective view of the exemplary refrigeration appliance of FIG. 1 with a door of the food preservation compartment shown in an open position to expose a camera assembly, according to an exemplary embodiment of the present invention.
  • FIG. 3 provides a method for operating an inventory management system for a refrigeration appliance in accordance with an exemplary embodiment of the present invention.
  • FIG. 4 provides an image obtained by the exemplary camera assembly of FIG. 2 in accordance with an exemplary embodiment of the present invention.
  • FIG. 5 provides another image obtained by the exemplary camera assembly of FIG. 2 in accordance with an exemplary embodiment of the present invention.
  • upstream refers to where the fluid flows from
  • downstream refers to where the fluid flows.
  • in downstream refers to where the fluid flows from
  • downstream refers to where the fluid flows.
  • includes and “including” are intended to be inclusive in a manner similar to the term “comprising.”
  • the term “or” is generally intended to be inclusive (ie, "A or B” is intended to mean “A or B or both”).
  • Approximate language is applied to modify any quantitative representation that is permissible to vary without causing a change in its associated basic function.
  • a value modified by terms such as “about”, “approximately” and “approximately” is not limited to the precise value specified.
  • the language of approximation may correspond to the precision of the instrument used to measure the value. For example, approximate language may be referred to within a 10% margin.
  • FIG. 1 provides a perspective view of a refrigeration appliance 100 according to an exemplary embodiment of the present invention.
  • Refrigeration appliance 100 includes a box or housing 102 extending in vertical direction V between top 104 and bottom 106 and laterally L between first side 108 and second side 110 , and extends along the transverse direction T between the front side 112 and the rear side 114 .
  • Each of the vertical direction V, the lateral direction L, and the lateral direction T are perpendicular to each other.
  • the housing 102 defines a refrigerated compartment for receiving food items for storage.
  • the housing 102 defines a food preservation compartment 122 disposed at or adjacent to the top 104 of the housing 102 and a freezer compartment 124 disposed at or adjacent to the bottom 106 of the housing 102 .
  • the refrigeration appliance 100 is generally referred to as a bottom-mounted refrigerator.
  • the benefits of the present invention are applicable to other types and styles of refrigeration appliances, such as overhead refrigeration appliances, side-by-side refrigeration appliances, or single door refrigeration appliances.
  • examples of the present invention may also be applied to other appliances, such as other appliances that include fluid dispensers. Accordingly, the descriptions set forth herein are for purposes of example only, and are not intended to limit any particular appliance or configuration in any way.
  • the refrigerator door 128 is rotatably hinged to the edge of the housing 102 for selective access to the food preservation compartment 122 .
  • a freezing door 130 is arranged below the refrigerating door 128 so as to selectively enter the freezing compartment 124 .
  • the freezer door 130 is coupled to a freezer drawer (not shown) slidably mounted within the freezer compartment 124 .
  • the refrigerator door 128 and the freezer door 130 are shown in a closed configuration in FIG. 1 .
  • FIG. 1 Those skilled in the art will appreciate that other chamber and door configurations are possible and within the scope of the present invention.
  • FIG. 2 provides a perspective view of the refrigeration appliance 100 shown with the refrigeration door 128 in an open position.
  • various storage components are installed within the food preservation compartment 122 to facilitate storage of food products therein.
  • the storage components may include boxes 134 and shelves 136 . Each of these storage components is used to receive food products (eg, beverages or/or solid food products) and may assist in organizing such food products.
  • the box 134 may be mounted on the refrigerator door 128 or may be slid into a receiving space in the food preservation compartment 122 .
  • the storage components shown are for illustration purposes only and that other storage components may be used and may have different sizes, shapes, and configurations.
  • dispensing assembly 140 in accordance with an exemplary embodiment of the present invention will be described. While several different exemplary embodiments of the dispensing assembly 140 will be illustrated and described, like reference numerals may be used to refer to like components and features.
  • the dispensing assembly 140 is typically used to dispense liquid water and/or ice. While an exemplary dispensing assembly 140 is illustrated and described herein, it should be understood that various changes and modifications may be made to dispensing assembly 140 while remaining within the scope of the present invention.
  • the dispenser assembly 140 and its various components may be disposed at least partially within a dispenser recess 142 defined on one of the refrigerated door bodies 128 .
  • a dispenser recess 142 is defined on the front side 112 of the refrigeration appliance 100 so that a user can operate the dispensing assembly 140 without opening the refrigerator door 128 .
  • the dispenser recess 142 is provided at a predetermined height that is convenient for the user to take ice and enables the user to take the ice without bending over.
  • the dispenser recess 142 is positioned near the level of the user's chest.
  • the dispensing assembly 140 includes an ice dispenser 144 that includes a drain 146 for discharging ice from the dispensing assembly 140 .
  • An actuation mechanism 148 shown as a paddle, is mounted below the drain 146 for operating the ice or water dispenser 144 .
  • ice dispenser 144 may be operated using any suitable actuation mechanism.
  • ice dispenser 144 may include sensors (such as ultrasonic sensors) or buttons instead of paddles.
  • Drain 146 and actuation mechanism 148 are external parts of ice dispenser 144 and are mounted in dispenser recess 142 .
  • the refrigerated door 128 may define an ice bin compartment 150 (FIG. 2) that houses an ice maker and ice bin (not shown) configured to supply ice to a dispenser the device recess 142 .
  • a control panel 152 is provided to control the mode of operation.
  • the control panel 152 includes one or more selection inputs 154, such as knobs, buttons, touch screen interfaces, etc., such as a water dispensing button and an ice dispensing button, for selecting a desired mode of operation, such as crushed ice or non-crushed ice.
  • the input 154 may be used to specify the fill volume or method of operating the dispensing assembly 140 .
  • the input 154 may communicate with the processing device or controller 156 . Signals generated in controller 156 operate refrigeration appliance 100 and distribution assembly 140 in response to selector input 154 .
  • a display 158 may be provided on the control panel 152, such as an indicator light or screen. Display 158 may be in communication with controller 156 and may display information in response to signals from controller 156 .
  • processing device may refer to one or more microprocessors or semiconductor devices, and is not necessarily limited to a single element.
  • the processing device may be programmed to operate the refrigeration appliance 100, the distribution assembly 140, and other components of the refrigeration appliance 100.
  • a processing device may include or be associated with one or more storage elements (eg, persistent storage media).
  • the storage element includes an electrically erasable programmable read only memory (EEPROM).
  • EEPROM electrically erasable programmable read only memory
  • a storage element may store information accessible by a processing device, including instructions executable by the processing device.
  • the instructions may be software or any collection of instructions and/or data that, when executed by the processing apparatus, cause the processing apparatus to perform operations.
  • the refrigeration appliance 100 may also include a camera assembly 160 that is generally provided and used to obtain images of the refrigeration appliance 100 during operation.
  • the camera assembly 160 includes a camera 162 mounted to the top end 104 of the case 102 .
  • the camera 162 is installed such that it faces downward along the vertical direction V toward the refrigeration compartment of the refrigeration appliance 100 .
  • the camera 162 is generally oriented for monitoring the entrance to the freezer 122, eg, for monitoring items being added to or removed from the freezer, as described in more detail below.
  • camera 162 may be oriented in any other suitable manner for monitoring any other suitable area within or around refrigeration appliance 100 .
  • camera 162 may capture images or video of food preservation compartment 122 and the surrounding area. Specifically, the camera 162 may be activated when the refrigerator door 128 is opened to identify items added to or removed from the food preservation compartment 122 .
  • camera assembly 160 is illustrated as including a single camera 162 disposed over food preservation compartment 122 and used to monitor the food preservation compartment, it should be understood that camera assembly 160 may include any suitable number, type, size, and Camera 162 is configured to obtain images of any suitable area or area within or around refrigeration appliance 100 .
  • camera assembly 160 may include multiple cameras 162, each camera 162 configured to monitor a single refrigerated compartment or a portion of a refrigerated compartment (eg, fresh food compartment 122 and freezer compartment 124).
  • camera assembly 160 may include features for adjusting the field of view and/or orientation of camera 162 such that a single camera 162 may be adjusted to monitor food preservation compartment 122, freezer compartment 124, and/or other chambers simultaneously . It should be understood that the images obtained by the camera assembly 160 may vary in number, frequency, angle, resolution, detail, etc., in order to improve the clarity of specific areas around or within the refrigeration appliance 100 . Additionally, according to an exemplary embodiment, the controller 156 may be used to illuminate the refrigerated compartment with one or more light sources prior to obtaining an image.
  • controller 156 (or any other suitable dedicated controller) of refrigeration appliance 100 may be communicatively coupled to camera assembly 160 and may be programmed or configured to analyze images obtained by camera assembly 160, eg, In order to identify items to be added to or removed from the refrigeration appliance 100, as described in detail below.
  • an exemplary method 200 for operating the camera assembly 160 is provided.
  • the method 200 may be used to operate the camera assembly 160, or any other suitable camera assembly for monitoring appliance operation or inventory.
  • the controller 156 may be used to implement the method 200 .
  • the exemplary method 200 is discussed herein only to describe exemplary aspects of the invention, and is not intended to be limiting.
  • the method 200 includes: at step 210 , using a camera assembly to obtain an original image of a refrigeration compartment of a refrigeration appliance.
  • camera assembly 160 of refrigeration appliance 100 may obtain one or more raw images of food preservation compartment 122 , freezer compartment 124 , or any other zone or area within or around refrigeration appliance 100 (eg, 4 and 5 generally identified by reference numeral 170).
  • camera 162 is oriented downward from the top center of bin 102 and has a field of view that covers width 172 of food preservation compartment 122 (eg, as shown in the photographs of FIGS. 4 and 5 ).
  • the field of view may be centered on the opening 174 at the front of the bin 102 where, for example, the refrigerated door 128 is positioned against the front of the bin 102 .
  • the field of view of camera 162 and the resulting raw images obtained can capture any movement or movement of objects into and/or out of food preservation compartment 122 .
  • camera assembly 160 can capture one or more raw images 170, which can include one or more still images, one or more video clips, or are suitable for identifying food products (eg, by Reference numeral 176 generally identifies) or any other suitable type and number of images for inventory analysis.
  • the original image 170 may be obtained continuously or periodically while the refrigerated door 128 is open.
  • obtaining the raw image 170 may include determining that the door of the refrigeration appliance is open, and capturing the image at a set frame rate while the door is open.
  • the movement of the food product between the original image frames can be used to determine whether the food product 176 has been removed from or added to the food preservation compartment 122 .
  • the images obtained by the camera assembly 160 may vary in number, frequency, angle, resolution, detail, etc. in order to improve the clarity of the food product 176 .
  • the controller 156 may be used to illuminate refrigerator lights (not shown) while the raw image 170 is obtained.
  • the raw image 170 obtained at step 210 may be used to facilitate inventory management within the refrigeration appliance 100 .
  • the raw image 170 may be analyzed to identify food items 176 inserted into or removed from the food preservation compartment 122 .
  • analysis of larger raw images is typically more computationally intensive, eg, requiring more processing power and/or memory at the controller 156 .
  • analysis of areas remote from or otherwise not associated with food 176 may reduce computational efficiency, result in wasted computer resources, and may introduce errors or inaccuracies into image analysis. Accordingly, aspects of the present invention relate to methods of reducing the overall size of the original image 170 in order to improve object detection while reducing the use of computer resources.
  • Step 220 includes analyzing the original image to identify anchor objects.
  • the anchoring object 178 may be a human hand placing the food product 176 into the food preservation compartment 122 .
  • anchoring object 178 may be any object used to add food 176 to or remove food 176 from food preservation compartment 122 .
  • the anchor object 178 may be a package, a shopping bag, or any other suitable object in which the food product 176 is placed.
  • anchor objects 178 may be identified in raw image 170 using any suitable image analysis technique, image decomposition, image segmentation, image processing, or the like. This analysis may be performed entirely by controller 156, may be offloaded to a remote server for analysis, may be analyzed with user assistance (eg, via control panel 152), or may be analyzed in any other suitable manner. According to an exemplary embodiment of the present invention, the analysis performed at step 220 may include a machine learning image recognition process.
  • image recognition process As used herein, the terms image recognition process, anchored object detection, food detection, and similar terms may be used generically to refer to the observation of one or more images or videos captured by the camera assembly 160 in or around the refrigeration appliance 100, Any suitable method of analysis, image decomposition, feature extraction, image classification, etc.
  • the image recognition process may use any suitable artificial intelligence (AI) technique, eg, any suitable machine learning technique, or eg, any suitable deep learning technique.
  • AI artificial intelligence
  • any suitable image recognition software or process may be used to analyze images captured by camera assembly 160, and controller 156 may be programmed to perform such processes and implement inventory management processes.
  • the controller may implement a form of image recognition known as region-based convolutional neural network (“R-CNN”) image recognition.
  • R-CNN region-based convolutional neural network
  • an R-CNN may include taking an input image and extracting region proposals that include potential objects such as specific anchor objects 178 or foods 176 .
  • region suggestions may be regions in the image that may belong to particular objects, such as particular anchor objects 178 or food items 176 .
  • a convolutional neural network is then used to compute features from the region proposals, and the extracted features will then be used to determine the classification of each specific region.
  • the image segmentation process can be used with R-CNN image recognition.
  • image segmentation creates pixel-based masks for the various objects in the image and provides a more detailed or refined understanding of the various objects within a given image.
  • image segmentation may involve dividing the image into segments (eg, into groups of pixels containing similar properties), which Fragments can be analyzed independently or in parallel to obtain a more detailed representation of one or more objects in the image. This may be referred to herein as "Mask R-CNN" etc.
  • the image recognition process may use any other suitable neural network process.
  • step 220 may include using a masked R-CNN instead of a regular R-CNN architecture.
  • Mask R-CNN is based on Fast R-CNN which is slightly different from R-CNN.
  • R-CNN first applies the CNN and then assigns it to the region recommendation on the covn5 feature map, instead of initially segmenting into region recommendations.
  • standard CNNs may be used to identify anchor objects 178 or food items 176 according to an exemplary embodiment.
  • the K-means algorithm can be used.
  • Other image recognition processes are possible and within the scope of the present invention.
  • the step 220 of analyzing the raw image may include using a Deep Belief Network (“DBN”) image recognition process.
  • DBN Deep Belief Network
  • the DBN image recognition process can often involve stacking many individual unsupervised networks that use the hidden layers of each network as the input to the next layer.
  • step 220 may include implementing a deep neural network (“DNN”) image recognition process, which typically involves the use of neural networks (computing systems inspired by biological neural networks) with multiple layers between input and output.
  • DNN deep neural network
  • Other suitable image recognition processes, neural network processes, artificial intelligence (“AI”) analysis techniques, and combinations of the foregoing or other known methods may be used while remaining within the scope of the present invention.
  • Step 230 may include cropping the original image to generate a reduced image around the anchored object.
  • step 230 may include any suitable image reduction technique, segmentation technique, cropping technique, or object isolation technique for creating a reduced image (eg, as generally identified by reference numeral 180 in FIGS. 4 and 5 ) of).
  • the reduced image 180 can be of any suitable size that is smaller than the original image 170 and can be used for further analysis and food inspection, as will be described in more detail below.
  • exemplary techniques for cropping the original image 170 are described herein, it should be understood that these techniques are not intended to limit the scope of the present invention.
  • the step 230 of cropping the original image 170 to generate the reduced image 180 may include: identifying an anchor boundary of the anchored object; and cropping the original image to generate a Zoom out the image.
  • the anchor boundary (generally identified by reference numeral 182) may be a rectangular area that includes or closely surrounds the human hand, or may otherwise correspond to the boundary of the human hand.
  • an expanded region of interest (generally identified by arrow 184 as the space between anchor boundary 182 and reduced image 180 ) may be added to anchor boundary 182 to create reduced image 180 . Areas of the original image 170 outside the reduced image 180 may be deleted or otherwise taken.
  • the expanded region of interest 184 is intended to include the boundaries of the food product 176 held or set by the anchoring object 178 . As such, if the reduced image 180 includes only the anchor object 178 , the image analysis may not include identification of significant portions of the food product 176 . Therefore, the reduced image 180 may be larger than the anchor boundary 182 . It should be appreciated that various methods may be used to determine or identify the extended region of interest 184 while remaining within the scope of the present invention. For example, according to an exemplary embodiment, the expanded region of interest 184 is a fixed area or size increase to the anchor boundary 182. For example, the size of the anchor border 182 may be increased by a fixed percentage or size.
  • the width and/or depth of the anchor border 182 may be increased by 20%, 40%, 50%, 60% or more. This percentage increase may be based on, for example, the relative size of the anchoring object 178 to a typical food item 176 placed within the food compartment 122 .
  • the expanded region of interest 184 may be determined by identifying pixel characteristics associated with the anchored object 178 and/or the food product 176 .
  • the extended region of interest 184 may be determined by identifying the pixel properties of the anchored object 178 and the food product 176, and determining the extended region of interest 184 to include outside the anchoring boundary 182 having the same Anchors regions of object 178 and food 176 with similar pixel properties.
  • image analysis may be performed to determine the color, intensity, texture, or other visible characteristics of the anchored object 178 and the food product 176 , and the expanded region of interest 184 may be sized to include the inclusion of the original image 170 Pixels with similar features.
  • the expanded region of interest 184 may include pixels adjacent to the anchor boundary 182 that include green or substantially green pixels.
  • the expanded region of interest 184 may include red pixels. It should be appreciated that pixels included in the expanded region of interest 184 may be identified and isolated in any other suitable manner.
  • Step 240 may include analyzing the reduced image to identify food items added to or removed from the refrigeration compartment.
  • the controller 156 or another suitable processing device may analyze the reduced image 180 to identify the food product 176 .
  • Controller 156 may monitor and track inventory within refrigeration appliance 100 by identifying whether food product 176 has been added to or removed from food preservation compartment 122 .
  • controller 156 may maintain a record of food items placed in or removed from food preservation compartment 122 .
  • the image analysis performed at step 240 may include a machine learning image recognition process.
  • the machine learning image recognition process may include the same or similar image analysis techniques as described above with respect to step 220 . It is worth noting, however, that such image analysis is performed on the smaller reduced image 180 as opposed to the larger original image 170 . In this way, the food product 176 can be detected while minimizing or reducing the necessary processing power, computer memory or other computing resources.
  • FIG. 3 depicts an exemplary control method having steps performed in a particular order for purposes of illustration and discussion.
  • the steps of any method described herein may be adapted, rearranged, expanded, omitted, or modified in various ways without departing from the scope of the present invention.
  • aspects of the methods are described using the camera assembly 160 as an example, it should be understood that the methods may be applied to the operation of any suitable electrical and/or camera assembly.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Economics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Chemical & Material Sciences (AREA)
  • Mechanical Engineering (AREA)
  • General Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Quality & Reliability (AREA)
  • Human Resources & Organizations (AREA)
  • Combustion & Propulsion (AREA)
  • Tourism & Hospitality (AREA)
  • Thermal Sciences (AREA)
  • Operations Research (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

提供了一种制冷电器，包括：箱体，该箱体限定制冷间室；门体，该门体可旋转地铰接到箱体，以提供选择性进入制冷间室的途径；相机组件，该相机组件安装到箱体，用于监测制冷间室。控制器可操作地联接到相机组件，并且被配置为使用相机组件获得原始图像，分析原始图像以识别锚定对象，裁剪原始图像以生成锚定对象周围的缩小图像，并且分析缩小图像以识别被添加到制冷间室或从其取出的食品。

Description

制冷电器的库存管理系统 技术领域
本发明总体涉及制冷电器，更具体地涉及用于制冷电器的库存管理系统。
背景技术
制冷电器通常包括箱体，该箱体限定用于接收食品以便储存的制冷间室。另外，制冷电器包括一个或多个门体，这些门体可旋转地铰接到箱体，以允许选择性地接近制冷间室中储存的食品。制冷电器还可以包括安装在制冷间室内并且设计成便于在其中储存食品的各种储存部件。这种储存部件可以包括在制冷间室内接收食品并且辅助组织和布置这种食品的搁架、盒、层架或抽屉。
值得注意的是，经常期望监测制冷电器中的食品，了解什么食品被添加到制冷电器内或从制冷电器内取出、以及与食品的存在有关的其他信息。某些传统的制冷电器具有用于在食品被添加到制冷电器或从制冷电器取出时监测食品的系统。然而，这种系统通常需要用户交互，例如，经由通过控制面板进行的关于添加或取出的食品的直接输入。与之相比，某些电器包括用于在添加食品时拍摄食品的图像的相机。然而，这些图像的分析是非常计算密集的，需要大量的处理能力和存储器，并且可能不准确地识别食品。
因此，具有用于改进库存管理的系统的制冷电器将是有用的。更特别地，包括用于监测向制冷电器添加或从其取出的食品同时最小化计算机资源的特征的库存管理系统将是特别有益的。
发明内容
本发明的各个方面以及优点将会在下文的描述中进行阐述，或者是通过描述可以显而易见的，或者是可以通过实施本发明而学到。
在一个示例性实施方式中，一种制冷电器包括：箱体，该箱体限定制冷间室；门体，该门体可旋转地铰接到箱体，以提供选择性进入制冷间室的途径；相机组件，该相机组件安装到箱体，用于监测制冷间室；以及控制器，该控制器可操作地联接到相机组件。控制器被配置为使用相机组件获得原始图像，分析原始图像以识别锚定对象，裁剪原始图像以生成锚定对象周围的缩小图像，并且分析缩小图像以识别被添加到制冷间室或从其取出的食品。
在另一示例性实施方式中，提供了一种在制冷电器内实施库存管理的方法。该制冷电器包括制冷间室和设置成用于监测制冷间室的相机组件。方法包括：使用相机组件获得原始图像；分析原始图像以识别锚定对象；裁剪原始图像以生成锚定对象周围的缩小图像；以及分析缩小图像以识别被添加到制冷间室或从其取出的食品。
参照下文的描述以及所附权利要求，本发明的这些和其它的特征、方面以及优点将变得更容易理解。结合在本说明书中并且构成本说明书一部分的附图显示了本发明的实施方式并且与描述一起用于对本发明的原理进行解释。
附图说明
参照附图，说明书中阐述了面向本领域普通技术人员的本发明的完整公开，这种公开使得本领域普通技术人员能够实现本发明，包括本发明的最佳实施例。
图1提供了根据本发明的示例性实施方式的制冷电器的立体图。
图2提供了根据本发明的示例性实施方式的图1的示例性制冷电器的立体图，其中食物保鲜室的门体被示出为处于打开位置以露出相机组件。
图3提供了根据本发明的示例性实施方式的用于操作制冷电器的库存管理系统的方法。
图4提供了根据本发明的示例性实施方式的通过图2的示例性相机组件获得的图像。
图5提供了根据本发明的示例性实施方式的通过图2的示例性相机组件获得的另一图像。
附图标记在本说明书和附图中的重复使用旨在表示本发明的相同或相似的特征或元件。
具体实施方式
现在将详细地参照本发明的实施方式，其中的一个或多个示例示于附图中。每个示例都以对发明进行解释的方式给出，并不对本发明构成限制。实际上，对于本领域技术人员而言显而易见的是，能够在不偏离本发明的范围或者精神的前提下对本发明进行多种改型和变型。例如，作为一个实施方式的一部分示出或者进行描述的特征能够用于另一个实施方式，从而产生又一个实施方式。因此，期望的是，本发明覆盖落入所附权利要求及其等同形式的范围内的这些改型以及变型。
如本文所用的，术语“第一”、“第二”和“第三”可以互换使用以将一个部件与另一个部件区分开，并且这些术语并不旨在表示各个部件的位置或重要性。术语“上游”和“下游”是指相对于流体通路中的流体流动的相对方向。例如，“上游”是指流体流动的来向，而“下游”是指流体流动的去向。术语“包括(includes)”和“包括(including)”旨在以类似于术语“包括(comprising)”的方式为包括的。类似地，术语“或”通常旨在是包括的(即，“A或B”旨在意指“A或B或两者”)。
如本文在整个说明书和权利要求书中使用的近似语言被应用于修饰任何定量表示，该定量表示可容许在不导致其相关的基本功能改变的情况下变化。因此，由诸如“大约”、“近似”以及“大致”的术语修饰的值不限于所指定的精确值。在至少一些情况下，近似语言可对应于用于测量值的仪器的精度。例如，近似语言可以指在10％的裕度内。
图1提供了根据本发明的示例性实施方式的制冷电器100的立体图。制冷电器100包括箱体或壳体102，该箱体或壳体沿着竖向V在顶部104与底部106之间延伸，沿着侧向L在第一侧108与第二侧110之间延伸，并且沿着横向T在前侧112与后侧114之间延伸。竖向V、侧向L以及横向T中的每一个彼此互相垂直。
壳体102限定用于接收食品以便储存的制冷间室。特别地，壳体102限定设置在壳体102的顶部104处或与其相邻设置的食物保鲜室122和布置在壳体102的底部106处或与其相邻布置的冷冻室124。由此可见，制冷电器100通常被称为底置式冰箱。然而，认识到，本发明的益处适用于其他类型和样式的制冷电器，例如，顶置式制冷电器、对开门式制冷电器或单门制冷电器。而且，本发明的示例也可以适用于其他电器，诸如包括流体分配器的其他电器。因此，本文阐述的描述仅出于示例目的，而无意于在任何方面限制任何特定的电器或配置。
冷藏门体128可旋转地铰接到壳体102的边缘，以便选择性地进入食物保鲜室122。另外，在冷藏门体128的下方布置冷冻门体130，以便选择性地进入冷冻室124。冷冻门体130联接至可滑动地安装在冷冻室124内的冷冻抽屉(未示出)。冷藏门体128和冷冻门体130在图1中被示出为处于关闭构造。本领域技术人员将理解，其它腔室和门体构造是可行的，并且在本发明的范围内。
图2提供了在冷藏门体128处于打开位置的情况下示出的制冷电器100的立体图。如图2所示，如本领域技术人员将理解的，各种储存部件被安装在食物保鲜室122内，以促进食品在其中的储存。特别地，储存部件可以包括盒134和层架136。 这些储存部件中的每一个用于接收食品(例如，饮料或/或固体食品)，并且可以辅助组织这种食品。如图所示，盒134可以安装在冷藏门体128上或者可以滑入食物保鲜室122中的容纳空间中。应当理解，所示的储存部件仅用于说明的目的，并且可以使用其它储存部件，并且其它储存部件可以具有不同的尺寸、形状以及构造。
再次参见图1，将描述根据本发明的示例性实施方式的分配组件140。虽然将示例并描述分配组件140的几个不同的示例性实施方式，但类似的附图标记可用于指代类似的部件和特征。分配组件140通常用于分配液态水和/或冰。虽然在本文中示例并描述了示例性分配组件140，但应当理解，可以在保留在本发明的范围内的同时对分配组件140进行各种变更和修改。
分配组件140及其各种部件可以至少部分地设置在限定于冷藏门体128中的一个上的分配器凹部142内。在这点上，分配器凹部142限定在制冷电器100的前侧112上，使得用户可以在不打开冷藏门体128的情况下操作分配组件140。另外，分配器凹部142设置在预定高度处，该预定高度方便用户取冰，并且使得用户能够在不需要弯腰的情况下取冰。在示例性实施方式中，分配器凹部142设置在接近用户的胸部水平的位置处。
分配组件140包括冰分配器144，该分配器包括用于从分配组件140排出冰的排放口146。被示出为拨片的致动机构148安装在排放口146下方，以便操作冰或水分配器144。在可选示例性实施方式中，可以使用任意合适的致动机构来操作冰分配器144。例如，冰分配器144可以包括传感器(诸如超声传感器)或按钮，而不是拨片。排放口146和致动机构148是冰分配器144的外部零件，并且安装在分配器凹部142中。与之相比，冷藏门体128可以限定容纳制冰机和储冰盒(未示出)的冰盒室150(图2)，该制冰机和储冰盒被构造成将冰供应至分配器凹部142。
设置控制面板152，以便控制操作模式。例如，控制面板152包括一个或多个选择输入154，诸如旋钮、按钮、触摸屏界面等，诸如水分配按钮和冰分配按钮，用于选择期望的操作模式，诸如碎冰或非碎冰。另外，输入154可以用于指定填充容积或操作分配组件140的方法。在这点上，输入154可以与处理装置或控制器156通信。在控制器156中生成的信号响应于选择器输入154操作制冷电器100和分配组件140。另外，可以在控制面板152上设置显示器158，诸如指示灯或屏幕。显示器158可以与控制器156通信，并且可以响应于来自控制器156的信号而显示信息。
如本文中使用的，“处理装置”或“控制器”可以指一个或多个微处理器或半导体装置，并且不必限于单个元件。处理装置可以被编程为操作制冷电器100、分 配组件140以及制冷电器100的其他部件。处理装置可以包括一个或多个存储元件(例如，永久存储介质)或与其关联。在一些这种实施方式中，存储元件包括电可擦可编程只读存储器(EEPROM)。通常，存储元件可以存储处理装置可访问的信息，包括可以由处理装置执行的指令。可选地，指令可以是软件或指令和/或数据的任意集合，该软件或指令和/或数据的任意集合在由处理装置执行时，使得处理装置执行操作。
现在具体参照图2，制冷电器100还可以包括相机组件160，其通常设置并用于在运行期间获得制冷电器100的图像。具体地，根据所示例的实施方式，相机组件160包括安装到箱体102的顶端104的相机162。具体地，相机162安装成使得其沿着竖向V朝向制冷电器100的制冷间室面向下。如图示例，相机162通常被定向为用于监测食物保鲜室122的入口，例如，用于监测向食物保鲜室添加或从其取出的物品，如以下更详细描述的。根据另一些实施方式，相机162可以以任意其他合适的方式定向成用于监测制冷电器100内或周围的任意其他合适的区域。
通常，相机162可以拍摄食物保鲜室122和周围区域的图像或视频。具体地，相机162可以在打开冷藏门体128时启动，以识别添加到食物保鲜室122或从其中取出的物品。尽管相机组件160被示例为包括设置在食物保鲜室122上方并且用于监测食物保鲜室的单个相机162，但是应当理解，根据可选实施方式，相机组件160可以包括任意合适数量、类型、尺寸和配置的相机162，用于获得制冷电器100内或周围的任意合适的区或区域的图像。例如，相机组件160可以包括多个相机162，各个相机162被设置用于监测单个制冷间室或制冷间室的一部分(例如，食物保鲜室122和冷冻室124)。
根据另一些实施方式，相机组件160可以包括用于调节相机162的视场和/或取向的特征，使得单个相机162可以被调节为同时监测食物保鲜室122、冷冻室124和/或其他腔室。应当理解，由相机组件160获得的图像可以在数量、频率、角度、分辨率、细节等方面变化，以便提高制冷电器100周围或内的特定区域的清晰度。另外，根据示例性实施方式，控制器156可以用于在获得图像之前使用一个或多个光源照亮制冷间室。值得注意的是，制冷电器100的控制器156(或任意其他合适的专用控制器)可以通信地联接到相机组件160，并且可以被编程或配置用于分析由相机组件160获得的图像，例如，以便识别被添加到制冷电器100或从其取出的物品，如以下详细描述的。
既然已经呈现了根据本发明的示例性实施方式的制冷电器100和相机组件160 的结构和构造，则提供用于操作相机组件160的示例性方法200。方法200可用于操作相机组件160，或操作用于监测电器操作或库存的任意其它合适的相机组件。在这点上，例如，控制器156可以用于实施方法200。然而，应当理解，示例性方法200在本文仅讨论为描述本发明的示例性方面，而不旨在限制。
如图3所示，方法200包括：在步骤210，使用相机组件获得制冷电器的制冷间室的原始图像。在这点上，继续上述示例，制冷电器100的相机组件160可以获得食物保鲜室122、冷冻室124或者制冷电器100内或周围的任意其它区或区域的一个或更多个原始图像(例如，在图4和图5中一般由附图标记170标识)。具体地，根据示例性实施方式，相机162从箱体102的顶部中心向下定向，并且具有覆盖食物保鲜室122的宽度172的视场(例如，如图4和图5的照片所示)。而且，该视场可以以箱体102的前部处的开口174为中心，例如，在该开口处，抵靠箱体102的前部安置冷藏门体128。这样，相机162的视场以及所获得的结果原始图像可以捕获对象进入和/或离开食物保鲜室122的任意运动或移动。
在这点上，继续上述示例，相机组件160可以拍摄一个或多个原始图像170，这些原始图像可以包括一个或多个静止图像、一个或多个视频剪辑、或者适合于识别食品(例如，由附图标记176一般识别)或库存分析的任意其他合适类型和数量的图像。根据示例性实施方式，在冷藏门体128打开的同时，可以连续地或周期性地获得原始图像170。在这点上，获得原始图像170可以包括确定制冷电器的门体是打开的，并且在门体打开的同时以设定的帧率捕获图像。值得注意的是，食品在原始图像帧之间的运动可以用于确定食品176是否从食物保鲜室122中取出或添加到其中。应当理解，由相机组件160获得的图像可以在数量、频率、角度、分辨率、细节等方面变化，以便提高食品176的清晰度。另外，根据示例性实施方式，控制器156可以用于在获得原始图像170的同时照亮冰箱灯(未示出)。
值得注意的是，在步骤210获得的原始图像170可以用于促进制冷电器100内的库存管理。在这点上，可以分析原始图像170，以识别被插入到食物保鲜室122中或从其取出的食品176。值得注意的是，对较大原始图像的分析通常是更计算密集的，例如在控制器156处需要更多处理能力和/或存储器。而且，对远离食品176或以其他方式不与食品相关联的区域的分析可能降低计算效率，导致计算机资源的浪费，并且可能将误差或不准确性引入图像分析。因此，本发明的各方面涉及减小原始图像170的总尺寸以便于在减少使用计算机资源的情况下改进对象检测的方法。
步骤220包括分析原始图像以识别锚定对象。具体地，根据示例性实施方式， 锚定对象178可以是将食品176放置到食物保鲜室122中的人手。根据另一些实施方式，锚定对象178可以是用于向食物保鲜室122添加食品176或从其取出食品176的任意对象。在这点上，例如，锚定对象178可以是食品176放置在其中的包装、购物袋或任意其它合适的对象。
应当理解，可以使用任意合适的图像分析技术、图像分解、图像分割、图像处理等在原始图像170中识别锚定对象178。该分析可以完全由控制器156执行，可以卸载到远程服务器来分析，可以在用户辅助下分析(例如，经由控制面板152)，或者可以以任意其他合适的方式分析。根据本发明的示例性实施方式，在步骤220执行的分析可以包括机器学习图像识别过程。
如本文所用的，术语图像识别过程、锚定对象检测、食品检测和类似术语可以一般地用于指代由制冷电器100内或周围的相机组件160拍摄的一个或多个图像或视频的观察、分析、图像分解、特征提取、图像分类等的任意合适的方法。在这点上，图像识别过程可以使用任意合适的人工智能(AI)技术，例如，任意合适的机器学习技术，或者例如，任意合适的深度学习技术。应当理解，任意合适的图像识别软件或过程可以用于分析由相机组件160拍摄的图像，并且控制器156可以被编程为执行这样的过程并实施库存管理过程。
根据示例性实施方式，控制器可以实施被称为基于区域的卷积神经网络(“R-CNN”)图像识别的图像识别形式。一般而言，R-CNN可包括取得输入图像并提取包括诸如特定锚定对象178或食品176的潜在对象的区域建议。在这点上，“区域建议”可以是图像中可能属于特定对象(诸如特定锚定对象178或食品176)的区域。然后使用卷积神经网络来从区域建议计算特征，然后将使用所提取的特征来确定各个特定区域的分类。
根据另一些实施方式，可以将图像分割过程与R-CNN图像识别一起使用。通常，图像分割为图像中的各个对象创建基于像素的掩码，并且提供对给定图像内的各种对象的更详细或更精细的理解。在这点上，代替处理整个图像(即，像素的大集合，其中许多像素可能不包含有用信息)，图像分割可以涉及将图像划分为片段(例如，划分为包含类似属性的像素组)，这些片段可以独立地或并行地分析，以获得图像中的一个或多个对象的更详细表示。这在本文中可以被称为“掩码R-CNN”等。
根据另一些实施方式，图像识别过程可以使用任意其他合适的神经网络过程。例如，步骤220可以包括使用掩码R-CNN而不是常规的R-CNN架构。在这点上，掩 码R-CNN基于与R-CNN略微不同的快速R-CNN。例如，R-CNN首先应用CNN，然后将其分配给covn5特性图上的区域推荐，而不是初始地分割为区域推荐。另外，根据示例性实施方式，标准CNN可用于识别锚定对象178或食品176。另外，可以使用K均值算法。其它图像识别过程也是可能的，并且在本发明的范围内。
应当理解，可以使用任意其它合适的图像识别过程，同时保留在本发明的范围内。例如，分析原始图像的步骤220可以包括使用深度信念网络(“DBN”)图像识别过程。DBN图像识别过程通常可以包括堆叠许多单独的无监督网络，这些网络使用各个网络的隐藏层作为下一层的输入。根据另一些实施方式，步骤220可以包括实施深度神经网络(“DNN”)图像识别过程，其通常包括使用在输入与输出之间具有多个层的神经网络(由生物神经网络启示的计算系统)。可以使用其他合适的图像识别过程、神经网络过程、人工智能(“AI”)分析技术以及上述或其他已知方法的组合，同时保持在本发明的范围内。
步骤230可以包括裁剪原始图像以生成锚定对象周围的缩小图像。在这点上，例如，步骤230可以包括用于创建缩小图像的任意合适的图像缩小技术、分割技术、裁剪技术或对象隔离技术(例如，如图4和图5中的附图标记180一般标识的)。通常，缩小图像180可以是小于原始图像170并且可以用于进一步分析和食品检测的任意合适的尺寸，如以下将更详细地描述的。尽管本文描述了裁剪原始图像170的示例性技术，但应当理解，这些技术不旨在限制本发明的范围。
根据本发明的示例性实施方式，裁剪原始图像170以生成缩小图像180的步骤230可以包括：识别锚定对象的锚定边界；以及裁剪原始图像以生成包括锚定边界和扩展的感兴趣区域的缩小图像。在这点上，具体参照图4和图5，锚定边界(一般由附图标记182标识)可以是矩形区域，该矩形区域包括或紧密包围人手，或者可以以其他方式对应于人手的边界。在确定锚定边界182之后，可以将扩展的感兴趣区域(一般由箭头184标识为锚定边界182与缩小图像180之间的空间)添加到锚定边界182，以建立缩小图像180。原始图像170在缩小图像180之外的区域可以被删除或以其他方式取出。
通常，扩展的感兴趣区域184旨在包括由锚定对象178保持或设置的食品176的边界。这样，如果缩小图像180仅包括锚定对象178，则图像分析可以不包括食品176的重要部分的识别。因此，缩小图像180可以大于锚定边界182。应当理解，可以使用各种方法来确定或识别扩展的感兴趣区域184，同时保留在本发明的范围内。例如，根据示例性实施方式，扩展的感兴趣区域184是到锚定边界182的固定 面积或尺寸增加。例如，锚定边界182的尺寸可以增加固定的百分比或尺寸。在这点上，锚定边界182的宽度和/或深度可增加20％、40％、50％、60％或更多。该百分比增加可以基于例如锚定对象178与放置在食物室122内的典型食品176的相对尺寸。
根据另一些实施方式，扩展的感兴趣区域184可通过识别与锚定对象178和/或食品176相关联的像素特性来确定。在这点上，扩展的感兴趣区域184可以通过以下方式来确定：识别锚定对象178和食品176的像素特性，并且将扩展的感兴趣区域184确定为包括锚定边界182之外的具有与锚定对象178和食品176的像素特性类似的像素特性的区域。在这点上，例如，可执行图像分析以确定锚定对象178和食品176的颜色、强度、纹理或其他可见特征，并且可将扩展的感兴趣区域184的尺寸构造为包括原始图像170的包括类似特征的像素。例如，如果食品176是绿叶生菜，则扩展的感兴趣区域184可以包括与锚定边界182相邻的像素，这些像素包括绿色或大致绿色的像素。与之相比，如果食品176是一袋苹果，则扩展的感兴趣区域184可以包括红色像素。应当理解，可以以任意其他合适的方式识别和隔离包括在扩展的感兴趣区域184中的像素。
步骤240可以包括分析缩小图像以识别被添加到制冷间室或从其取出的食品。在这点上，继续上述示例，控制器156或另一合适的处理装置可以分析缩小的图像180以识别食品176。通过识别食品176是否被添加到食物保鲜室122或从其中取出，控制器156可以监测和跟踪制冷电器100内的库存。例如，控制器156可以保持放置在食物保鲜室122内或从其中取出的食品的记录。值得注意的是，应当理解，在步骤240执行的图像分析可以包括机器学习图像识别过程。例如，机器学习图像识别过程可以包括与以上关于步骤220描述的相同或类似的图像分析技术。然而，值得注意的是，与较大的原始图像170相反，对较小的缩小图像180执行这样的图像分析。这样，可以检测食品176，同时最小化或减少必要的处理能力、计算机存储器或其他计算资源。
图3描述了具有为了示例和讨论的目的而以特定顺序执行的步骤的示例性控制方法。使用本文所提供的发明内容，本领域普通技术人员将理解，本文所述的任意方法的步骤可以以各种方式改编、重新排列、扩展、省略或修改，而不脱离本发明的范围。而且，虽然使用相机组件160作为示例来说明了这些方法的各方面，但是应当理解，这些方法可以应用于任意合适的电器和/或相机组件的操作。
本书面描述使用示例对本发明进行了公开(其中包括最佳实施例)，并且还使本领域技术人员能够实施本发明(其中包括制造和使用任意装置或系统并且执行所 包含的任意方法)。本发明的可专利范围通过权利要求进行限定，并且可以包括本领域技术人员能够想到的其它的示例。如果这种其它的示例包括与权利要求的字面语言没有区别的结构元件，或者如果这种其它的示例包括与权利要求的字面语言没有实质区别的等同结构元件，则期望这种其它的示例落入权利要求的范围中。

Claims (20)

  1. 一种制冷电器，其特征在于，该制冷电器包括：
    箱体，该箱体限定制冷间室；
    门体，该门体可旋转地铰接到所述箱体，以提供选择性地进入所述制冷间室的途径；
    相机组件，该相机组件安装到所述箱体，用于监测所述制冷间室；以及
    控制器，该控制器可操作地联接到所述相机组件，所述控制器被配置为：
    使用所述相机组件获得原始图像；
    分析所述原始图像以识别锚定对象；
    裁剪所述原始图像以生成所述锚定对象周围的缩小图像；并且
    分析所述缩小图像以识别被添加到所述制冷间室或从其中取出的食品。
  2. 根据权利要求1所述的制冷电器，其特征在于，使用所述相机组件获得所述原始图像包括：
    确定所述制冷电器的所述门体是打开的；以及
    在所述门体打开的同时以设定帧率捕获图像。
  3. 根据权利要求1所述的制冷电器，其特征在于，分析所述原始图像以识别所述锚定对象包括：
    使用机器学习图像识别过程来分析所述原始图像以识别所述锚定对象。
  4. 根据权利要求3所述的制冷电器，其特征在于，所述机器学习图像识别过程包括卷积神经网络(“CNN”)、基于区域的卷积神经网络(“R-CNN”)、深度信念网络(“DBN”)或深度神经网络(“DNN”)图像识别过程中的至少一个。
  5. 根据权利要求1所述的制冷电器，其特征在于，分析所述缩小图像以识别所述食品包括：
    使用机器学习图像识别过程来分析所述缩小图像以识别所述食品。
  6. 根据权利要求5所述的制冷电器，其特征在于，所述机器学习图像识别过程包括卷积神经网络(“CNN”)、基于区域的卷积神经网络(“R-CNN”)、深度信念网络(“DBN”)或深度神经网络(“DNN”)图像识别过程中的至少一个。
  7. 根据权利要求1所述的制冷电器，其特征在于，所述锚定对象是人手。
  8. 根据权利要求1所述的制冷电器，其特征在于，裁剪所述原始图像以生成所述锚定对象周围的所述缩小图像包括：
    识别所述锚定对象的锚定边界；以及
    裁剪所述原始图像以生成包括所述锚定边界和扩展的感兴趣区域的所述缩小图像。
  9. 根据权利要求8所述的制冷电器，其特征在于，所述扩展的感兴趣区域是大于所述锚定边界的固定区域。
  10. 根据权利要求8所述的制冷电器，其特征在于，所述扩展的感兴趣区域通过以下方式确定：
    识别所述锚定对象和所述食品的像素特性；以及
    将所述扩展的感兴趣区域确定为包括所述锚定边界之外的具有与所述锚定对象和所述食品的所述像素特性类似的像素特性的区域。
  11. 根据权利要求1所述的制冷电器，其特征在于，所述相机组件包括：
    相机，该相机安装到所述箱体的顶部，所述相机沿着竖向向下定向。
  12. 根据权利要求1所述的制冷电器，其特征在于，所述相机组件包括：
    多个相机，该多个相机设置在所述箱体内，所述多个相机中的每个相机具有指定的监测区或区域。
  13. 根据权利要求1所述的制冷电器，其特征在于，所述控制器还被配置为：
    保持放置在所述制冷间室内或从其中取出的食品的记录。
  14. 一种在制冷电器内实施库存管理的方法，其特征在于，所述制冷电器包括制冷间室和设置成用于监测所述制冷间室的相机组件，所述方法包括：
    使用所述相机组件获得原始图像；
    分析所述原始图像以识别锚定对象；
    裁剪所述原始图像以生成所述锚定对象周围的缩小图像；以及
    分析所述缩小图像以识别被添加到所述制冷间室或从其中取出的食品。
  15. 根据权利要求14所述的方法，其特征在于，分析所述原始图像以识别所述锚定对象包括：
    使用机器学习图像识别过程来分析所述原始图像以识别所述锚定对象。
  16. 根据权利要求14所述的方法，其特征在于，分析所述缩小图像以识别所述食品包括：
    使用机器学习图像识别过程来分析所述缩小图像以识别所述食品。
  17. 根据权利要求14所述的方法，其特征在于，所述锚定对象是人手。
  18. 根据权利要求14所述的方法，其特征在于，裁剪所述原始图像以生成所述 锚定对象周围的所述缩小图像包括：
    识别所述锚定对象的锚定边界；以及
    裁剪所述原始图像以生成包括所述锚定边界和扩展的感兴趣区域的所述缩小图像。
  19. 根据权利要求18所述的方法，其特征在于，所述扩展的感兴趣区域是大于所述锚定边界的固定区域。
  20. 根据权利要求18所述的方法，其特征在于，所述扩展的感兴趣区域通过以下方式确定：
    识别所述锚定对象和所述食品的像素特性；以及
    将所述扩展的感兴趣区域确定为包括所述锚定边界之外的具有与所述锚定对象和所述食品的所述像素特性类似的像素特性的区域。
PCT/CN2021/130690 2020-11-17 2021-11-15 制冷电器的库存管理系统 WO2022105718A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/950,284 US11692769B2 (en) 2020-11-17 2020-11-17 Inventory management system for a refrigerator appliance
US16/950,284 2020-11-17

Publications (1)

Publication Number Publication Date
WO2022105718A1 true WO2022105718A1 (zh) 2022-05-27

Family

ID=81588385

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/130690 WO2022105718A1 (zh) 2020-11-17 2021-11-15 制冷电器的库存管理系统

Country Status (2)

Country Link
US (1) US11692769B2 (zh)
WO (1) WO2022105718A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115493343B (zh) * 2021-06-18 2023-10-27 海信集团控股股份有限公司 一种冰箱及其食材管理方法及装置
DE102022208678A1 (de) * 2022-08-22 2024-02-22 BSH Hausgeräte GmbH Verarbeiten von Bildern eines Haushaltsgeräts

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN208752722U (zh) * 2018-07-29 2019-04-16 上海图漾信息科技有限公司 自动售货柜
CN110617670A (zh) * 2019-08-27 2019-12-27 广州伊思高科技有限公司 一种基于图像识别的无人售冰箱及其识别方法
CN111503991A (zh) * 2020-04-15 2020-08-07 海信集团有限公司 一种识别冰箱食材存取位置的方法及冰箱
US20200327601A1 (en) * 2019-04-10 2020-10-15 Lg Electronics Inc. Refrigerator and method for managing products in refrigerator

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006317109A (ja) * 2005-05-13 2006-11-24 Toshiba Corp 冷蔵庫
JP6498866B2 (ja) * 2013-03-12 2019-04-10 東芝ライフスタイル株式会社 冷蔵庫、カメラ装置
JP6402353B2 (ja) * 2013-03-29 2018-10-10 パナソニックＩｐマネジメント株式会社 冷蔵庫及び冷蔵庫システム
DE102013211097A1 (de) * 2013-06-14 2014-12-18 BSH Bosch und Siemens Hausgeräte GmbH Kältegerät mit einem Kameramodul
CN105474087A (zh) * 2013-08-28 2016-04-06 株式会社东芝 储藏箱用摄像机装置及具备该装置的储藏箱
JP2015068525A (ja) 2013-09-27 2015-04-13 三菱電機株式会社 冷蔵庫
KR102437698B1 (ko) * 2015-08-11 2022-08-30 삼성전자주식회사 전자 장치 및 전자 장치의 이미지 인코딩 방법
JP6324360B2 (ja) * 2015-11-04 2018-05-16 三菱電機株式会社 冷蔵庫及びそれを含むネットワークシステム
KR102469567B1 (ko) * 2016-04-11 2022-11-22 삼성전자주식회사 이미징 장치 및 그 동작 방법
JP6938116B2 (ja) 2016-05-30 2021-09-22 シャープ株式会社 在庫管理装置および在庫管理方法
CN106529427A (zh) 2016-10-24 2017-03-22 合肥华凌股份有限公司 图像中目标物体识别方法、冰箱内食品识别方法及系统
JP6832155B2 (ja) * 2016-12-28 2021-02-24 ソニーセミコンダクタソリューションズ株式会社 画像処理装置、画像処理方法、及び画像処理システム
CN107024073A (zh) * 2017-04-26 2017-08-08 中国石油大学(华东) 基于深度学习的多传感器智能冰箱控制方法和智能冰箱
US10552707B2 (en) 2017-12-07 2020-02-04 Qualcomm Incorporated Methods and devices for image change detection
JP6991960B2 (ja) * 2018-12-28 2022-01-13 Kddi株式会社 画像認識装置、画像認識方法及びプログラム
CN110287759B (zh) * 2019-03-25 2022-08-02 广东工业大学 一种基于简化输入卷积神经网络o-cnn的眼部疲劳检测方法
WO2020222335A1 (ko) * 2019-05-02 2020-11-05 엘지전자 주식회사 인공 지능형 냉장 장치와 그 제어 방법

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN208752722U (zh) * 2018-07-29 2019-04-16 上海图漾信息科技有限公司 自动售货柜
US20200327601A1 (en) * 2019-04-10 2020-10-15 Lg Electronics Inc. Refrigerator and method for managing products in refrigerator
CN110617670A (zh) * 2019-08-27 2019-12-27 广州伊思高科技有限公司 一种基于图像识别的无人售冰箱及其识别方法
CN111503991A (zh) * 2020-04-15 2020-08-07 海信集团有限公司 一种识别冰箱食材存取位置的方法及冰箱

Also Published As

Publication number Publication date
US20220155007A1 (en) 2022-05-19
US11692769B2 (en) 2023-07-04

Similar Documents

Publication Publication Date Title
WO2022105718A1 (zh) 制冷电器的库存管理系统
US11499773B2 (en) Refrigerator and method for managing articles in refrigerator
US9989295B2 (en) Ice imaging system
US20220325946A1 (en) Selective image capture using a plurality of cameras in a refrigerator appliance
CN110966834A (zh) 识别冰箱内食材种类的方法与冰箱
WO2021091481A1 (en) System for object identification and content quantity estimation through use of thermal and visible spectrum images
US20220414391A1 (en) Inventory management system in a refrigerator appliance
WO2023036099A1 (zh) 制冷电器中的库存管理系统
WO2023185835A1 (zh) 制冷电器中的多相机视觉系统
CN110705371A (zh) 一种冰柜结构及其商品排面检测方法
US11796250B1 (en) Multi-camera vision system facilitating detection of door position using audio data
US20240056555A1 (en) Adjustable camera assembly in a refrigerator appliance
CN107526991B (zh) 一种冷藏装置存储物管理系统和冷藏装置
US20230097905A1 (en) Inventory management system in a refrigerator appliance
Lee et al. Smart refrigerator inventory management using convolutional neural networks
CN114299388A (zh) 物品信息确定方法、陈列柜以及存储介质
US11747071B2 (en) Systems and methods for detecting and monitoring ice formation within an ice maker
US11965691B2 (en) Refrigerator appliance with smart drawers
WO2022262683A1 (zh) 制冷电器中的库存管理系统
US20230057240A1 (en) Four camera system for a refrigerator appliance
US20230058922A1 (en) Appliance with collocated cameras
WO2024022401A1 (zh) 用于制冷电器的智能可调节层架
US20240167759A1 (en) Systems and methods for camera settings adjustments in refrigerator appliances
US20230375258A1 (en) Refrigerator appliances and image-based methods of detecting a door position
WO2023151694A1 (zh) 具有智能门体警报的制冷电器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21893865

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21893865

Country of ref document: EP

Kind code of ref document: A1