WO2021057820A1 - 制冷电器和用于跟踪所储存物品的方法 - Google Patents

制冷电器和用于跟踪所储存物品的方法 Download PDF

Info

Publication number
WO2021057820A1
WO2021057820A1 PCT/CN2020/117264 CN2020117264W WO2021057820A1 WO 2021057820 A1 WO2021057820 A1 WO 2021057820A1 CN 2020117264 W CN2020117264 W CN 2020117264W WO 2021057820 A1 WO2021057820 A1 WO 2021057820A1
Authority
WO
WIPO (PCT)
Prior art keywords
item
attempt
identification
refrigeration
purchase
Prior art date
Application number
PCT/CN2020/117264
Other languages
English (en)
French (fr)
Inventor
柳春宰
施罗德·迈克尔·古德曼
凯里亚库·斯蒂芬诺斯
Original Assignee
青岛海尔电冰箱有限公司
海尔智家股份有限公司
海尔美国电器解决方案有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 青岛海尔电冰箱有限公司, 海尔智家股份有限公司, 海尔美国电器解决方案有限公司 filed Critical 青岛海尔电冰箱有限公司
Publication of WO2021057820A1 publication Critical patent/WO2021057820A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/51—Housings

Definitions

  • the present invention generally relates to tracking items in storage enclosures such as refrigeration appliances.
  • Storage enclosures such as refrigeration appliances and food storage cabinets generally provide enclosed chambers for receiving multiple items or objects.
  • a refrigerating appliance usually includes a box that defines a refrigerating compartment. The user can place food or objects in the refrigeration room to prevent the food from decay. Thus, the usable life of perishable items or objects can be increased.
  • Some existing electrical appliances have tried to solve these problems by requiring the user to manually input the stored items.
  • Other electrical appliances have used various methods such as rulers to estimate or guess the quantity or identification of stored items.
  • this attempt is not complicated enough and prone to inaccuracy. If one or more items are left in a bag (e.g., a food bag) when stored in the appliance, these disadvantages may be magnified.
  • a refrigerating appliance having a feature for assisting users in tracking the contents of the refrigerating compartment of the refrigerating appliance will be useful.
  • a refrigeration appliance having a feature for assisting a user to accurately track the contents of the refrigerating compartment of the refrigerating appliance so as to establish an inventory of stored items arranged in the refrigerating compartment will be useful.
  • a refrigeration appliance may include a box defining a refrigerating compartment, a door rotatably hinged to the box, a camera module, and a controller.
  • the camera module can be mounted to the box.
  • the controller may be operably coupled to the camera module.
  • the controller can be configured to initiate operating routines.
  • the operation routine may include receiving a purchase invoice, starting the item identification sequence in the refrigeration appliance, generating an initial identification attempt of the first stored item based on the item identification sequence, and comparing the initial identification attempt of the first stored item with the purchase invoice, based on the first The initial identification attempt of the stored item is compared with the purchase invoice to generate a modified identification attempt of the first stored item, and the label of the first stored item is recorded in the inventory in the refrigerating appliance.
  • a method of operating a refrigeration appliance may include the following steps: receiving a purchase invoice; and initiating an item identification sequence in the refrigeration appliance.
  • the method may further include the steps of: generating an initial identification attempt of the first stored item based on the item identification sequence; and comparing the initial identification attempt of the first stored item with the purchase invoice.
  • the method may still further include the following steps: generating a modified identification attempt of the first storage item based on the comparison between the initial identification attempt of the first storage item and the purchase invoice; and recording the label of the first storage item in the inventory in the refrigeration appliance.
  • Fig. 1 provides a front view of a refrigeration appliance according to an exemplary embodiment of the present invention.
  • Figure 2 provides a front view of a refrigeration appliance according to an exemplary embodiment of the present invention, in which the refrigeration door is shown in an open position.
  • Fig. 3 provides a schematic diagram of a refrigeration appliance according to an exemplary embodiment of the present invention.
  • FIG. 4 provides a flowchart illustrating a method of operating a refrigeration appliance according to an exemplary embodiment of the present invention.
  • the present invention provides a method to assist in managing the inventory of food in a storage enclosure such as a refrigeration appliance or a food storage cabinet.
  • the method may include one or more steps for determining what items (ie, stored items) are added to or removed from the storage enclosure. This determination can be made using information about what items the user has purchased and information inferred by the system in the storage enclosure.
  • the user can confirm the system's inference.
  • FIG. 1 provides a front view of a refrigerating appliance 100 according to an exemplary embodiment of the present invention, wherein the refrigerating door 128 of the refrigerating appliance 100 is shown in a closed position.
  • 2 provides a front view of the refrigerating appliance 100, in which the refrigerating door 128 is shown in an open position to expose the food preservation compartment 122 of the refrigerating appliance 100.
  • the refrigerating appliance 100 includes a housing or box 120 that extends along the vertical direction V between the top 101 and the bottom 102.
  • the box body 120 defines a refrigerated compartment for receiving food for storage.
  • the box body 120 defines a food preservation compartment 122 arranged at or adjacent to the top 101 of the box 120, and a freezing compartment 124 arranged at or adjacent to the bottom 102 of the box 120.
  • the refrigerating appliance 100 is generally called a bottom-mounted refrigerator.
  • the benefits of the present invention are applicable to other types and styles of storage enclosures, such as overhead refrigeration appliances, side-by-side refrigeration appliances, or non-refrigerated food storage cabinet enclosures. Therefore, the description set forth herein is for illustrative purposes only, and is not intended to limit any specific storage enclosure or refrigerator cavity configuration in any respect.
  • the refrigerating door 128 is rotatably hinged to the edge of the box 120 to selectively enter the fresh food compartment 122.
  • a freezing door 130 is arranged below the refrigerating door 128 to selectively enter the freezing compartment 124.
  • the freezer door body 130 is coupled to a freezer drawer 142 (not shown) slidably installed in the freezer compartment 124. As described above, the refrigerating door body 128 and the freezing door body 130 are shown in the closed position in FIG. 1, and the refrigerating door body 128 is shown in the open position in FIG. 2.
  • the storage component includes a box 140 installed in the food preservation compartment 122, a drawer 142, and a shelf 144.
  • the box 140, the drawer 142, and the shelf 144 are configured to receive stored items (e.g., beverages or solid foods) and can help organize these foods.
  • the drawer 142 may receive fresh food (for example, vegetables, fruits, or cheese) and increase the service life of these fresh foods.
  • the refrigerating appliance 100 also includes features for assisting the user in identifying the food in the food preservation compartment 122 or the freezing compartment 124.
  • the user can use these features, for example, to view the food (ie, stored items) stored in the food preservation compartment 122 or the freezer compartment 124, or to create an inventory of such stored items. This feature is discussed in more detail below.
  • FIG. 3 provides a schematic diagram of the refrigeration appliance 100.
  • the refrigeration appliance 100 includes a controller 150 that is operatively coupled or communicated with components of a refrigeration system (not shown) of the refrigeration appliance 100, the refrigeration system being configured to cool the food preservation compartment 122 or the freezing compartment 124.
  • the components include a compressor 170, an evaporator fan 172, and a condenser fan 174.
  • the controller 150 can selectively operate such components in order to cool the food preservation compartment 122 or the freezing compartment 124.
  • the controller 150 also communicates with a thermostat (e.g., a thermocouple or a thermistor).
  • the thermostat may be provided in the food preservation compartment 122 or the freezing compartment 124 (FIG. 2).
  • the controller 150 may receive a signal corresponding to the temperature of the food preservation compartment 122 or the freezing compartment 124 from the thermostat.
  • the controller 150 may also include an internal timer for calculating the elapsed time period.
  • the controller 150 may include a memory and one or more microprocessors, CPUs, etc., such as general-purpose or special-purpose microprocessors, which are used to execute programming instructions or micro-control codes associated with the operation of the refrigeration appliance 100.
  • the memory may mean random access memory such as DRAM or read-only memory such as ROM or FLASH.
  • the processor executes permanent programming instructions stored in memory.
  • the instructions include a software package that is configured to operate the appliance 100, or execute an operating routine (for example, see the exemplary method 400 described below with reference to FIG. 4).
  • the memory may be a separate component from the processor, or may be included on a board within the processor.
  • the controller 150 can be constructed without using a microprocessor (for example, using a combination of discrete analog or digital logic circuits; such as switches, amplifiers, integrators, comparators, flip-flops, AND gates, etc.) To perform control functions instead of relying on software.
  • a microprocessor for example, using a combination of discrete analog or digital logic circuits; such as switches, amplifiers, integrators, comparators, flip-flops, AND gates, etc.
  • the controller 150 may be arranged in various positions in the entire refrigerating appliance 100. Input/output ("I/O") signals may be routed between the controller 150 and various operating components of the refrigeration appliance 100.
  • I/O Input/output
  • One or more components of the refrigeration appliance 100 may communicate with the controller 150 (for example, electrical communication) via one or more conductive signal lines or a shared communication bus. Additionally or alternatively, one or more components of the refrigeration appliance 100 may communicate with the controller 150 via one or more wireless signal bands (for example, wireless communication).
  • the refrigerating appliance 100 further includes a camera or a camera module 160.
  • the camera 160 may be any type of device suitable for capturing two-dimensional photos or images.
  • the camera 160 may be a video camera or a digital camera with an electronic image sensor [e.g., a charge coupled device (CCD) or CMOS sensor].
  • CCD charge coupled device
  • CMOS complementary metal-oxide-semiconductor
  • the camera 160 communicates with the controller 150 (for example, electrical communication or wireless communication) so that the controller 150 can receive a signal corresponding to the image captured by the camera 160 from the camera 160.
  • the camera 160 is installed on the refrigerating appliance 100.
  • the camera 160 is installed at the top (eg, adjacent to the top 101) inside the fresh food compartment 122.
  • the camera 160 may be fixed to or guided through the top wall of the inner container defining the fresh food compartment 122.
  • the camera 160 may point downward,
  • the camera is pointed at one or more refrigerated compartments (e.g., food preservation compartment 122- Figure 2).
  • the camera 160 can be pointed at any specific one or at least a part of a combination of the drawer 142 and the shelf 144 (FIG. 2).
  • the camera 160 can capture images of one of the drawers 142, all the drawers 142, one of the shelves 144, all the shelves 144, or any suitable combination thereof.
  • the camera 160 is exemplified as being installed in the food preservation compartment 122, it should be understood that additional or alternative embodiments include a camera assembly [e.g., a digital camera with an electronic image sensor [e.g., a charge coupled device (CCD) or CMOS sensor] ], the camera assembly is installed in another suitable part of the refrigerating appliance, such as the refrigerating door 128.
  • a camera assembly can be pointed to the outside or front of the refrigerating appliance 100, so that an image of the user or area positioned forward from the refrigerating appliance 100 can be captured (for example, when the corresponding door is in the closed position).
  • the refrigeration appliance 100 includes an integrated display 180.
  • the integrated display 180 may be installed on the refrigerating door 128 (FIG. 1) or any other suitable position on the refrigerating appliance 100.
  • the integrated display 180 communicates with the controller 150 so that the integrated display 180 can receive a signal corresponding to the image captured by the camera 160 from the controller 150.
  • the integrated display 180 may receive such a signal from the controller 150 and visually present an image to the user.
  • the integrated display 180 may include, for example, a liquid crystal display panel (LCD), a plasma display panel (PDP), or any other suitable mechanism (such as a projector) for displaying images.
  • the refrigeration appliance 100 may further include a communication component 184 that communicates with the controller 150.
  • the communication component 184 may be installed on any suitable part of the refrigerating appliance 100, such as the box 120 or the refrigerating door 128.
  • the communication components include speakers (eg, electrodynamic speakers, electrostatic speakers, planar magnetic speakers, piezoelectric speakers, etc.).
  • the speaker may be configured to generate sound waves using one or more electrical signals (e.g., digital sound signals received from the controller 150). In this way, the speaker can audibly convey information to the user.
  • the communication component includes a microphone (eg, dynamic microphone, ribbon microphone, fiber optic microphone, piezoelectric microphone, etc.).
  • the microphone may utilize one or more received sound waves (e.g., from a user) to generate one or more electrical signals (e.g., to be received by the controller 150).
  • the microphone can receive an audible prompt or command from the user that can be directed to the controller 150.
  • the refrigeration appliance 100 includes a network interface (not shown) that connects the refrigeration appliance 100 (for example, the controller 150) to the network 190 so that the refrigeration appliance 100 can transmit and receive information through the network 190 .
  • the network 190 may be any wired or wireless network, such as WAN, LAN, or HAN.
  • the refrigeration appliance 100 includes a user detection component 186.
  • the user detection component 186 includes one or more sensors that are configured to detect biometric characteristics or personalized indicia corresponding to a specific individual user.
  • the user detection component 186 may include a front-facing camera configured to recognize the user's face based on the captured two-dimensional image.
  • the user detection component 186 may include a fingerprint imaging sensor configured to visually detect a user's fingerprint.
  • the user detection component 186 may include a signal detection sensor configured to detect in the wireless communication frequency band (for example, the BLE frequency band using short-wavelength UHF radio waves in the ISM frequency band of 2.4 to 2.485 GHz) Device address.
  • the device address may be, for example, a programmed Bluetooth address of the mobile display 182.
  • the user detection component 186 can determine whether and when the mobile display 182 is close to the refrigeration appliance 100.
  • the camera 160 is used as part of the user detection component 186.
  • the user detection component 186 may be configured to recognize a user from a two-dimensional image captured at the camera 160.
  • the controller 150 is also configured to recognize one or more defining characteristics under the user's elbow, such as skin color, arm/hand size, jewelry, typical clothing, and so on. As understood, identification of such defined features can be achieved through edge matching, divide-and-conquer search, gray-scale matching, histogram of receptive field response, or another appropriate routine (e.g., based on one or more images captured from camera 160). Execute at the controller 150) to execute.
  • the camera 160 may capture one or more two-dimensional images (e.g., as a video feed or a series of continuous still images) that can be transmitted to the controller 150 (e.g., as a data signal) , As commonly understood.
  • the image capture sequence can be initiated by a predetermined user action, such as opening the refrigerating door 128, detecting movement in the refrigerated compartment (for example, the food preservation compartment 122), occupying the user detection component 186, or in the user Detect movement within the range of the component 186, verify user input at the communication component 184 or the integrated display 180, and so on.
  • one or more remote servers 192 are in operative communication with the controller 150.
  • the remote server 192 may be used to host the retailer's point of sale system.
  • the remote server 192 may be or include a retailer point-of-sale server that tracks, for example, the identifier and quantity of the purchased item, the time or date stamp of the purchased item, the price of the purchased item, the customer identifier (ie, purchase Customer's identifier) and so on.
  • the remote server 192 may be used to host the retailer's inventory management system.
  • the remote server 192 may include or be provided as a retailer inventory management server that, for example, tracks inventory data (eg, expiration data, location data, cost data, etc.) related to items offered for sale by the retailer. And additionally or alternatively, the remote server 192 may be used to host one or more information databases.
  • inventory data eg, expiration data, location data, cost data, etc.
  • the remote server 192 can be implemented using any suitable computing device.
  • the remote server 192 may include one or more processors and one or more storage devices (ie, memories).
  • the one or more processors may be any suitable processing device (eg, processor core, microprocessor, ASIC, FPGA, microcontroller, etc.), and may be one processor or multiple processors operably connected .
  • the storage device may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.
  • the storage device may store data and instructions executed by the processor to cause the remote server 192 to perform operations.
  • the instructions may be instructions for receiving/transmitting point-of-sale data signals, receiving/transmitting inventory management data signals, receiving/transmitting data signals related to stored items, and the like.
  • the storage device may also include data retrieved, manipulated, created, or stored by the processor, such as point-of-sale data, customer identification data, inventory data, expiration data, and the like.
  • the data can be stored in one or more databases.
  • One or more databases may be connected to the remote server 192 through a high-bandwidth LAN or WAN, or may also be connected to the remote server 192 through the network 502.
  • One or more databases can be partitioned so that they are located in multiple locations.
  • the remote server 192 includes a network interface so that the remote server 192 can connect to one or more networks (for example, the network 190) and communicate with one or more network nodes through the network.
  • the remote server 192 can exchange data with one or more nodes via the network 190.
  • the remote server 192 can exchange data with the controller 150.
  • the remote server 192 can also exchange data with any number of client devices via the network 190 (for example, the mobile display 182).
  • the camera 160 and the controller 150 are configured to capture multiple, continuous two-dimensional images (e.g., at a predetermined rate or pattern) as part of an image capture sequence.
  • Sequential images eg, previously captured images and recently captured images
  • the controller 150 may be compared at the controller 150. Based on the comparison, changes or differences between sequential images can be detected.
  • the image capturing sequence may be stopped or paused according to one or more end conditions, such as expiration of a predetermined time period, failure to detect further changes in sequential images, or closing of the refrigerated door 128.
  • the controller 150 may automatically try to identify items within the field of view of the camera 160 (eg, stored items such as food, or non-storage items such as user appendages, shelves, movable drawers, etc. Wait). As understood, identifying such items can be achieved by edge matching, divide-and-conquer search, grayscale matching, histogram of receptive field response, or another appropriate routine (e.g., based on one or more images captured from camera 160 in The controller 150 executes) to execute. The identification may include or attempt to estimate the identity (e.g., what type of food or object is identified) or quantity (e.g., the number, volume, or mass of discrete units) of the stored items.
  • stored items such as food, or non-storage items such as user appendages, shelves, movable drawers, etc. Wait.
  • identifying such items can be achieved by edge matching, divide-and-conquer search, grayscale matching, histogram of receptive field response, or another appropriate routine (e.g., based
  • an identification score can be assigned to each identified storage item.
  • the recognition score provides the probability or likelihood that the recognition or estimation is correct. Based on additional data or confirmation (e.g., received from one or more remote servers 192), the recognition score can be updated or improved.
  • FIG. 4 provides a method of operating the refrigerating appliance 100 (FIG. 1), which includes a camera 160, as described above.
  • the method 400 may be executed by the controller 150 (FIG. 3), for example.
  • the controller 150 may communicate with the camera 160, the integrated display 180 (FIG. 3), the mobile display 182 (FIG. 3), the communication component 184, the user detection component 186, or the remote server 192.
  • the controller 150 may send and receive signals to and from the camera 160, the integrated display 180, the mobile display 182, the communication component 184, the user detection component 186, or the remote server 192.
  • the controller 150 may also generally be operably coupled to other suitable components of the electric appliance 100 to facilitate the operation of the electric appliance 100.
  • the method according to the invention may allow improved detection of one or more items in, for example, refrigeration appliances. Additionally or alternatively, the detection may be performed while reducing the processing demand on one or more controllers, processors, or processing devices of the refrigeration appliance.
  • Figure 4 depicts the steps performed in a specific order for purposes of example and discussion. Using the content of the invention provided herein, those of ordinary skill in the art should understand that the steps of any method described herein can be modified, adapted, rearranged, omitted or expanded in various ways without departing from the scope of the present invention (except for Description in other ways).
  • a purchase invoice may include data related to one or more items purchased from a retailer, such as a sales receipt.
  • the purchase invoice may include the item identifier of the purchased item.
  • the item identifier may generally or specifically indicate the type or name of the purchased item (for example, the type of food).
  • the purchase invoice may include a unit count of the purchased item. The unit count of purchases can calibrate the volume, quality, or discrete share of the purchased item (for example, the quantity of a particular food item).
  • the purchase invoice may include a time or date stamp for the purchase date of the purchased item.
  • the purchase invoice may include data indicating the time or date when the item was purchased.
  • the purchase invoice may include customer identification data. Such customer identification data can be included when the specific phone number, card, etc. of the corresponding user have been stored in the retailer's point-of-sale system and associated with a specific purchase.
  • the purchase invoice may include the following data: Customer A purchased 3 apples on July 17, where "apple” is the item identifier, "3" is the unit count, and "user A” is the customer identification data , And "July 17" is a date stamp.
  • the purchase invoice is received directly from the retailer.
  • the purchase invoice can be transmitted from the retailer's point-of-sale server to the refrigeration appliance.
  • the retailer’s point of sale server may be prompted to transmit the purchase invoice to the specific refrigeration appliance when the sale is completed.
  • the retailer's point-of-sale server may be prompted to transmit the purchase invoice.
  • the sales receipt can be scanned directly by the camera of the refrigeration appliance, and one or more character recognition sequences can be activated to interpret the text of the sales receipt (for example, in the controller of the refrigeration appliance ). Thereby, the text of the sales receipt can be recognized and interpreted, so that the controller can record (e.g., temporarily) each purchase item detailed in the sales receipt.
  • Inventory data may include, for example, expiration data (e.g., estimated expiration date, emergency recall information, etc.) or location data (e.g., related to a manufacturer, supplier, or retailer).
  • expiration data e.g., estimated expiration date, emergency recall information, etc.
  • location data e.g., related to a manufacturer, supplier, or retailer.
  • the method 400 includes the following steps: initiating an item identification sequence in the refrigeration appliance.
  • the item identification sequence may include capturing or measuring the characteristics of one or more stored items in or near the refrigeration appliance.
  • the item identification sequence may include an image capture sequence (eg, the first image capture sequence) at the initiation of the camera module, as described above.
  • the first image capture sequence may include capturing multiple two-dimensional images (e.g., a first two-dimensional image, a subsequent second two-dimensional image, etc.), such as in a video source or a series of continuous still images (e.g., according to a predetermined rate or Conditional shooting or capture).
  • the two-dimensional image can be transmitted to the controller (for example, as a data signal).
  • the two-dimensional image can then be recorded (e.g., temporarily) for comparison or evaluation.
  • 420 depends on identifying a specific user. Before identifying a specific user, it is possible to prevent the recording or evaluation of two-dimensional images from the camera module.
  • the identification of a specific user may include receiving a user detection signal indicating a biometric feature or a personalized mark corresponding to a specific individual user, as described above.
  • the user detection signal may include a captured two-dimensional image of a specific user's face. From the captured two-dimensional image, the user can be identified.
  • the user detection signal may include a fingerprint image from which the fingerprint of a specific user can be identified.
  • the user detection signal may include the device address of the detected mobile display of the specific user (for example, detection on a wireless communication frequency band, such as the BLE frequency band using short-wavelength UHF radio waves in the ISM frequency band of 2.4 to 2.485 GHz) .
  • a wireless communication frequency band such as the BLE frequency band using short-wavelength UHF radio waves in the ISM frequency band of 2.4 to 2.485 GHz
  • 420 is activated in response to detecting movement (for example, at a camera module in a refrigeration appliance).
  • movement for example, at a camera module in a refrigeration appliance.
  • a camera module pointing to the refrigerated compartment can be used to detect movement.
  • changes in light or pixels captured by the camera module can be detected, which indicates one or more of the camera module’s field of view. The movement of objects.
  • the item identification sequence (e.g., the first image capture sequence) may continue until one or more end conditions are met.
  • the end condition may include expiration of a predetermined time period (for example, a time span) after the corresponding movement detection is started.
  • the end condition may include failure to detect further changes in the sequential images of the first image capture sequence.
  • the first image capture sequence may end after the sequential images stop changing or further movement is detected.
  • the end condition may specifically require a preset number or time period of unaltered sequential images.
  • the end condition may include detecting the closing of the door of the refrigerated compartment. In other words, the first image capturing sequence may end in response to the door being moved to the closed position.
  • the method 400 includes the steps of: generating an initial recognition attempt of the first stored item based on the item recognition sequence at 420.
  • the attempt may include an attempt to identify the first storage item based on the first two-dimensional image of the first image capture sequence.
  • the first storage item e.g., food
  • the attempt to identify may require the first stored item to pass under or within the field of view of the camera module.
  • the first two-dimensional image may include at least the captured portion of the first storage item.
  • the identification of the first stored item can be performed by edge matching, divide-and-conquer search, gray-scale matching, histogram of receptive field response, or another suitable routine.
  • the initial identification attempt may include the estimated identifier of the first stored item.
  • the estimated identifier may generally or specifically indicate the type or name of the first storage item (for example, the type of food). In other words, the initial identification attempt may attempt to name what type of food was identified at 430.
  • the initial identification attempt may include an estimated unit count of the first stored item. The estimated unit count can calibrate the volume, mass, or discrete share of the purchased item (for example, the quantity of a particular food item). In other words, the initial identification attempt may attempt to measure or count how many foods were identified at 430.
  • the initial identification attempt may include multiple possible objects corresponding to the first stored item.
  • a recognition score can be assigned to the initial recognition attempt.
  • the recognition score may correspond to the probability (e.g., percentage) of correctly identifying the first stored item.
  • the recognition score may assign a value to the likelihood that the first stored item is a specific identifiable object.
  • the initial recognition can assign multiple discrete recognition scores to the first item—each recognition score corresponds to a different identifiable object.
  • the recognition score of object I can be assigned to 90% to indicate that the probability that the first stored item is identifiable object I is 90%
  • the recognition score of object II can be assigned to 70% to indicate the first stored item The probability of being an identifiable object II is 70%, and so on.
  • the method 400 includes the step of comparing the initial identification attempt of the first stored item with the purchase invoice.
  • the comparison may seek to correlate or link the initial identification attempt to the item in the purchase invoice.
  • the comparison may match one or more possible objects (e.g., estimated identifier, estimated unit count, etc.) from 430 to one or more entries in the purchase invoice (e.g., purchased item identifier, purchased item identifier, etc.). Purchase unit count, etc.).
  • 440 includes a purchase invoice selected for comparison based on date stamp or customer identification data.
  • a purchase invoice can be selected from multiple invoices based on the date stamp.
  • the date stamp of the purchase invoice is selected based on the most recent purchase invoice.
  • the purchase invoice may be selected from multiple invoices based on customer identification data.
  • the customer data is matched to a specific user that has been previously identified by the appliance.
  • the method 400 includes the steps of generating a modified recognition attempt of the first stored item based on the comparison at 440.
  • the initial recognition attempt can be updated based on the comparison. For example, if a recognition score is generated at 430, the score may be changed based on whether one or more identifiable objects are matched in the purchase invoice. If an identifiable object from 440 is matched (for example, found) in the purchase invoice, the recognition score can be increased, which indicates that the first stored item is more likely to be the estimated identifier or the estimated unit count is more likely to be correct .
  • the recognition score can be reduced, which indicates that the first stored item is less likely to be the estimated identifier or the estimated unit count It is more unlikely to be correct. If the initial identification attempt includes multiple possible objects, 440 may include discarding or downgrading any possible objects that do not match the corresponding entries in the purchase invoice.
  • the method 400 includes the following steps: requesting user confirmation of the modification identification attempt.
  • the appliance may generate a visual or audible query (e.g., via a communication component) that asks the user whether the identification attempt (e.g., the object with the highest recognition score) is correct.
  • the user can confirm or reject the modification recognition attempt, such as via physical input or voice control, as understood.
  • the request for user confirmation is only initiated when it is determined that the recognition score is below a predetermined threshold (eg, 50%). Therefore, in this embodiment, when it is determined that the recognition score is higher than the predetermined threshold, no request for user confirmation is generated.
  • a predetermined threshold eg, 50%
  • the method 400 includes the following steps: recording the label of the first storage item into the inventory in the refrigerating appliance.
  • the tag can be selected based on the modified recognition attempt (or its confirmation).
  • the label provides text or images related to the first stored item (e.g., as identified by modified identification attempts).
  • the tag may include the estimated identifier or the estimated unit count.
  • the tag may include inventory data (e.g., received from an inventory management server), such as expired data or location data.
  • the record tag may prompt an external search for information related to one or more purchased items.
  • the search can be performed via the Internet or the World Wide Web (for example, using a known search engine).
  • the search can be limited to one or more predetermined URLs, websites, or servers.
  • a search can represent a request for information about stored items, such as descriptive information, processing information, or the way the first item appears.
  • the user can selectively call up or view the inventory (including the label of the first stored item) on the integrated display or mobile display to check which items are currently stored in the appliance.
  • the electrical appliance may update the inventory, such as automatically or in response to manual user input, to remove the tag from the inventory. Additionally or alternatively, the appliance may automatically determine when the first stored item expires (e.g., based on inventory data), and generate an alert (e.g., audio or visual alert message) at the integrated display, mobile display, or communication component.
  • an alert e.g., audio or visual alert message

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Signal Processing (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

一种制冷电器（100），其可以包括限定制冷间室的箱体（120）、可旋转地铰接到箱体（120）的门体（128）、相机模块（160）以及控制器（150），相机模块（160）可以安装到箱体（120），控制器（150）可操作地联接到相机模块（160），该控制器（150）可以被配置为启动操作例程，操作例程可以包括接收购买发票（410），启动制冷电器（100）内的物品识别序列（420），基于物品识别序列生成第一储存物品的初始识别尝试（430），将第一储存物品的初始识别尝试与购买发票进行比较（440），基于第一储存物品的初始识别尝试与购买发票的比较生成第一储存物品的修改识别尝试（450），以及将第一储存物品的标签记录到制冷电器内的库存（460）。

Description

制冷电器和用于跟踪所储存物品的方法 技术领域
本发明总体涉及跟踪诸如制冷电器的储存外壳内的物品。
背景技术
诸如制冷电器和食品储藏柜的储存外壳通常提供用于接收多个物品或物体的封闭腔室。例如，制冷电器通常包括限定制冷间室的箱体。用户可以将食品或物体放置在制冷间室内，以便阻止这种食品的腐烂。从而，可以增加易腐物品或物体的可用寿命。
随着时间的推移，大量的储存物品(例如，食品)可能累积在冰箱的制冷间室内。随着所储存的物品的累积，制冷电器的用户可能难以识别位于制冷电器内的物品。另外，用户可能难以确定制冷电器内的某些物品的数量。当多个用户在不与其他用户通信的情况下向公共制冷电器添加/从其移除物品时，尤其如此。因此，用户可能意外地购买了过多或不需要的物品。例如，某些食品在制冷间室内不容易腐烂，并且这种食品可能不经常消耗。由此，这种食品可以在制冷间室内保持延长的时间段。用户可能忘记这种食品并购买替代品，尽管已经具有可接受的物品。这样，可能给用户造成不便或者不必要地花钱。另外或可选地，一些用户可能不知道某些物品已经被取出或消耗。由此，用户可能无法更换或补充这种物品。
一些现有电器已经尝试通过要求用户手动输入被储存的各个物品来解决这些问题。其它电器已经使用诸如标尺的各种方法来估计或猜测所储存的物品的数量或标识。然而，这种尝试不够繁复并且易于不准确。如果一个或多个物品在储存在电器内时被留在袋(例如，食品袋)中，则这些缺点可能被放大。
因此，具有用于辅助用户跟踪制冷电器的制冷间室的内容物的特征的制冷电器将是有用的。特别地，具有用于辅助用户准确跟踪制冷电器的制冷间室的内容物以便建立设置在制冷间室内的储存物品的库存的特征的制冷电器将是有用的。
发明内容
本发明的各个方面以及优点将会在下文的描述中进行阐述，或者是通过描述可以显而易见的，或者是可以通过实施本发明而学到。
在本发明的一个示例性方面，提供了一种制冷电器。该制冷电器可以包括限定 制冷间室的箱体、可旋转地铰接到箱体的门体、相机模块以及控制器。相机模块可以安装到箱体。控制器可以可操作地联接到相机模块。该控制器可以被配置为启动操作例程。操作例程可以包括接收购买发票，启动制冷电器内的物品识别序列，基于物品识别序列生成第一储存物品的初始识别尝试，将第一储存物品的初始识别尝试与购买发票进行比较，基于第一储存物品的初始识别尝试与购买发票的比较生成第一储存物品的修改识别尝试，以及将第一储存物品的标签记录到制冷电器内的库存。
在本发明的另一个示例性方面中，提供了一种操作制冷电器的方法。该方法可以包括以下步骤：接收购买发票；以及启动制冷电器内的物品识别序列。方法还可以包括以下步骤：基于物品识别序列生成第一储存物品的初始识别尝试；以及将第一储存物品的初始识别尝试与购买发票进行比较。方法仍还可以包括以下步骤：基于第一储存物品的初始识别尝试与购买发票的比较生成第一储存物品的修改识别尝试；以及将第一储存物品的标签记录到制冷电器内的库存。
参照下文的描述以及所附权利要求，本发明的这些和其它的特征、方面以及优点将变得更容易理解。结合在本说明书中并且构成本说明书一部分的附图显示了本发明的实施方式并且与描述一起用于对本发明的原理进行解释。
附图说明
参照附图，说明书中阐述了面向本领域普通技术人员的本发明的完整公开，这种公开使得本领域普通技术人员能够实现本发明，包括本发明的最佳实施例。
图1提供了根据本发明的示例性实施方式的制冷电器的正视图。
图2提供了根据本发明的示例性实施方式的制冷电器的正视图，其中，冷藏门体被示出为处于打开位置。
图3提供了根据本发明的示例性实施方式的制冷电器的示意图。
图4提供了例示了根据本发明的示例性实施方式的操作制冷电器的方法的流程图。
具体实施方式
现在将详细地参照本发明的实施方式，其中的一个或多个示例示于附图中。每个示例都以对发明进行解释的方式给出，并不对本发明构成限制。实际上，对于本领域技术人员而言显而易见的是，能够在不偏离本发明的范围的前提下对本发明进 行多种改型和变型。例如，作为一个实施方式的一部分示出或者进行描述的特征能够用于另一个实施方式，从而产生又一个实施方式。因此，期望的是，本发明覆盖落入所附权利要求及其等同形式的范围内的这些改型以及变型。
如本文所用的，术语“或”通常旨在是包括的(即，“A或B”旨在意指“A或B或两者”)。术语“第一”、“第二”和“第三”可以互换使用以将一个部件与另一个部件区分开，并且这些术语并不旨在表示各个部件的位置或重要性。
通常，本发明提供了辅助管理诸如制冷电器或食品储藏柜的储存外壳内的食品的库存的方法。方法可以包括用于确定什么物品(即，所储存物品)被添加到储存外壳或从其取出的一个或多个步骤。该确定可以使用与用户已经购买什么物品有关的信息以及由储存外壳内的系统推断的信息来进行。可选地，用户可以确认系统的推断。
现在转向附图，图1提供了根据本发明的示例性实施方式的制冷电器100的正视图，其中制冷电器100的冷藏门体128被示出为处于关闭位置。图2提供了制冷电器100的正视图，其中冷藏门体128被示出为处于打开位置，以露出制冷电器100的食物保鲜室122。
制冷电器100包括壳体或箱体120，该壳体或箱体沿着竖向V在顶部101与底部102之间延伸。箱体120限定用于接收食品以便储存的制冷间室。特别地，箱体120限定设置在箱体120的顶部101处或与其相邻设置的食物保鲜室122，和布置在箱体120的底部102处或与其相邻布置的冷冻室124。由此可见，制冷电器100通常被称为底置式冰箱。然而，认识到，本发明的益处适用于其他类型和样式的储存外壳，诸如顶置式制冷电器、对开门式制冷电器或非冷藏食品储藏柜外壳。因此，本文阐述的描述仅出于例示性目的，而无意于在任何方面限制任何特定的储存外壳或冰箱腔室构造。
冷藏门体128可旋转地铰接到箱体120的边缘，以便选择性地进入食物保鲜室122。另外，在冷藏门体128的下方布置冷冻门体130，以便选择性地进入冷冻室124。冷冻门体130联接至可滑动地安装在冷冻室124内的冷冻抽屉142(未示出)。如上所述，冷藏门体128和冷冻门体130在图1中被示出为处于关闭位置，并且冷藏门体128在图2中被示出为处于打开位置。
现在转向图2，如本领域技术人员将理解的，各种储存部件被安装在食物保鲜室122内，以促进食品在其中的储存。特别地，储存部件包括安装在食物保鲜室122内的盒140、抽屉142以及搁物架144。盒140、抽屉142以及搁物架144被构造为 接收所储存物品(例如，饮料或固体食品)，并且可以帮助整理这些食品。作为示例，抽屉142可以接收新鲜食品(例如，蔬菜、水果或奶酪)，并且增加这些新鲜食品的使用寿命。
制冷电器100还包括用于辅助用户识别位于食物保鲜室122或冷冻室124内的食品的特征。用户可以利用这些特征，例如，来查看储存在食物保鲜室122或冷冻室124内的食品(即，所储存物品)，或创建这种所储存物品的库存。下面更详细地讨论这种特征。
图3提供了制冷电器100的示意图。制冷电器100包括控制器150，该控制器与制冷电器100的制冷系统(未示出)的部件可操作地联接或通信，该制冷系统被配置为冷却食物保鲜室122或冷冻室124。部件包括压缩机170、蒸发器风扇172以及冷凝器风扇174。控制器150可以选择性地操作这种部件，以便冷却食物保鲜室122或冷冻室124。控制器150还与温控器(例如，热电偶或热敏电阻)通信。温控器可以设置在食物保鲜室122或冷冻室124(图2)中。控制器150可以从温控器接收与食物保鲜室122或冷冻室124的温度对应的信号。控制器150还可以包括用于计算经过的时间段的内部计时器。
控制器150可以包括存储器和一个或多个微处理器、CPU等，诸如通用或专用微处理器，该微处理器用于执行与制冷电器100的运行关联的编程指令或微控制代码。存储器可以表示诸如DRAM的随机存取存储器或诸如ROM或FLASH的只读存储器。在一些实施方式中，处理器执行存储在存储器中的永久编程指令。对于某些实施方式，指令包括软件包，该软件包被配置为操作电器100，或者执行操作例程(例如，下面参见图4描述的示例性方法400)。存储器可以是与处理器分开的部件，或者可以包含在处理器内的板上。可选地，控制器150可以在不使用微处理器的情况下(例如，使用离散的模拟或数字逻辑电路的组合；诸如开关、放大器、积分器、比较器、触发器、与门等)构建为执行控制功能，而不是依靠软件。
控制器150可以设置在整个制冷电器100中的各种位置。输入/输出(“I/O”)信号可以在控制器150与制冷电器100的各种操作部件之间路由。制冷电器100的一个或多个部件可以经由一条或多条传导信号线或共享的通信总线与控制器150通信(例如，电气通信)。另外或可选地，制冷电器100的一个或多个部件可以经由一个或多个无线信号带与控制器150通信(例如，无线通信)。
在一些实施方式中，制冷电器100还包括相机或相机模块160。相机160可以是适于捕获二维照片或图像的任意类型的装置。作为示例，相机160可以是具有电子 图像传感器【例如，电荷耦合器件(CCD)或CMOS传感器】的摄像机或数字相机。当组装时，相机160与控制器150通信(例如，电通信或无线通信)，使得控制器150可以从相机160接收与由相机160捕获的图像对应的信号。
通常，相机160设置在制冷电器100上。在一些实施方式中，相机160安装在食物保鲜室122内部的顶部(例如，邻近顶部101)处。比如，相机160可以固定到或被引导穿过限定食物保鲜室122的内胆的顶壁。在这种实施方式中，如图2所示，相机160可以指向下方，
在某些实施方式中，相机指向一个或多个制冷间室(例如，食物保鲜室122-图2)。比如，相机160可以指向抽屉142和搁物架144(图2)中的任何特定一个或组合的至少一部分。由此，相机160可以捕获抽屉142中的一个、所有抽屉142、搁物架144中的一个、所有搁物架144或其任何合适组合的图像。
虽然相机160被示例为安装在食物保鲜室122内，但是应当理解，另外或可选实施方式包括相机组件【例如，具有电子图像传感器【例如，电荷耦合器件(CCD)或CMOS传感器】的数字相机】，该相机组件安装在制冷电器的另一合适部分，诸如冷藏门体128。这种相机组件可以指向制冷电器100的外部或前面，由此可以捕获从制冷电器100向前定位的用户或区域的图像(例如，当对应的门体处于关闭位置时)。
在某些实施方式中，制冷电器100包括集成显示器180。集成显示器180可以安装在冷藏门体128(图1)上或制冷电器100上的任何其它合适位置处。集成显示器180与控制器150通信，使得集成显示器180可以从控制器150接收与由相机160捕获的图像对应的信号。集成显示器180可以从控制器150接收这种信号，并且向用户可视地呈现图像。集成显示器180可以包括例如液晶显示面板(LCD)、等离子显示面板(PDP)或用于显示图像的任何其他合适的机构(例如投影仪)。
与集成显示器180分开或除了集成显示器180之外，制冷电器100还可以包括与控制器150通信的通信组件184。通常，通信组件184可以安装在制冷电器100的任何合适的部分上，诸如箱体120或冷藏门体128内。在某些实施方式中，通信组件包括扬声器(例如，电动扬声器、静电扬声器、平面磁力扬声器、压电扬声器等)。如所理解的，扬声器可以被配置为利用一个或多个电信号(例如，从控制器150接收的数字声音信号)生成声波。由此，扬声器可以从听觉上将信息传达给用户。在其他或可选实施方式中，通信组件包括麦克风(例如，动圈麦克风、带式麦克风、光纤麦克风、压电麦克风等)。如所理解的，麦克风可以利用一个或多个接 收的声波(例如，从用户)生成一个或多个电信号(例如，将由控制器150接收)。由此，麦克风可以从用户处接收可被引导到控制器150的可听见的提示或命令。
在示例性实施方式中，制冷电器100包括网络接口(未示出)，该网络接口将制冷电器100(例如，控制器150)联接到网络190，使得制冷电器100可以通过网络190传输和接收信息。网络190可以是任何有线或无线网络，诸如WAN、LAN或HAN。
在可选实施方式中，制冷电器100包括用户检测组件186。通常，用户检测组件186包括一个或多个传感器，这些传感器被配置为检测对应于特定单个用户的生物特征或个性化标记。作为示例，用户检测组件186可以包括前向相机，该前向相机被配置为基于捕获的二维图像来识别用户的面部。作为另一示例，用户检测组件186可以包括被配置为视觉地检测用户指纹的指纹成像传感器。作为又一示例，用户检测组件186可以包括信号检测传感器，该信号检测传感器被配置为检测在无线通信频带(例如，使用2.4至2.485GHz的ISM频带中的短波长UHF无线电波的BLE频带)中的设备地址。设备地址可以是比如移动显示器182的编程的蓝牙地址。由此，用户检测组件186可以确定移动显示器182是否以及何时接近制冷电器100。
在某些实施方式中，相机160被作为用户检测组件186的一部分。作为示例，用户检测组件186可以被配置为从在相机160处捕获的二维图像识别用户。在一些这种实施方式中，控制器150还被配置为识别用户肘部下方的一个或多个定义特征，诸如肤色、手臂/手的大小、珠宝、典型的服装等。如所理解的，识别这种定义特征可以通过边缘匹配、分治搜索、灰度匹配、感受野响应的直方图或另一适当的例程(例如，基于从相机160捕获的一个或多个图像在控制器150处执行)来执行。
在使用期间，诸如在图像捕获序列期间，相机160可以捕获可被传输到控制器150(例如，作为数据信号)的一个或多个二维图像(例如，作为视频馈送或一系列连续静态图像)，如通常理解的。可选地，可以通过预定的用户动作来启动图像捕获序列，该用户动作诸如打开冷藏门体128、制冷间室(例如，食物保鲜室122)内检测到移动、占用用户检测组件186或在用户检测组件186的范围内移动、在通信组件184或集成显示器180处验证用户输入等。
在一些实施方式中，一个或多个远程服务器192，诸如网络服务器，与控制器150可操作地通信。远程服务器192可用于托管零售商的销售点系统。换言之，远程服务器192可以是或包括零售商销售点服务器，该服务器跟踪例如所购买物品的标 识符和数量、所购买物品的时间或日期戳、所购买物品的定价、顾客标识符(即，购买顾客的标识符)等。另外或可选地，远程服务器192可以用于托管零售商的库存管理系统。换言之，远程服务器192可以包括或被提供为零售商库存管理服务器，该服务器例如跟踪与由零售商提供出售的物品有关的库存数据(例如，过期数据、位置数据、成本数据等)。而且另外或可选地，远程服务器192可用于托管一个或多个信息数据库。
远程服务器192可以使用任何合适的计算装置来实施。远程服务器192可以包括一个或多个处理器和一个或多个存储装置(即，存储器)。一个或多个处理器可以是任何合适的处理装置(例如，处理器内核、微处理器、ASIC、FPGA、微控制器等)，并且可以是一个处理器或可操作地连接的多个处理器。存储装置可以包括一个或多个非瞬态计算机可读存储介质，诸如RAM、ROM、EEPROM、EPROM、闪存装置、磁盘等及其组合。存储装置可以存储由处理器执行以使远程服务器192执行操作的数据和指令。例如，指令可以是用于接收/传输销售点数据信号、接收/传输库存管理数据信号、接收/传输与所储存物品有关的数据信号等的指令。
存储装置还可以包括由处理器检索、操纵、创建或存储的数据，诸如销售点数据、顾客标识数据、库存数据、过期数据等。数据可以存储在一个或多个数据库中。一个或多个数据库可以通过高带宽LAN或WAN连接到远程服务器192，或者也可以通过网络502连接到远程服务器192。一个或多个数据库可以被分割，使得它们位于多个地点。
远程服务器192包括网络接口，使得远程服务器192可连接到一个或多个网络(例如，网络190)并通过该网络与一个或多个网络节点通信。远程服务器192又可通过网络190与一个或多个节点交换数据。特别地，远程服务器192可以与控制器150交换数据。虽然未示出，但可以理解，远程服务器192还可以通过网络190(例如，移动显示器182)与任意数量的客户端装置交换数据。
在某些实施方式中，相机160和控制器150被配置成捕获多个、连续的二维图像(例如，以预定速率或模式)，作为图像捕获序列的一部分。可以记录(例如，暂时)顺序图像(例如，先前捕获的图像和最近捕获的图像)并且在控制器150处进行比较。根据比较，可以检测顺序图像之间的变化或差异。可选地，随后，可以根据一个或多个结束条件，诸如预定时间段的到期、未能检测到顺序图像的进一步变化、或冷藏门体128的关闭，来停止或暂停图像捕获序列。
根据所捕获的图像，控制器150可以自动尝试识别在相机160的视场内的物品 (例如，诸如食物的所储存物品、或诸如用户附肢、搁物架、可移动抽屉等的非储存物品等)。如所理解的，识别这种物品可以通过边缘匹配、分治搜索、灰度匹配、感受野响应的直方图或另一适当的例程(例如，基于从相机160捕获的一个或多个图像在控制器150处执行)来执行。该识别可以包括或尝试估计所储存物品的标识(例如，识别出什么类型的食物或物体)或数量(例如，离散单元的数量、体积或质量)。可选地，可以向各个识别的储存物品分配识别分数。通常，识别分数提供识别或估计正确的概率或可能性。基于附加数据或确认(例如，从一个或多个远程服务器192接收)，可以更新或改善识别分数。
现在转向图4，提供了根据本发明的示例性实施方式的方法400的流程图。通常，图4提供了一种操作制冷电器100(图1)的方法，该制冷电器包括相机160，如上所述。方法400可以比如由控制器150(图3)执行。例如，如所讨论的，控制器150可以与相机160、集成显示器180(图3)、移动显示器182(图3)、通信组件184、用户检测组件186或远程服务器192通信。在操作期间，控制器150可以向相机160、集成显示器180、移动显示器182、通信组件184、用户检测组件186或远程服务器192发送信号和从其接收信号。控制器150通常还可以可操作地联接到电器100的其他合适部件，以促进电器100的操作。
有利地，根据本发明的方法可以允许改善检测例如制冷电器内的一个或多个物品。另外或可选地，检测可以在降低对制冷电器的一个或多个控制器、处理器或处理装置的处理需求的同时进行。
图4描述了为了示例和讨论的目的而以特定顺序执行的步骤。使用本文所提供的发明内容，本领域普通技术人员应该理解，本文所述的任何方法的步骤可以以各种方式修改、改编、重新排列、省略或扩展，而不脱离本发明的范围(除了以其他方式描述)。
在410处，方法400包括以下步骤：接收购买发票。通常，购买发票可以包括与从零售商购买的一个或多个物品有关的数据，诸如销售收据。作为示例，购买发票可以包括所购买物品的物品标识符。物品标识符可以通常或具体地指示所购买物品的类型或名称(例如，食品的类型)。作为另一示例，购买发票可以包括所购买物品的单位计数。购买的单位计数可以标定所购买物品的体积、质量或离散份额(例如，特定食品的数量)。作为又一示例，购买发票可以包括用于所购买物品的购买日期的时间或日期戳。由此，购买发票可以包括指示购买物品的时间或日期的数据。作为又一示例，购买发票可以包括顾客标识数据。这种顾客标识数据在对应 用户的特定电话号码、卡等已存储在零售商的销售点系统内并与特定购买关联的情况下可以被包括。
对于示例性示例，购买发票可以包括以下数据：顾客A在7月17日购买了3个苹果，其中，“苹果”是物品标识符，“3”是单位计数，“用户A”是顾客标识数据，并且“7月17日”是日期戳。
在一些实施方式中，购买发票直接从零售商接收。比如，购买发票可以从零售商销售点服务器传输到制冷电器。可选地，比如，如果用户先前已经在零售商的销售点系统处将她的顾客标识数据与制冷电器相关联，则可以提示零售商销售点服务器在完成销售时向特定的制冷电器传输购买发票。可选地，可以响应于用户在制冷电器的相机处扫描销售收据上的条形码，零售商销售点服务器被提示传输购买发票。在可选实施方式中，如将理解的，可以由制冷电器的相机直接扫描销售收据，并且可以启动一个或多个字符识别序列，以便解释销售收据的文本(例如，在制冷电器的控制器内)。由此，销售收据的文本可以被识别和解释，使得控制器可以记录(例如，临时地)销售收据中详述的各个购买物品。
与购买发票分开或除了购买发票之外，还可以接收与购买发票中的物品有关的补充数据。在一些实施方式中，可以从库存管理服务器接收购买发票中的物品的库存数据。库存数据可以包括比如过期数据(例如，预计过期日期、紧急召回信息等)或位置数据(例如，与制造商、供应商或零售商有关)。
在420处，方法400包括以下步骤：启动制冷电器内的物品识别序列。通常，物品识别序列可以包括捕获或测量在制冷电器中或附近的一个或多个储存物品的特性。比如，物品识别序列可以包括启动相机模块处的图像捕获序列(例如，第一图像捕获序列)，如上所述。第一图像捕获序列可以包括捕获多个二维图像(例如，第一二维图像、随后的第二二维图像等)，诸如在一个视频源或一系列连续静态图像(例如，根据预定速率或条件拍摄或捕获的)内。在相机模块处被捕获时，可以将二维图像传输到控制器(例如，作为数据信号)。然后可以记录(例如，暂时)二维图像以用于比较或评估。
在可选实施方式中，420取决于识别特定用户。在识别特定用户之前，可以防止记录或评估来自相机模块的二维图像。特定用户的识别可以包括接收指示生物特征或个性化标记的用户检测信号，该标记对应于特定单个用户，如上所述。作为示例，用户检测信号可以包括捕获的特定用户面部的二维图像。从所捕获的二维图像，可以识别用户。作为另一示例，用户检测信号可以包括一个指纹图像，从中可 以识别特定用户的指纹。作为又一示例，用户检测信号可以包括检测到的特定用户移动显示器的设备地址(例如，在无线通信频带上检测，诸如使用2.4至2.485GHz的ISM频带中的短波长UHF无线电波的BLE频带)。
在其他或可选实施方式中，420响应于检测移动(例如，在制冷电器内的相机模块处)而启动。由此，在检测到移动之前，可以防止记录或评估来自相机模块的二维图像。作为示例，可以利用指向制冷间室的相机模块检测移动。具体地，如通常理解的，可以检测由相机模块捕获的光或像素的变化(例如，在随时间推移捕获的多个图像之间)，该变化指示相机模块的视场内的一个或多个物体的移动。
可选地，物品识别序列(例如，第一图像捕获序列)可以持续进行直到满足一个或多个结束条件为止。作为示例，结束条件可以包括在开始对应的移动检测之后的预定时间段(例如，时间跨度)的到期。作为另一示例，结束条件可包括未能检测到第一图像俘获序列的顺序图像的进一步变化。换言之，第一图像捕获序列可以在顺序图像停止变化或检测到进一步的移动之后结束。结束条件可以具体地要求未改变的顺序图像的预设数目或时间段。作为又一示例，结束条件可以包括检测制冷间室的门体的关闭。换言之，第一图像捕获序列可以响应于门体被移动到关闭位置而结束。
在430处，方法400包括以下步骤：基于420处的物品识别序列生成第一储存物品的初始识别尝试。尝试可以包括尝试基于第一图像捕获序列的第一二维图像识别第一储存物品。换言之，使用图像捕获序列的至少一个二维图像，可以识别第一储存物品(例如，食品)。如上所述，尝试识别可能要求第一储存物品在相机模块的视野下方或视野内通过。由此，第一二维图像可以至少包括第一储存物品的所捕获部分。而且，一旦被捕获，就可以由边缘匹配、分治搜索、灰度匹配、感受野响应的直方图或另一合适的例程执行第一储存物品的识别。
可选地，初始识别尝试可以包括第一储存物品的估计标识符。估计标识符可以通常或具体地指示第一储存物品的类型或名称(例如，食品的类型)。换言之，初始识别尝试可以尝试命名在430处识别的食品是什么类型。另外或可选地，初始识别尝试可以包括第一储存物品的估计单位计数。估计的单位计数可以标定所购买物品的体积、质量或离散份额(例如，特定食品的数量)。换言之，初始识别尝试可以尝试测量或统计在430处识别了多少食品。可选地，初始识别尝试可以包括对应于第一储存物品的多个可能对象。
在某些实施方式中，可以为初始识别尝试分配识别分数。识别分数可以对应于 正确识别第一储存物品的概率(例如，百分比)。换言之，识别分数可以向第一储存物品是特定可识别对象的可能性分配一个值。可选地，初始识别可以向第一物品分配多个离散的识别分数——各个识别分数对应于不同的可识别对象。于示例性示例，可以分配对象I的识别分数为90％，以指示第一储存物品是可识别对象I的概率为90％，可以分配对象II的识别分数为70％，以指示第一储存物品是可识别对象II的概率为70％，等。
在440处，方法400包括以下步骤：将第一储存物品的初始识别尝试与购买发票进行比较。该比较可以寻求使初始识别尝试与购买发票中的条目相关或联系。比如，该比较可以将来自430的一个或多个可能对象(例如，估计的标识符、估计的单位计数等)匹配到购买发票中的一个或多个条目(例如，所购买物品标识符、所购买单位计数等)。
在可选实施方式中，440包括基于日期戳或顾客标识数据选择用于比较的购买发票。作为示例，购买发票可以基于日期戳从多个发票中选择。在某些实施方式中，购买发票的日期戳基于是最近的购买发票来选择。作为另外或可选的示例，购买发票可以基于顾客标识数据从多个发票中选择。在一些实施方式中，如上所述，顾客数据被匹配到先前已经由电器识别的特定用户。
在450处，方法400包括以下步骤：基于440处的比较生成第一储存物品的修改识别尝试。特别地，可以根据比较更新初始识别尝试。比如，如果在430处生成识别分数，则可以基于在购买发票中是否匹配一个或多个可识别对象来改变分数。如果在购买发票中匹配到(例如，找到)来自440的可识别对象，则可以增加识别分数，这指示第一储存物品更可能是所估计的标识符或所估计的单位计数更可能是正确的。相反，如果在购买发票中未匹配到(例如，找到)来自440的可识别对象，则可以减小识别分数，这指示第一储存物品更不可能是所估计的标识符或所估计的单位计数更不可能是正确的。如果初始识别尝试包括多个可能的对象，则440可以包括丢弃或降级未匹配到购买发票中的对应条目的任意可能对象。
在可选实施方式中，方法400包括以下步骤：请求修改识别尝试的用户确认。比如，电器可以生成视觉或听觉查询(例如，经由通信组件)，该查询询问用户识别尝试(例如，具有最高识别分数的对象)是否正确。作为响应，用户可以确认或拒绝修改识别尝试，诸如经由物理输入或语音控制，如所理解的。在一些这种实施方式中，对用户确认的请求仅在确定识别分数低于预定阈值(例如，50％)时才启动。由此，在这种实施方式中，在确定识别分数高于预定阈值时，不生成对用户确 认的请求。
在460处，方法400包括以下步骤：将第一储存物品的标签记录到制冷电器内的库存。特别地，标签可以基于修改识别尝试(或其确认)来选择。通常，标签提供与第一储存物品有关的文本或图像(例如，如由修改识别尝试识别的)。可选地，标签可以包括所估计的标识符或所估计的单位计数。其他或可选地，标签可以包括库存数据(例如，从库存管理服务器接收)，诸如过期数据或位置数据。
在可选实施方式中，记录标签会提示对与一个或多个所购买物品有关的信息进行外部搜索。比如，可以使用标签来启动一个或多个远程服务器的搜索。该搜索可以通过因特网或万维网来执行(例如，使用已知的搜索引擎)。可选地，搜索可以限于一个或多个预定的网址、网站或服务器。通常，搜索可以表示对关于所储存物品的信息(诸如描述性信息、处理信息或第一物品出现的方式)的请求。
一旦被记录，用户就可以在集成显示器或移动显示器处选择性地调出或查看库存(包括第一储存物品的标签)，以检查电器内当前储存了哪些物品。在从电器取出第一储存物品之后，电器可以比如自动地或响应于手动用户输入来更新库存，以从库存去除标签。其他或可选地，电器可以自动确定第一储存物品何时过期(例如，基于库存数据)，并且在集成显示器、移动显示器或通信组件处生成警报(例如，音频或视觉警报消息)。
本书面描述使用示例对本发明进行了公开(其中包括最佳实施例)，并且还使本领域技术人员能够实施本发明(其中包括制造和使用任何装置或系统并且执行所包含的任何方法)。本发明的可专利范围通过权利要求进行限定，并且可以包括本领域技术人员能够想到的其它的示例。如果这种其它的示例包括与权利要求的字面语言没有区别的结构元件，或者如果这种其它的示例包括与权利要求的字面语言没有实质区别的等同结构元件，则期望这种其它的示例落入权利要求的范围中。

Claims (18)

  1. 一种制冷电器，其特征在于，该制冷电器包括：
    箱体，该箱体限定制冷间室；
    门体，该门体旋转地铰接到所述箱体，以提供选择性地进入所述制冷间室；
    相机模块，该相机模块安装到所述箱体；以及
    控制器，该控制器可操作地联接到所述相机模块，所述控制器被配置为启动操作例程，该操作例程包括：
    接收购买发票；
    启动所述制冷电器内的物品识别序列；
    基于所述物品识别序列生成第一储存物品的初始识别尝试；
    将所述第一储存物品的所述初始识别尝试与所述购买发票进行比较；
    基于所述第一储存物品的所述初始识别尝试与所述购买发票的所述比较，来生成所述第一储存物品的修改识别尝试；以及
    将所述第一储存物品的标签记录到所述制冷电器内的库存。
  2. 根据权利要求1所述的制冷电器，其特征在于，所述购买发票包括所购买物品的物品标识符，其中，所述初始识别尝试和所述修改识别尝试包括所述第一储存物品的估计标识符。
  3. 根据权利要求1所述的制冷电器，其特征在于，所述购买发票包括所购买物品的单位计数，其中，所述初始识别尝试和所述修改识别尝试包括所述第一储存物品的估计单位计数。
  4. 根据权利要求1所述的制冷电器，其特征在于，所述购买发票包括所购买物品的购买日期的日期戳。
  5. 根据权利要求1所述的制冷电器，其特征在于，所述购买发票从零售商销售点服务器接收。
  6. 根据权利要求5所述的制冷电器，其特征在于，还包括：接收与一个或多个所述购买物品有关的过期数据。
  7. 根据权利要求1所述的制冷电器，其特征在于，还包括：请求所述修改识别尝试的用户确认。
  8. 根据权利要求1所述的制冷电器，其特征在于，记录提示对与一个或多个所 述购买物品有关的信息进行外部搜索。
  9. 根据权利要求1所述的制冷电器，其特征在于，所述相机模块安装在箱体内，并位于制冷间室的顶部处，其中，所述物品识别序列包括：
    启动所述相机模块处的第一图像捕获序列，
    其中，基于所述物品识别序列生成第一储存物品的初始识别尝试包括基于所述第一图像捕获序列的二维图像识别所述第一储存物品。
  10. 一种操作制冷电器的方法，其特征在于，所述方法包括以下步骤：
    接收购买发票；
    启动所述制冷电器内的物品识别序列；
    基于所述物品识别序列生成第一储存物品的初始识别尝试；
    将所述第一储存物品的所述初始识别尝试与所述购买发票进行比较；
    基于所述第一储存物品的所述初始识别尝试与所述购买发票的所述比较，来生成所述第一储存物品的修改识别尝试；以及
    将所述第一储存物品的标签记录到所述制冷电器内的库存。
  11. 根据权利要求10所述的方法，其特征在于，所述购买发票包括所购买物品的物品标识符，其中，所述初始识别尝试和所述修改识别尝试包括所述第一储存物品的估计标识符。
  12. 根据权利要求10所述的方法，其特征在于，所述购买发票包括所购买物品的单位计数，其中，所述初始识别尝试和所述修改识别尝试包括所述第一储存物品的估计单位计数。
  13. 根据权利要求10所述的方法，其特征在于，所述购买发票包括所购买物品的购买日期的日期戳。
  14. 根据权利要求10所述的方法，其特征在于，所述购买发票从零售商销售点服务器接收。
  15. 根据权利要求14所述的方法，其特征在于，还包括以下步骤：接收与一个或多个所述购买物品有关的过期数据。
  16. 根据权利要求10所述的方法，其特征在于，还包括以下步骤：请求所述修改识别尝试的用户确认。
  17. 根据权利要求10所述的方法，其特征在于，记录提示对与一个或多个所述购买物品有关的信息进行外部搜索。
  18. 根据权利要求10所述的方法，其特征在于，相机模块安装在箱体内，并位 于所述制冷电器的制冷间室的顶部处，其中，所述物品识别序列包括：
    启动所述相机模块处的第一图像捕获序列，
    其中，基于所述物品识别序列生成第一储存物品的初始识别尝试包括基于所述第一图像捕获序列的二维图像识别所述第一储存物品。
PCT/CN2020/117264 2019-09-25 2020-09-24 制冷电器和用于跟踪所储存物品的方法 WO2021057820A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/582,158 2019-09-25
US16/582,158 US11288506B2 (en) 2019-09-25 2019-09-25 Refrigerator appliances and methods for tracking stored items

Publications (1)

Publication Number Publication Date
WO2021057820A1 true WO2021057820A1 (zh) 2021-04-01

Family

ID=74882104

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/117264 WO2021057820A1 (zh) 2019-09-25 2020-09-24 制冷电器和用于跟踪所储存物品的方法

Country Status (2)

Country Link
US (1) US11288506B2 (zh)
WO (1) WO2021057820A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11892339B2 (en) * 2021-09-01 2024-02-06 Alejandro Alemany Food inventory method and system
US20230097905A1 (en) * 2021-09-24 2023-03-30 Haier Us Appliance Solutions, Inc. Inventory management system in a refrigerator appliance
US11928769B2 (en) 2022-07-20 2024-03-12 Samsung Electronics Co., Ltd. Employing controlled illumination for hyperspectral or multispectral imaging of food in an appliance

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103196277A (zh) * 2013-03-29 2013-07-10 惠州Tcl家电集团有限公司 冰箱食品管理方法和系统、服务器、冰箱
CN103383747A (zh) * 2012-05-04 2013-11-06 海尔集团公司 冰箱、食品管理系统及方法
CN105758108A (zh) * 2016-03-09 2016-07-13 北京京东尚科信息技术有限公司 用于智能冷藏设备的信息反馈方法和装置
CN106247752A (zh) * 2016-09-29 2016-12-21 四川长虹电器股份有限公司 基于物联网的冰箱物品存放系统
CN106871571A (zh) * 2017-02-08 2017-06-20 佛山市小鲜互联电器科技有限公司 一种根据存储物情况进行购物提示的冰箱
US20180100681A1 (en) * 2016-10-07 2018-04-12 Bsh Hausgeraete Gmbh Refrigeration device with ice maker
CN108759294A (zh) * 2018-07-14 2018-11-06 王长贵 一种冰箱智能识别和管理系统
CN208620698U (zh) * 2018-06-29 2019-03-19 北京金茂绿建科技有限公司 一种食材监控装置和冰箱

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002115956A (ja) 2000-10-11 2002-04-19 Matsushita Electric Ind Co Ltd 在庫管理冷蔵庫
US9208520B2 (en) 2011-04-28 2015-12-08 International Business Machines Corporation Shelf life expiration date management
KR101375906B1 (ko) 2011-09-19 2014-03-21 인텔렉추얼디스커버리 주식회사 간편한 식품 관리를 위한 냉장고, 식품 관리 서버 및 식품 관리 방법
US20160088262A1 (en) 2013-04-10 2016-03-24 Lg Electronics Inc. Method For Managing Storage Product In Refrigerator Using Image Recognition, And Refrigerator For Same
WO2017010804A1 (en) 2015-07-15 2017-01-19 Samsung Electronics Co., Ltd. Object recognition for a storage structure
US10474987B2 (en) 2015-08-05 2019-11-12 Whirlpool Corporation Object recognition system for an appliance and method for managing household inventory of consumables
US20180232689A1 (en) * 2017-02-13 2018-08-16 Iceberg Luxembourg S.A.R.L. Computer Vision Based Food System And Method
US20190164116A1 (en) * 2017-11-30 2019-05-30 Perfect Company, INC. System for capturing verified consumption profiles for food items

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103383747A (zh) * 2012-05-04 2013-11-06 海尔集团公司 冰箱、食品管理系统及方法
CN103196277A (zh) * 2013-03-29 2013-07-10 惠州Tcl家电集团有限公司 冰箱食品管理方法和系统、服务器、冰箱
CN105758108A (zh) * 2016-03-09 2016-07-13 北京京东尚科信息技术有限公司 用于智能冷藏设备的信息反馈方法和装置
CN106247752A (zh) * 2016-09-29 2016-12-21 四川长虹电器股份有限公司 基于物联网的冰箱物品存放系统
US20180100681A1 (en) * 2016-10-07 2018-04-12 Bsh Hausgeraete Gmbh Refrigeration device with ice maker
CN106871571A (zh) * 2017-02-08 2017-06-20 佛山市小鲜互联电器科技有限公司 一种根据存储物情况进行购物提示的冰箱
CN208620698U (zh) * 2018-06-29 2019-03-19 北京金茂绿建科技有限公司 一种食材监控装置和冰箱
CN108759294A (zh) * 2018-07-14 2018-11-06 王长贵 一种冰箱智能识别和管理系统

Also Published As

Publication number Publication date
US11288506B2 (en) 2022-03-29
US20210089769A1 (en) 2021-03-25

Similar Documents

Publication Publication Date Title
WO2021057820A1 (zh) 制冷电器和用于跟踪所储存物品的方法
US9547851B2 (en) Shelf life expiration date management
JP2018146177A (ja) 物品貯蔵庫の在庫管理装置
US11335010B2 (en) Methods for viewing and tracking stored items
US11493268B2 (en) Refrigerator appliance having a weight-detecting drawer assembly
JP7012263B2 (ja) 冷蔵庫
WO2021212993A1 (zh) 制冷电器摄像头模块和用于防止镜头起雾的方法
JP7281755B2 (ja) 冷蔵庫
US10785456B1 (en) Methods for viewing and tracking stored items
WO2018045839A1 (zh) 用于智能冰箱的购物触发方法、设备和计算机可读存储介质
CN113465264A (zh) 一种智能冰箱及食材管理方法
JP2022103368A (ja) 冷蔵庫
JP2006317109A (ja) 冷蔵庫
US11732960B2 (en) Refrigerator appliance having a weight-detecting shelf assembly
JP7475000B2 (ja) 入出庫管理装置、入出庫管理システム、及び、入出庫管理方法
WO2022095995A1 (zh) 制冷电器和用于跟踪所储存物品的方法
WO2023193756A1 (zh) 制冷设备及其控制方法
WO2021057768A1 (zh) 制冷电器和用于选择性地锁定其中的子容器的方法
US11927389B2 (en) Refrigerator appliance having a weight-detecting shelf assembly
CN113483525A (zh) 保鲜设备及食材管理方法
CN109387024B (zh) 用于辨识制冷器具中的对象的方法以及制冷器具
JP7289084B2 (ja) 冷蔵庫
US11692767B2 (en) Refrigerator appliance and methods of user identification
JP7289083B2 (ja) 冷蔵庫
CN116343196A (zh) 一种食材管理方法及智能冰箱

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20869178

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20869178

Country of ref document: EP

Kind code of ref document: A1